Situated in 17 acres of woodland, Fernie Castle consists of an altered 16th century L-plan tower house, to which has been added a three-storey block with a round tower at one corner. There may have been a castle here from the 13th century. The stair-wing is crowned by a corbiestepped gabled watch-chamber. There are other extensions, and the walls are harled and yellow-washed. Fernie was a property of the MacDuff Earls of Fife, but passed to the Balfour family, then by the 15th century to the Fernies. The castle went by marriage to the Lovells, who in 1586 sold it to the Arnots, but then passed by marriage back to the Balfours. The Balfours were forfeited for their part in the Jacobite Rising of 1715 although they recovered the property in 1720. In recent years the castle has become a hotel.
The west tower of the building is said to be haunted by 'A Green Lady', a girl who ran off with her lover. They sought refuge in the castle, but were discovered by her father's men - her father disapproved of her lover. The poor woman fell three floors from the west tower to her death, and her apparition is said to have been seen in some of the bedrooms. Other manifestations include electrical equipment and lights switching themselves on and off.