The castle consists of an altered 15th century keep of three storeys and a garret within a corbelled-out parapet. It has been altered, with the insertion of larger windows, and then extended in the 18th century. The original entrance, which has been built into later work, led to the barrel-vaulted basement, and to the main stair to the first floor hall.
Culcleuch was a property of the Galbraiths of Culcreuch, but sold to the Setons in 1630, then to the Napiers two years later. In 1796 it was acquired by Alexander Speirs, who built a large profitable cotton mill on his estate, which had passed from the family by 1875. The fine building is now a hotel, and has the largest colony of bats in Britain living in the roof area above the dining room. The Chinese Bird Room, within the old part of the building, has hand-painted Chinese wallpaper dating from 1723, believed to be the only surviving example of this period in Scotland.
The castle is reputed to have several ghosts. The Phantom Harper of Culcleuch relates to events believed to have taken place in 1582. One of the Buchanan family was mortally wounded by Robert Galbraith, son the 16th chief. The dying man was taken to what is now the Chinese Bird Room, accompanied by his mistress. When he died, to comfort herself she began to play a wire-strung harp and it is said that her soft music has often been heard since, particularly in the dead of night.