The current house was built in 1639 for John Maitland, 1st Duke of Lauderdale, incorporating an L-plan house dating from the 1560s and early 14th century elements built for the Crichton family. It was extended by Sir William Bruce in 1672 and bought by Andrew Fletcher, Lord Milton in 1733. He employed William Adam to rebuild parts of the house and install interior panelling, plasterwork and other features. The house includes some of the earliest known examples of sash windows in Scotland (invented in 1690).