The estate subsequently passed to the Bruce family, and then to Lord Carleton, who bequeathed it to his nephew Charles Douglas, 3rd Duke of Queensberry. The grounds include a Chinese summer house commissioned by the Duke of Queensbury, and designed by Sir William Chambers. The estate remained in the Queensberry family until 1824. It is believed that at some point in the early 19th century, William Douglas, 4th Duke of Queensberry planted the Nile Clumps to commemorate Admiral Nelson, and had the hill-fort landscaped as part of the grounds around the mansion.