Danby was not finished: further Gothic alterations were effected by Robert Lugar: turrets and battlements were added, so that the building took on the appearance of a castle; the richly furnished museum of minerals, which has since become a family chapel, was built, and at the same time a tower Describing a tour which he made in 1829, the poet Robert Southey remarked, "The most interesting person whom I saw during this expedition was Mr Danby of Swinton Park, a man of very large fortune, and now very old." During the early 19th century the building was substantially altered, and two-storey west and north wings were added. Danby died in 1833, but his widow continued to live at the house with her second husband, naval officer Octavius Vernon Harcourt (High Sheriff for 1849) until her own death in 1879. She devised her Yorkshire estates to George, fifth son of Sir Robert Affleck, Bt., a member of the family of William Danby's mother, who then took the name Danby.