To the east of the church is Ilsington House, also known as the Old Manor, which was built in the late 17th to early 18th century. It was originally owned by the 3rd Earl of Huntingdon and in 1724 by Robert Walpole. Between 1780 and 1830 it was leased to General Thomas Garth, principal equerry to King George III. The General adopted King George III's illegitimate grandson by Princess Sophia, and brought him up at the manor. In 1861 the house was acquired by John Brymer and remained in the possession of the Brymer family for the next century. The family built new cottages and a reading room in the village, and a new manor next to the church, which they restored.