In 1837 a programme of budget cuts at the Admiralty led to Taylor's dismissal. He took up general practice, and qualified as a district surveyor. In 1838 he began the spectacular Gothick tower at Hadlow Castle, a late eighteenth century house in Kent. The tower was based in part on James Wyatt's at Fonthill Abbey. It was built of brick rendered in Roman cement to imitate stone, the finer architectural detail built up with the cement. He had previously used the Gothick style at a church in Walham Green, Fulham in 1827-8.