By 1834, the castle was unoccupied again. In 1840, an advert was placed in the "Newcastle Courant" by Revd. John Wood for "Hylton Castle Boarding School" and the 1841 census shows Wood, his family, pupils and staff as living on the estate. Joseph Swan was one of the pupils there around this time. The school does not seem to have existed for long as Howitt commented in 1842, that it was "a scene of great desolation ... the windows for the most part, all along the front, are boarded up ... the whole of this large old house is now empty ... and in the most desolate state". However, he does go on to say the kitchen was occupied a poor family. By 1844, the chapel was used as a carpenter's workshop, and according to the "Durham Chronicle" in January 1856, the castle set on fire while in the occupation of a farmer, Mr. Maclaren.