The school occupies a site close to the centre of the small market town of Caistor, with remarkably steep topography. The ironstone school hall dates from 1631, and is still in daily use. The school library alongside is housed in what was the Congregational Church, built in 1842. Casterby House, once a large private house, later one of three boarding houses, and now the Sixth Form Centre, overlooks the churchyard from the south side of the school gates. The name Casterby House is derived from former pupil Sir Henry Newbolt's semi-autobiographical novel the Twymans, in which Caistor is given the name Casterby.