Some Ancient British, Viking and Anglo-Saxon remains have been found in the vicinity. The Norman baron Robert de Stuteville built a wooden moated castle on Vivers Hill. The estate passed to the Wake family in the 13th century, who brought prosperity to the town. However, it was badly hit by the Black Death of the mid-14th century, after which the wooden castle lay in ruins. Prosperity returned after 1408, when the Neville family took over, although little remains of the fortified manor they built to the north of the town. The Nevilles remained Catholic and took part in the Rising of the North of 1569. By 1660 there was a grammar school. (The building is now part of the library.) The great Toll Booth in the middle of the town was built about 1730 with stone taken from the Nevilles' manor. The old Market Hall was gutted by fire but rebuilt in 1872. By 1881 the population of the town was 2,337.