The Manor House is a Grade II* listed building, includes a Great Hall (or perhaps a barn) of the 15th century, and a small irregular manor house of the late 16th century. It stands on the site of, and incorporates parts of, the castle built by William Malet, a Norman knight who fought at the Battle of Hastings.The entrance hall extension and other parts were upgraded and connected with the main house c 1939 by Clough Williams-Ellis whose client was the Selous family. The Manor House was lived in by the Pyne family who are buried in an Elizabethan family tomb to be found in the Mallet Chapel in the village Church of St James. Elizabeth II visited the house as part of her duties when she was head of the Duchy of Cornwall. The house is associated with the memory of Derek Jarman, film-maker, artist and gay activist.