In 1106 Savaric fitz Cana (Fitzcane) received land in Midhurst and the neighbouring village of Easebourne from Henry 1, and in 1158 his son built a fortified manor house on St. Anne's Hill. The family later adopted the de Bohun name, and in about 1280 abandoned the fortified manor house to build their principal home on flat land across the River Rother from St. Ann's Hill, in the neighbouring parish of Easebourne, 'at a place called Coudreye' (old French for a "hazel grove"). Between 1284 and 1311 St Ann's Castle was in the hands of the Bishop of Durham, and during that period was largely dismantled.