From 1365 to 1565, the manor was in the possession of the Tyrrell family. The manor passed through various hands in subsequent centuries. The last significant owners were the Bursey family in the 19th century, and in the 1890s the remaining lands of the estate were subdivided and sold. In close proximity to Milton was the manor of Fernhill. In the Domesday book, it was held by Nigel from Roger de Montgomerie, 1st Earl of Shrewsbury. In the 15th century, John Fromond, the owner of this scattered estate, willed the lands to Winchester College, in whose hands the estate remained down to the 19th century. Another estate called "Gore" appeared by the 15th century, and still survives (partially) as a farm to the west of the town.