The manor of Milford Baddesley originated in an estate held in Milford by the Knights Templar. In the time of King John, Hugh de Whitwell and his son William granted land at Milford to William Mackerel which he granted to the Templars, for their preceptory of Baddesley. It was held of Christchurch manor. On the suppression of the order of Knights Templar this estate was granted, about 1312 to the Knights Hospitaller, to whom it continued to belong until the Dissolution of 1540. It was acquired by Robert Rickman around 1609, in whose family it remained into the 18th century. In 1829 Mary Anna Theresa Whitby, who had inherited Admiral Cornwallis' estates purchased Milford Baddesley, thus uniting all three estates.