There is a legend that the tower was built by "Lord de Freston" in the 15th century for his daughter Ellen, so she could study a different subject on a different floor six days of the week: the 1st floor was dedicated to reception, the 2nd to tapestry working, the 3rd to music, the 4th to painting, the 5th to literature and the 6th to astronomy, complete with instruments for taking observations. This was written about in a novel by Reverend Richard Cobbold entitled "Freston Tower" and so should not be taken as fact. There is much evidence against this legend, such as documents apparently referring to the construction of the tower within the twelve years preceding 1569, and it is likely that the tower was constructed by Edmond Latymer as a lookout over Freston Reach of the River Orwell. There are further descriptions suggesting a construction date of 1655.