It is sometimes stated that Castle Upton contains parts of the earlier priory buildings, but other sources conclude that no part of the building predates the work of the Norton family. Even the existence of a house of the Knights of St John at this site has been questioned, due to lack of evidence. Sir Robert Norton, an officer under Sir Arthur Chichester, Governor of Carrickfergus, obtained lands along the Six Mile Water, and started work on the castle in the late 16th century, building what is now the east wing. This work was completed by Sir Humphrey Norton in the first part of the 17th century, and like many castles of the Plantation period was surrounded by a bawn wall.