Although it is supposed that a castle had already been built on Sakato-yama Mountain at this time, placed at the center of the manor, it is considered that it was in the period of the Northern and Southern Courts (Japan) or later that the castle was constructed on a large scale.
当時、荘の中心を扼する坂戸山に城郭が築かれていたものとも推定されるが、本格的な造営は南北朝時代(日本)にはいって以降のことと考えられている。