After Takanoa, Ueno Hizen no kami Takatoku, his legitimate child, took over as head of the family and became the lord of Kimurayyama-jo Castle, but he moved to Bizen Tsuneyama-jo Castle at a certain point between 1555 and 1558 and resided in this castle.
高直の後は、嫡子上野肥前守高徳が家督を継ぎ喜村山城主となったが、高徳は、弘冶年中(1555～1558)に備前常山城に移り、この城を居城とした。