After that, the castle was used by Hirosue ANDO, Kiyosue ANDO, Chikasue ANDO, Sanesuke AKITA as the headquarters of the Hiyama Ando clan of Shimonokuni family during five generations, and it seems that the clan extended its influence toward Hinai as well as Ani in Mutsu Province, using the castle as its stronghold.
以後、安東尋季、安東舜季、安東愛季、秋田実季まで5代にわたり下国家檜山安東氏の居城となり、ここを本拠に陸奥国比内、同国阿仁方面に勢力を拡大したと見られる。