It is said that the family lived in Nakashi-jo Castle in Kugami in Inaba Province in the days of his father Yoshichiro Masayoshi, and expanded influence by taking advantage of a conflict started between Takanobu TAKEDA and the Yamana clan, a military governor, in the Eiroku era.
父の与七郎正義の代には陸上の中島城(因幡国)に居城し、永禄年間に武田高信と守護・山名氏の間で起こった内乱に乗じて勢力を拡大したという。