It is said that when MINAMOTO no Yoritomo raised his army, he was a hikan (a low-level bureaucrat) of TAIRA no Tomonori together with Mitsutomo AKIYAMA, his older brother, and they joined Yoritomo's army by having the authorization from Tomonori to return home under the pretext of their mother's becoming sick.
源頼朝挙兵時は、兄・秋山光朝とともに京都で平知盛の被官であったとされ、母の病気を理由に帰国を願い出て許され、源頼朝の下に参じたと伝えられる。