Since his older brother Kagetaka and others were forfeited their land by Hideyoshi HASHIBA as they took the side of Katsuie SHIBATA, Kagetomo succeeded the family headship of the Yamaoka clan.
兄景隆たちが柴田勝家に付き羽柴秀吉により領地没収となるなどしたため、山岡氏家督となる。