It is said that his family descended from the Sugawara clan, but there are a lot of suspicious records about it (it is also considered that the Yagyu was the same family of the Akamatsu clan in Harima Province).
遠祖は菅原氏支流とされるが、疑わしい箇所も多い（或いは播磨国の赤松氏と同族ともされる）。