He was from the Yasutomi clan, a chief vassal clan of the Hosokawa-Keicho family, that handed down the position of Shugodai (deputy of Shugo, provincial constable) of the eastern region of Sanuki Province for generations; additionally, he was a soryo (heir) of the clan.
細川京兆家の重臣として讃岐国東方守護代を世襲していた安富氏の一族、もしくは惣領。