When his father Yoshihi was killed in battle in 1581, his brother Tadafusa SAGARA succeeded to the position of the family head of the Sagara clan, and Yorifusa was sent to Satsuma as a hostage of the Shimazu clan.
天正9年（1581年）に父・義陽が戦死すると、相良氏の家督は兄の相良忠房が継ぎ、頼房は島津氏の人質として薩摩に送られた。