As his sisters were concubines of Yoshiharu ASHIKAGA who was the Seii taishogun (literally, "great general who subdues the barbarians"), Harumitsu enjoyed a deep confidence of the Shogun Ashikaga family and conferred Jushiinoge (Junior Fourth Rank, Lower Grade) in 1559, succeeded in the government service such as Emon fu (the Gate Guards), Kokushi (an officer of local government) in Kazusa Province and Mutsu Province.
また、姉妹が征夷大将軍足利義晴の側室であった事から足利将軍家の覚えもめでたく、永禄2年（1559年）には従四位下に叙せられ、衛門府・上総国国司・陸奥国国司などを歴任した。