In November, 1179, when FUJIWARA no Tadakiyo, a powerful member of the Heike, was appointed the Kazusa no suke, Hirotsune fought with him over state affairs and was expelled by TAIRA no Kiyomori.
治承3年（1179年）11月、平家の有力家人藤原忠清が上総介に任ずると、広常は国務を巡って忠清と対立し、平清盛に勘当された。