In 1185, when Takamoto, who was accused by MINAMOTO no Yoritomo of MINAMOTO no Yoshitsune's imperial decree to hunt down and kill his older brother Yoritomo, was overthrown together with TAKASHINA no Yasutsune and others, Hirofusa was appointed to Sadaishi.
文治元年（1185年）、源義経による兄・源頼朝追討宣旨を巡って頼朝から糾弾された隆職が高階泰経らとともに失脚すると、広房が左大史となる。