One of the main functions of Yasutsune was to negotiate with samurai but, being at the center of the administration, he had numerous downfalls with the first one occurring in 1179 when the Retired Emperor Goshirakawa was confined by TAIRA no Kiyomori whereby Yasutsune was removed from office and, in 1183, he was removed from office when the Retired Emperor Goshirakawa was once again confined this time by MINAMOTO no Yoshinaka.
主に武家との折衝にあたるが、政権の中枢にあったために失脚することも多く、治承3年（1179年）平清盛による後白河院の鳥羽殿幽閉の際に解官されたのを手始めとして、復帰後の寿永2年（1183年）源義仲による再度の後白河院幽閉の際にも再び解官。