He was a close associate of the Cloistered Emperor Goshirakawa, and in 1179 when TAIRA no Kiyomori confined the Cloistered Emperor to Toba-dono Palace and dismissed a group of many of his kinshin (trusted vassals), Sueyoshi was relieved of his post as the Echigo no kuni no kami (Governor of Echigo Province) (Coup of the Third Year of Jisho.)
後白河法皇の側近として活動し、治承3年（1179年）に平清盛が法皇を鳥羽殿に幽閉しその近臣一派を大量に解任した際には、一旦越後国守の職を解かれている（治承三年の政変）。