His elder brother FUJIWARA no Norinaga was exiled to Hitachi Province as an aide of Emperor Sutoku at the Hogen War, and Yorisuke had to stay in his house in Higashiyama (Kyoto Prefecture) although Yorisuke was not willingly involved in the incident.
保元の乱では、兄の藤原教長が崇徳天皇の側近として常陸国に配流され、頼輔は積極的に関与はしなかったものの、東山(京都府)に籠居した。