At first, when he assumed the position of the Saemon no Gon no Suke (Provisional Assistant Captain of the Left Division of Outer Palace Guards), his elder brother, Tsunefuji YOSHIDA became a priest, as he was ashamed of being overtaken in promotion of an official rank and, when he became Junii (Junior Second Rank), Tadakata ANEKOJI, who was a relative, also became a priest due to the shock of being out promoted, furthermore, when he became the Gon Dainagon, he assumed the position in a form to kick Takaaki SHIJO (uncle of the Gofukakusain no Nijo) out of the position.
まず、左衛門権佐就任時には彼の異母兄吉田経藤が官職を抜かされた屈辱から出家し、従二位叙任の際にも縁戚に当たる姉小路忠方が出世争いに敗れた衝撃からこれも出家、更に権大納言就任時には四条隆顕（後深草院二条の叔父）を蹴落とす形で就任している。