His father, Takaakira, who was at the high rank of Sadaijin (Minister of Left), experienced a bitter moment when he was demoted to the rank of Dazai gonno sochi (Viceroy of Dazai-fu (the government of Kyushu)) due to the conspiracy of the Northern House of the FUJIWARA clan (Anna no Hen (Conspiracy of Anna)) in March 969, when Toshikata was eleven years old.
冷泉天皇の安和2年（969年）3月、11歳の時、左大臣の高位にあった父高明が藤原北家の陰謀によって大宰権帥に落とされる憂き目に見舞われる（安和の変）。