In 1218, he recommended his son Tadasada for Sangi, and retired from Dainagon (chief councilor of state), but two years after in 1220 he demanded Tadasada's resignation because of '子強好交衆, 又増不孝耳' (according to the entry dated April 9, 1229 (old calendar) in "Meigetsuki" (Chronicle of the Bright Moon)), and the conflict between the father and son heated up.
建保6年（1218年）息子の忠定を参議に申請して自らは大納言を辞すが、2年後の承久2年には「子強好交衆、又増不孝耳」（『明月記』寛喜元年4月9日条）のため忠定を解官させており、父子の確執は深まる。