There is a theory that he was out of favor with the eldest brother Tokihira, because his wife, MINAMOTO no Junshi, was the daughter (or the adopted daughter by some theories) of Emperor Uda, and because he was on good terms with the Emperor's close advisor SUGAWARA no Michizane, both of whom were in conflict with Tokihira. (It is also possible that Junshi was Michizane's niece; see MINAMOTO no Junshi for the details.)
妻・源順子は宇多天皇の皇女（養女とする説もある）であり、宇多天皇側近であった菅原道真とも親交があった（順子が道真の姪であった可能性がある。詳細は源順子を参照のこと）ために、宇多天皇や道真と対立していた長兄・時平からは疎んじられていたという説がある。