He was appointed to the Director of Kyoto by Yoritomo who fully trusted him, and also built a large network within the Imperial Court and the bakufu, coming to hold political power by marrying his daughters off to Yoshitsune KUJO (the son of Kanezane KUJO) and Kintsune SAIONJI, and also by becoming an in-law of Kanemasa KAZANIN and Michichika TSUCHIMIKADO.
さらに頼朝から京都守護に任命されるなど、頼朝からの信任は厚く、九条良経（九条兼実の子）や西園寺公経を娘と娶わせ、花山院兼雅や土御門通親とも姻戚関係を結ぶなど、朝廷や幕府に広いネットワークを形成し、有力な実力者にのし上がった。