Later, its ownership was transferred to Imperial Prince Shigeakira, the husband of Tadahira's daughter, but after the prince died, it was owned by FUJIWARA no Kaneie, a grandchild of Tadahira, and Kaneie reformed the building in 969 to make it his residence.
後に忠平の娘婿重明親王に譲られるが、その没後は忠平の孫である藤原兼家の所有となり、安和2年（969年）この屋敷を改築した兼家はここを自分の屋敷と定めた。