His family's fortunes declined when his father Hajime was the family head, so he became a priest and lived in the Rokujo Kawara no In, which his great-grandfather Toru built.
父適（はじめ）の頃から家運が衰え、出家して曾祖父融（とおる）が造営した六条河原院（かわらのいん）に住んだ。