In 1203, his father Zenjo was killed due to the conflict between his nephew, who was the second shogun, MINAMOTO no Yoriie, but with the help of his grandfather on his mother's side, Tokimasa HOJO, and aunt Masako HOJO, he was able to escape complicity and live in seclusion in Ano-sho, the Province of Suruga, his father's property inheritance.
建仁3年（1203年）、父の全成は甥で二代将軍の源頼家と対立して殺害されるが、この時は母方の祖父北条時政や伯母の北条政子の尽力もあって連座を免れ、父の遺領である駿河国阿野荘に隠棲した。