His grandfather Tomotsuna, who was forgiven as well, became a priest and came to live in retirement in Oha of the Shimotsuke Province (present day Oha, Mashiko-machi, Haga-gun, Tochigi Prefecture), it is believed that at this time Yoritsuna succeeded to head of the Utsunomiya family.
同じく赦免された祖父・朝綱は出家して下野国尾羽（現・栃木県芳賀郡益子町尾羽）にて隠居生活を送ることとなり、このとき頼綱が宇都宮家を継いだものと考えられる。