In 1203 Yoriie became seriously ill, was confined to Shuzen-ji Temple in Izu province by his maternal grandfather Tokimasa and died in 1204, the year after his younger brother MINAMOTO no Sanetomo assumed the position of Kamakura-dono/shogun.
1203年、重病に陥った頼家は、外祖父時政の手により伊豆の修善寺へ幽閉され、弟の源実朝が次の鎌倉殿・将軍位に就くと、翌1204年に死亡した。