For the next ten years he was under house arrest, but he was allowed to visit the imperial palace with Emperor Shirakawa's consent, although Udaijin (Minister of the Right) Nakamikado, also as known as FUJIWARA no Munetada, wrote in his diary, "Chuyu-ki" (dated October 23, 1099), 'Minamoto no Asomi (second highest of the eight hereditary titles) is the Emperor's best and bravest bushi.
その後約10年間は閉塞状態であったが、白河天皇の意向で昇殿を許されたが、中御門右大臣・藤原宗忠はその日記『中右記』承徳2年10月23日条に「義家朝臣は天下第一武勇の士なり。