After the First Konoe Cabinet was formed in the same year, he gradually retreated from the political stage, and he passed away on November 24, 1940, two months after the formation of Tripartite Military Pact, which he had continued to oppose.
同年の第一次近衛内閣成立以降は次第に政治の表舞台から退き、反対し続けた日独伊三国軍事同盟成立の2ヶ月後の昭和15年（1940年）11月24日に死去した。