In 1940, Inoue became commander of the Imperial Japanese Navy Aviation Bureau, and submitted his thesis for a radical restructuring of the Imperial Japanese Navy to Naval Minister Koshirō Oikawa early in 1941. He was highly critical of the Navy's shipbuilding programme, with its emphasis on battleships over aircraft carriers. Inoue was given command of the IJN Fourth Fleet later the same year, based out of Truk. He was thus in command of Japanese naval forces during the Battle of Guam and Battle of Wake Island. He subsequently relocated his headquarters to Rabaul for Operation Mo, intended to occupy Port Moresby. However, after the Japanese defeat at the Battle of the Coral Sea in May 1942, he was relieved of his command in October, and returned to Japan to become commander of the Imperial Japanese Naval Academy. He became Vice Minister of the Navy in the closing stages of World War II, was promoted to full admiral on May 15, 1945 (one of the last two promotions made to this rank), and officially retired on October 15 of the same year.