It was around this time that Izac's popularity began to wane for his liberal positions and opposition to reactionary efforts by Congress. His voting record came under scrutiny by the "San Diego Union", in part for his opposition to permanent status for the House Un-American Activities Committee and for his protests against the deportation of prominent West Coast union leader Harry Bridges. Izac subsequently won re-election by smaller margins. He had won his 1938 re-election campaign by 20,000 votes in a vote of 58,806 to 38,333, but was re-elected by 4,373 votes in 1940, and 777 votes in 1942 in a close election that came down to the count of absentee ballots. Izac despised Adolf Hitler but nonetheless never advocated for entering the war against him, and his isolationist views were increasingly unpopular in favor of national unity to support the war effort, especially after the attack on Pearl Harbor. In 1942 following redistricting, Izac shifted to the 23rd congressional district, and Republican Carl Hinshaw replaced him in the 20th congressional district.