It was racist because the Japanese driver represented the WWII era Japanese Imperial military to Frei.  Would he have felt "uncomfortable" if the Indy500 winner was a naturalized American citizen from Japan?  My guess is "yes."