Wai was the child of a Native Hawaiian mother and a Chinese father. Growing up, he often surfed with Duke Kahanamoku regarded as the father of surfing. He also surfed with Buster Crabbe who later became an actor. He attended the Punahou School in Honolulu where he earned athletic letters in track, football and baseball. He went to college at the Sacramento Junior College before transferring to UCLA. At UCLA, he was a four sport athlete and graduated in 1939 with a bachelor's degree in Banking and Finance. He intended to work alongside his father in real estate and banking but instead joined the military upon the outbreak of World War II.