Inouye was born in 1924 in the Territory of Hawaii. He was the son of a father who was born in Japan and a mother whose parents were born in Japan. He was a "Nisei", which means that he was a second generation Japanese-American.