After the Pearl Harbor attack, the Japanese occupied the International Settlement. In early 1943 they began interning Allied civilians. Ballard was sent to the Lunghua Civilian Assembly Center with his parents and younger sister. He spent over two years in the internment camp. His family lived in a small area in G block, a two-story residence for 40 families. He went to school in the camp.