Ballard's father was a chemist at a company called the Calico Printers Association. The company was based in Manchester and made cloth. Ballard was born and raised in the Shanghai International Settlement, an area under foreign control where people "lived an American style of life". He went to the Cathedral School in Shanghai. After the start of the Second Sino-Japanese War, Ballard's family temporarily left their suburban home and rented a house in downtown Shanghai to avoid the shells fired by Chinese and Japanese forces.