Wolfensohn was born on 1 December 1933 in Sydney, New South Wales, Australia. His father Hyman, known as Bill, was born in London to Austrian-Jewish immigrants, while his mother Dora was born in Belgium to Polish parents. His father was a "highly intelligent but failed businessman" who had previously worked for the Rothschild banking family. Wolfensohn's parents arrived in Australia in 1928. He was named after James Armand de Rothschild, his father's former employer, whose birthday he shared. His mother sang with Australian radio and gave him piano lessons, instilling in him a love of the symphony. In high school, he was noted to have taken part in operas, including playing female roles in Gilbert and Sullivan operettas.