During the Second World War, he served as a lieutenant in an entertainment unit of the South African Army, and subsequently took up acting as a career. He moved to the United Kingdom in December 1946, financed by his service gratuity. According to rumour, James had an affair with the daughter of an important member of Johannesburg society; his decision to move away from South Africa was partly because he was "advised" to leave due to the possible controversy. Initially, he worked in repertory before being spotted for the nascent British post-war film industry.