On 21 August 1952, James married Valerie Elizabeth Patsy Assan (born 1928), an actress who used Ashton as her stage name. They had a son, Steve James, born in 1954, who is now a music producer, and a daughter, Sue, who is a television producer. During the latter part of their marriage, they lived in a house partly designed by James himself, called Delaford Park, situated in Iver, Buckinghamshire, a location close enough to Pinewood Studios to allow him to return home for lunch while filming. During his marriage to Valerie, he had a well-publicised affair with "Carry On" co-star Barbara Windsor lasting 3 years. The affair was dramatised in the 1998 stage-play "Cleo, Camping, Emmanuelle and Dick" and its 2000 television adaptation "Cor, Blimey!". James's obsession with Windsor was such that it was rumoured that her then husband Ronnie Knight had all of James's furniture rearranged at home as a subtle threat and, on another occasion, that he had put an axe in James's floor, but close friends of the time, including Vince Powell and William G. Stewart, have dismissed the suggestions.