In 1968, at Cambridge, James married Prudence A. "Prue" Shaw, a graduate from Somerville College, Oxford and emeritus reader in Italian studies at University College London and the author of "Reading Dante: From Here to Eternity". James and Shaw had two daughters, one the artist Claerwen James. In April 2012, the Australian Channel Nine programme "A Current Affair" ran an item in which the former model Leanne Edelsten admitted to an eight-year affair with James beginning in 2004. Shaw evicted her husband from the family home following the revelation. Prior to this, for most of his working life, James divided his time between a converted warehouse flat in London and the family home in Cambridge.