After the death of his first wife Sir James remarried in 1966 to Beatrice Mary Moore Oliphant (1910–2003), widow of Philip Blair Oliphant.
After the passing of his first spouse, Sir James formed a new partnership with Beatrice Mary Moore Oliphant in 1966, who had been married to Philip Blair Oliphant prior to their union.