Following the divorce from his first wife in 1926, Sir James Dunn married Irene Clarice Richards, the former wife of Francis Archibald Kelhead Douglas, 11th Marquess of Queensberry. The 1928 death of his daughter Mona Dunn from his first marriage sent him into a state of depression and for a time he thought about cutting back on his business activities and returning home to his native New Brunswick. However, in 1929, his new wife gave birth to a daughter, Anne, who lifted his spirits a great deal. Anne Dunn grew up to be an artist, studying under Henry Moore and in Paris, France with Fernand Léger and at the Académie Julian. Nevertheless, Dunn and his second wife divorced in 1942 following which he married his long-time personal secretary, Marcia Anastasia Christoforides. Thirty-six years his junior, "Christofor" as she was nicknamed, was very bright, intensely loyal, and a devoted partner. James Dunn would seek her input on most every business matter for the rest of his life.