His wealthy wife Florence (1845–1878) had previously been married, in 1864, to Alexander Louis Ricardo, son of John Ricardo MP, but had been separated from him because of his affairs and violent alcoholism. She herself had had an extramarital affair with the much older Dr James Manby Gully, a fashionable society doctor who was also married at the time, and she had fallen out of favour with her family and society. Ricardo died in 1871 and Florence married Charles, a respected up-and-coming barrister, on 7 December 1875, terminating her affair with Gully.