Dutt returned from Madras to Calcutta in February 1856, after his father's death (in 1855), abandoning his wife and four children in Madras. No records of his divorce from Rebecca or remarriage have been found. In 1858, he was joined there by a 22-year old of French extraction, Emelia Henrietta Sophie White, the daughter of his colleague at the Madras Male Orphan Asylum. They had two sons, Frederick Michael Milton (23 July 1861 – 11 June 1875) and Albert Napoleon (1869 – 22 August 1909), and a daughter, Henrietta Elizabeth Sermista (1859 – 15 February 1879). A fourth child was stillborn. Their relationship lasted until the end of his life, Henrietta pre-deceasing him by three days, on 26 June 1873.