Hart's devotion to his work played havoc with his emotional life. As a young man, in spite of his Methodist conscience, he had bouts of promiscuity. In 1857 he took a Chinese concubine, Ayaou, with whom he had three children and for whom he developed genuine affection and respect. After becoming Inspector General at the end of 1863 one of his resolutions was to set a good example to his staff. For him this included parting with Ayaou (who seems to have been still in the south) and finding a respectable British wife. In December 1864 he visited Hong Kong and Canton, and it seems that while there, he made generous arrangements for Ayaou and made plans for the removal of the children to Britain. This was delayed as it seems in saying farewell he had also made her pregnant with their third child. Also one of the key persons whose help he needed was away. As he recorded in his diary for 15 January 1865 "I had to leave undone the private business I was most anxious to have got arranged". In May 1866 he arrived in Britain for his first leave. It seems likely the three children, (Anna, Herbert and Arthur Hart), travelled on the same boat with his Chinese steward, and his lawyer immediately found them a foster home.