He came from a very important and rich family. His father, Aaron Hart, was member of the British forces, and a well-known businessman in Canada. His brothers, Moses and Benjamin, were important businessmen at Trois-Rivières and Montreal. His cousin, Henry Judah, later became a member of the "Legislative Assembly of the Province of Canada".