Hart was born at Three Mile Creek, near Wangaratta, Victoria, in 1859. His parents, Richard Hart and Bridget Young had come from Ireland. They had other children, Richard 'Dick' Hart (Jr.), Hugh Hart, Thomas Hart, Nicholas Hart, Esther 'Ettie' Hart, Jane Hart and Winifred Hart.