Joe was born at Woolshed, near Beechworth, in November 1856. His father, Patrick Byrne, had come from County Carlow, Ireland, in 1849, to join his father, Joseph, who had come to Australia as a convict in 1834. Joe's mother was an Irish lady from Galway. The family moved to Sebastopol, a small gold mining town, about two kilometres farther up the valley. Joe learned to speak Chinese from the Chinese miners who lived in a small camp near his house. He was educated at a small Roman Catholic school at Woolshed.