Cannon was born in Guilford County, North Carolina, and in 1840 moved with his parents to Annapolis, Indiana, about 30 miles north of Terre Haute. He was the elder of two sons of Gulielma (née Hollingsworth) and Horace Franklin Cannon, a country doctor. Horace Cannon drowned on August 7, 1851 when Joseph was fifteen years old as he tried to reach a sick patient by crossing Sugar Creek. Young Cannon took charge of the family farm. His brother William would become a successful banker and realtor.