Joseph Reid Anderson was born at "Walnut Hill" near Fincastle, the county seat of Botetourt County, Virginia in 1813. The grandson of Scotch-Irish immigrants, he was the son of Colonel William Anderson (1764–1839) and Anne (née Thomas) Anderson. The elder Anderson had served in the American Revolutionary War, and was also a colonel of a Virginia regiment in the War of 1812. Joseph's father was a self-taught engineer and surveyor and was later responsible for the building of the turnpike that is now U.S. Route 220 and (for part of the way) U.S. Route 60 from Fincastle to Covington. Col. Anderson's son was to follow in similar work.