 Gordon was born in Wilkes County, North Carolina to Nathaniel and Sarah Gwyn Gordon.James B. Gordon was a distant cousin of Confederate General John Brown Gordon.After studying at Emory and Henry College in Emory, Virginia, he returned to Wilkes County and became a successful mercantile businessman and a prominent member of the community.