Branner was born in the town of New Market, Tennessee, where his father was a merchant. In 1852, the family went to live on a farm given to his father by his grandfather a mile east of Dandridge on the French Broad River. He grew up there and on an adjacent property to which the family moved in 1859. During the Civil War he was very anxious to join the Confederate army, and on two occasions left school for the purpose of enlisting. His age, however, prevented his being accepted; he was only thirteen.