Conner was born at Slate Springs, in Calhoun County, Mississippi. His father, Robert H. Conner, was a soldier in the Confederate States Army who was wounded several times during the American Civil War. In his final engagement, the Battle of Atlanta, Robert Conner was shot in the head and lost his sight. After the war he was nicknamed "Blind Bob". He learned to gauge the grades of cotton by touch, and became a successful cotton trader. In addition, he began teaching at the Slate Springs Academy. The school had been founded by Fuller Fox in 1872, and several members of the Fox family were on the faculty. Robert Conner met Nancy (Nannie) Hughes Fox when both were teaching at the academy, and they married in 1873.