Joseph Emerson Brown was born on April 15, 1821, in Pickens County, South Carolina, to Mackey Brown and Sally (Rice) Brown. At a young age he moved with his family to Union County, Georgia. In 1840, he decided to leave the farm and seek an education. With the help of his younger brother James and his father's plow horse, Brown drove a yoke of oxen on a 125-mile trek to an academy near Anderson, South Carolina. There Brown traded the oxen for eight months' board and lodging.