The county had been developed along the waterways for cotton plantations, dependent on enslaved laborers, many of whom were transported to the South in the domestic slave trade during the antebellum years. Cotton brought a high return from local and international markets, making large planters wealthy. At the time of the 1860 federal census, Brooks County had a white population of 3,067, a Free people of color population of 2, and a slave population of 3,282. The Atlantic and Gulf Railroad reached Quitman, the county seat, on October 23, 1860.