Your 4% number diminishes the enormity of the slavery issue. The 1860 census shows that in the states that would soon secede from the Union, an average of more than 32% of white families owned slaves. Some states had far more slave owners (46% in South Carolina, 49% in Mississippi) while some had far less (20% in Arkansas). Slaves comprised 13% of the U.S. population.