You know, that's a good point.  The evil done to the southern blacks was largely physical, as they emerged with their morals and faith intact, while the slavery warped the morals of many southern whites, and scarred them psychology to this day.