Racism is a separate issue from treason despite the treason being based on the desire to save slavery.  Andrew Johnson liked neither former slaves nor their former owners. He loved his own former class of poor whites, the dupes of the time. He could not envision that poor whites and former slaves had more in common with each other than they had with white elites. Things have not changed much.