Now let's imagine your story if you were a person of color. How much harder do you think your grandparents and their parents would have had it?

No matter how tough and hard-working your parents and grandparents were, they would have had to work twice as hard with fewer opportunities if they were of color. There weren't explicitly racist policies in place against them, and at no point were your ancestors slaves. "Slavery" or "racist laws" aren't things that ever stood in the way of the generations that brought you to where you are. 

That's what white privilege means, and though the impacts of those policies are not your fault you inevitably benefit from them. All anyone is asking you to do is acknowledge that. Consider those things when you think about our current policies. 

No white person says "my grandparents were slaves". Really try to think what that might mean for someone. Listen to people of color, their experience is not your own. We cannot simply erase the past.