Jaxi, you are 100% correct.  "I'm sorry you feel that way" is usually meant as a put down, as if there's something wrong with the person who feels this way; it's often meant as a rejection of responsibility for having damaged someone. 
An actual empathetic response might have been something like "I can imagine it must be pretty stressful to feel people are watching you all the time."
And actually, as anyone who is black or hangs out with black friends around here  knows, they are likely to get stared at from time to time.  Sometimes with hostility, but not nearly as bad as I remember from the '50s; more from idle curiosity  than hostility and sometimes even with genuine friendliness .  At least that's my impression and I could be wrong. 
 As our community becomes more diverse people of color will not seem so "unusual" around here.  And I hope the nation's current tension over race relations does not pull our community back to our less openminded past.