I once panhandled.
It was in  the  early1960s when I was a grad student, and we went to SF for the summer for fun but couldn't find much in jobs...my donut shop work wasn't making it.
So one night I went up by North Beach, stood on the corner and  asked the first man for some change......he reached in his pocket, pulled out a couple of quarters, and handed them to me with such a look of contempt I about shriveled up.....that was it, never again.
Moral of story, seems to me many of these pan handlers have lost any sense of pride, of self respect, of expectation of a better or more useful life.  When I ask some healthy looking young guy why he doesn't go and at least try to find a job, they mostly  just seem to go blank, like  that's not in their vision of the universe, they've given up.  I find that really sad.