The song is about alienation and being an outsider, and Morrison may have addressed the song both to the hippie culture, to outsiders in general or to users of drugs such as LSD, or both.
The song deals with a sense of isolation and belonging to the periphery, and Morrison could be referring to hippies, general outsiders, drug users like those who use LSD, or perhaps multiple categories simultaneously.