Why should I try recreational drugs? I see very little benefit and a noticeable risk.
Conversely, I fly airplanes for fun and have a wonderful time doing it. Should I look down on people who have never tried that, the way you stupidly look down on people who don't agree with you on drugs?
Don't you find that in some way, you _do_ "look down" on those who don't at least try flying recreationally?
I know many of the sub-cultures I'm involved in do exactly that. Motorcycle riders. Musicians. DJs. Burners. HackerSpace members. Artists. All of those circles of friends of mine have some level of scorn/pity for people not participating in their specific thing.
You _shouldn't_ take drugs if it doesn't interest you to do so. Anyone who tries to pressure you into going so should be avoided. Just the same as you'd consider someone trying to pressure people into flying, as opposed to offering an opportunity and backing off if there's no interest shown - would be being a jerk.
There's a huge gulf between the sort of mild sense of superiority a lot of hobbies confer on those who partake, and saying that you couldn't trust anybody who hasn't tried your hobby.
Conversely, I fly airplanes for fun and have a wonderful time doing it. Should I look down on people who have never tried that, the way you stupidly look down on people who don't agree with you on drugs?