I am glad that you said this. I came to the same conclusion earlier today, when I had this conversation with a guy I know, and he basically said something like "god, I just... I hate poor people, you know?" (he said this in a "joking" voice, but he was kidding himself if he thought he didn't mean it.)
That basically told me that my impulses are WRONG and I should suppress them rather than doing the "it's just my first impulse, I know maybe it's bad, haha" thing.
at the same time, there are a hell of a lot of things that Most People want to do that are not good at all. obviously a line needs to be drawn somewhere.
I'm pretty liberal and white and whatever, but I can only guilt myself about my lifestyle so much. I can only take so much "how dare you buy used clothes, you can afford new ones!" "how dare you think that you know what's best for other people!" "How dare you feel better about yourself for donating to a charity!" "How dare you think that punk rock is/was authentic, do you know that blah blah" before I am all liberal-guilted out.
I mean, it's one thing to say "it's a rich person's law" or whatever, but dying of lung cancer or emphysema or obesity-related health problems is incredibly unpleasant. I am not at all in favor of public-smoking bans, but, give me a break.
I realize it's a free country, and of course it's people's own choice. But the thread of "who the hell are you to tell me how to live my life!" gets old when the answer is "a motherfuck*ng doctor, I have spent my life studying the way the human body works. the human brain is not equipped to understand long-term consequences in a rational way, you're fuck*ng killing yourself."
Some people are trained or educated to understand certain kinds of things, and the whole anti-intellectual thing where people are snobs/elitists/jerks if they cite their knowledge gets real old.