Yeah, I've worked in a library and three bookstores on top of a movie rental place and having edited/written for the entertainment section of a paper, so I am well aware of the diaspora that is out there, and the differentiation between sci-fi and fantasy. Generally the two are within the same category on shelves, I suppose because they both draw from a concept of a reality greatly separated from the one we live in now. I genuinely do not like sci-fi. I don't draw from a place where I think it's wrong to like Star Wars, or the Matrix or Asimov or whatever. I, myself, don't like it. And I'm well aware of the fact that that's fairly close to saying something like "I don't like chocolate." By in large, most people like this stuff. They like being removed from the here and now and imagining a far different world (either futuristic or otherworldly), they enjoy imagining possibilities outside what is known.
As for the handjob comparison... I have to disagree. I mean, you could get a single hand job from one guy and discern that you don't like gay sex. You don't have to go all the way or try out a variety of men to figure out that something isn't palatable to you, right? So I can look at the Alien series and get why other people like it, why it is something many people really enjoy, and I can still know it's just not to my tastes.