It feels like dudes are just ugly and animal-looking to me. But I'll admit that all minds are onions.
This is definitely a part of it. See, I couldn't find a woman (or a man, for that matter) physically attractive if I knew she was seriously ugly on the inside - like, a hardcore racist or a homophobe or a Jews for Jesus canvasser. On the one hand, she'd still look the same to me, but seeing what made me find her hot in the first place would just further remind me how horrible she is deep down.
Edward Norton isn't the best specimen, but he's by all accounts a pretty decent guy and I like him as an actor; he's got this weird soft-spokenness about him that prevents him from being himself on camera for lots of money like 90% of actors (and 99% of actresses). He's big on the environment and human rights and taught English in Japan before he was famous, which says something about his intelligence outside of acting.
I guess for me part of it is that most of attraction is personality-based, which makes judging who is cute to me based on looks a mug's game. But I think this is true for most people at least to some extent. Most men could not have sex with Ann Coulter without feeling dirty afterwards, just like most men could not have sex with a normal man without feeling dirty afterwards.
Basically looks aren't everything. They're important, which is why I find women attractive who have a lot less going on upstairs than it'd take for me to find a man attractive - I'm aesthetically attracted to the female form - but personality can trump them.
If it absolutely can't, if some part of looks are so important that you absolutely can't find someone attractive without them, there are usually some issues at play there. At the very least, upbringing takes some part in this: after all, the ancient Greeks considered you weird as a grown man if you didn't mostly prefer pubescent boys, and thought that men who exclusively went for women (whose purpose was to keep the house and produce more boys) were weird, almost the modern/Western equivalent of 'fruity'.
I'm trying to picture how this assertion was made. Self reporting (seems unlikely)? Observation of nude dudes watching The Game? That has a subtext to it that seems comical. Some sort of electronic cuff the researchers talked average sports watching fellows into wearing while chilling at home that was later downloaded and compared to time-stamped journals of activity? That would seem most objective, if not for the fact that erections seem to be self perpetuating once you're aware of any action going on down there, which (I would think) a cuff would sort of accentuate.

Sculpted Adonises at play
I'll admit it's a little dubious, but I still find it pretty interesting - and based on how hard a lot of sports enthusiasts I've met overcompensate with the straight thing, it rings sort of true. I'll look for the link; I don't recall their methodology either.
But it makes a sort of sense: they're basically watching dudes who are the best at one of their favorite things do it as hard as possible, adrenaline and all. It's pretty easy to get your 'excited' wires crossed - we are animals, after all - and if you add an upbringing that encourages you to take that man-lust and bury it real deep instead of expressing it and leaving it well enough alone, well, you usually wind up with a situation.