I think the notion largely goes back to "PCU" in which one fellow implores the other fellow not to "be that guy."
The bigger question here is one of relative douchery: Are you a larger douche for wearing the shirt of a band you enjoy to a collection of people who also enjoy that band, or is the larger douchery wrapped up in setting a limit on your life based on a movie from a couple dozen decades ago?
Don't let Jeremy Piven boss you around.
That said: I did not wear a Beastie Boys shirt to the Sasquatch music festival. I wore (in this order) my "What we need more of is SCIENCE" shirt, my MC Front-a-Lot shirt, and then I finished with my Social Distortion shirt.
The Social Distortion shirt is my biggest shirt oops. I wore it to a Campfire USA event. All havin' the little kids running around, while I have a picture of a chained up pin-up lady on my chest. The special ladyfriend gave me such a scold! Even now, when I think to wear it to bars I pull my hand away just before grabbing it, as though the shirt might be white-hot, or produce some kind of behavioral modification shocks. "OH NO! We dare not even gaze at the FORBIDDEN SHIRT!"