Remember that Joey Scar called the gay wizard a "joke."
And this is not about Silver, really. Maybe his model was better than simply averaging state polls, maybe not. The point is that if the polls had any actual information value - and possibly they didn't! - then the elections results were clear for awhile. If you thought Silver was crap, you were basically saying the polling was crap. That was certainly possible. But if you thought the polling was crap, then you had zero information to base your election outcome prediction on. Except vibrations, I guess.