Interesting interview with Philip Tetlock by John Brockman. Excerpt

Let me say something about how dangerous it is to draw strong inferences about accuracy from isolated episodes. Imagine, for example, that Silver had been wrong and that Romney had become President. And let’s say his prediction had been a 0.8 probability two weeks prior to the election that made Romney President. You can imagine what would have happened to his credibility. It would have cratered. People would have concluded that, yes, his Republican detractors were right, that he was essentially an Obama hack, and he wasn’t a real scientist. That’s, of course, nonsense. When you say there’s a .8 probability there’s 20 percent chance that something else could happen. And it should reduce your confidence somewhat in him, but you shouldn’t abandon him totally. There’s a disciplined Bayesian belief adjustment process that’s appropriate in response to mis-calibrated forecasts.

What we see instead is overreactions. Silver would either be a fool if he’d gotten it wrong or he’s a god if he gets it right. He’s neither a fool nor a god. He’s a thoughtful data analyst who knows how to work carefully through lots of detailed data and aggregate them in sophisticated ways and get a bit of a predictive edge over many, but not all of his competitors. Because there are other aggregators out there who are doing as well or maybe even a little bit better, but their methodologies are quite strikingly similar and they’re relying on a variant of the wisdom of the crowd, which is aggregation. They’re pooling a lot of diverse bits of information and they’re trying to give more weight to those bits of information that have a good historical track record of having been accurate. It’s a weighted averaging kind of process essentially and that’s a good strategy.