Black swans, and the problem with prediction markets
Thursday, July 12th, 2007“Who knows what’s going to happen?/Lottery or car crash/Or you join a cult.” - Bjork, “Possibly Maybe”
I’m reading “The Black Swan”, the new book by Nassim Nicholas Taleb, whose “Fooled by Randomness” I read a few months ago and really liked. I didn’t think there was much more for him to say on the topic of uncertainty, but this book proves that wrong: in fact, there’s quite a bit more to say. Whereas the first one focused on human psychology and all the various ways we fool ourselves into thinking we can predict the future, this one takes a more mathematical tone, explaining why the future is inherently unpredictable. This is a very big statement: after all, maybe the only reason that we can’t predict the future very well is that each of us is cursed with our inherent biases, and limited information. If that were true, then if you could aggregate everyone’s thoughts, using, say, a prediction market, you’d have a good chance of getting at the truth.
Prediction markets: 2004-era buzzword, and the inspiration for my own Betocracy site. It’s far from a dead concept, with a site like Media Predict, launched two months ago, which is designed to help media companies figure out how well their movie, music and book properties will sell. And yes, Betocracy is still operational, though in all honesty I’ve lost interest in it; and so, apparently, has the world. (No need for sympathy, please! It was an important learning experience, I think.)
Anyway, the “holy grail”, to anyone who’s been interested in prediction markets, is James Surowiecki’s 2003 book “The Wisdom of Crowds”, the book which directly inspired me, and which I still have a high opinion of (though I may have to rethink some of my praise). Surowiecki captured many people’s imaginations with his examples of large groups making uncanny predictions. There was the first such demonstration, in which a crowd at an 1800’s fair guessed the weight of an enormous ox to within a pound or two. There are horse-race crowds, who collectively have odds-making abilities that are nearly unbeatable. And more recently, there are election-prediction markets, that have consistenly beaten the polls in predicting election results. So, to extrapolate, asks the book (and many people), why can’t we use prediction markets as an all-around forecasting tool? For movie grosses, say, or flu outbreaks, or terrorist strikes?
Taleb doesn’t directly talk about prediction markets, though he does talk about capital markets, which are just a more established version of the same thing. But his logic can be easily applied. All of these things have something in common: the weight of an ox (okay, that’s really an observation and not a prediction, but you could phrase it as some sort of prediction), sporting events, political elections. Taleb says that they all fall within the world of what he calls “Mediocristan”, which is not a comment on their quality but rather on the nature of their probabilities. If you plotted the possible outcomes of any of these, they’d all end up in a nice bell curve graph, where, once you get outside of a rather narrow range of possibilities near the center, the probability of an outcome declines dramatically. The chance of a U.S. presidential candidate winning anything more than 65% of the vote, for instance, is rather small; more than 80%, nearly impossible. Similarly, if you ran the same set of horses against one another over and over, the times for each horse would be fairly similar from one run to the next - for a horse to suddenly double or halve its usual racing speed is unheard of.
Most of real life, on the other hand, according to Taleb, takes place in what he calls “Extremistan”. There, there’s no nice trailing-off around the center. Things like personal income, product success, and the severity of wars all fall into this category. For every person who makes a certain amount of money, for instance, there’s a very real chance that someone else will be making twice that much, and someone else ten times as much, regardless of what that original number was. Things that happen in Extremistan are much more unpredictable for just that reason. That’s why the prediction market Hollywood Stock Exchange, which gets headlines for predicting Oscar winners, fails spectacularly when it comes to guessing box office revenues (and there’s a link I wish I had read before starting Betocracy; though who knows if it would have had any effect on me at the time.)
There’s a mathematical explanation for the difference between the two “worlds” of Mediocristan and Extremistan, and it has to do with conditional vs. independent probabilities. In the Mediocristan world of sports, elections, etc., all the factors going into the final outcome are fairly independent of one another: the number of points a team scores in the first half of a game doesn’t really affect the number of points they score in the second; whether a person votes for a certain candidate doesn’t affect whether their neighbor will vote for that candidate. Thus, for a result to be significantly different from expectations, many things would have to go right (or wrong) independently - enough to make such a result all but impossible. On the other hand, in Extremistan, every event affects every subsequent event. If a book sells a million copies, bookstores begin displaying it prominently; the author gets invited on talk shows to plug it, etc: selling the next million becomes a much easier proposition. Similarly with the price of a stock, or the success of a website, or really most of the other interesting questions in life. On the negative side, events like wars can easily snowball as well. Taleb notes that before World War I, which is a classic case of a small event mushrooming completely out of control, stock markets in Europe were doing good business - no one had any inkling of the grand tragedy that was just about to befall them.
So there’s a mathematical basis for explaining why the systems that do so well in predicting certain outcomes will fail at all the rest. And why we’ll have to remain in the dark about the really important issues, like maybe the most pressing unknown of the day: whether Iran will “push the button”, to quote a contemporary Israeli song. And it goes without saying that, in retrospect, that might not even be the thing we need to worry about the most.
UPDATE: Sorry I was too harsh about the Hollywood Stock Exchange - “fails spectacularly” was sort of a spur-of-the-moment phrase on my part, and probably unwarranted.
UPDATE 2: Oh, damn, Taleb linked to this post! I wouldn’t have predicted that.
290c