This week I witnessed two presentations by New York University professor Bruce Bueno de Mesquita, a noted political scientist and futurist. His latest book is called The Predictioneer’s Game and claims to use complex game theory to predict political and economic events. He claims his predictions have been 90% accurate, which is why the CIA and others pay close attention to them. He never told us exactly how his models work, except to say several times that they are “very complex.”
As he spoke about the world and proffered future events — Iran will not develop a nuclear weapon, Colombia and Venezuela will not go to war — it was clear that Mesquita is a smart man who knows a great deal about international politics. He also is a talented public speaker.
Yet something bothered me. During his talk my buddy Justin Rockefeller (also in the room) texted me, “What do you think?” I replied, “Entertaining but I’m deeply skeptical. Nassim Taleb would have a field day.” He replied, “Yep.”
In Fooled by Randomness, Taleb talks about “why human beings are so prone to mistake dumb luck for consummate skill.” The idea of survivorship bias figures prominently in Taleb’s work. If I play the lottery 100 times, and I win every time, this doesn’t necessarily mean I’ve developed the skill to regularly win the lottery. Someone has to win. We ignore those who lose.
Mesquita wasted no breath acknowledging the improbability of developing a mathematical model that reliably predicts world events. He offered no qualifications on how much of his success might be due to luck and randomness. Instead, he dished predictions with breathtaking arrogance and certainty, returning again and again to his 90% success rate. He never once elucidated how this 90% number got calculated (I predict Hugo Chavez will die, eventually) despite it being the source of his credibility.
It took only a few minutes of Googling to find long, detailed criticisms of Mesquita. You’d think such a body of criticisms would temper his certitude. No sirree.
Surprisingly, people in the room seemed taken by Mesquita. He had spot-on observations, to be sure, about the selfishness of Mother Theresa or the self-interest of the Iranian regime. But why didn’t more people eye his prediction schemes with skepticism?
Charisma, for one. He was entertaining. We are so often bored by speakers that anyone who can dance a gig on stage gets a vote for “keeping us awake.” I hear that. But it is a dangerous heuristic — equating entertainment with substance.
He wasn’t just charismatic; he was an “expert.” In general, people are too deferential to experts who make predictions inasmuch as experts sometimes do no better than laypeople at predicting. In particular, people are too deferential to experts toting fancy credentials (such as a PhD), even if those credentials have little to do with the topic at hand. We should be especially skeptical of experts who feel a need to remind us again and again of their expertise, as Mequita did.
Look, Mequita is more than qualified to riff on current affairs and the state of the world. He has written a dozen plus books on international politics and economics and he is more knowledgeable than me on most of the issues he discussed. But it’s unimpressive to commentate under the vague guise of “complex game theory.” Do analysis and make assertions as an informed pundit, like everyone else, not some mathematically gifted prophet whose models only the CIA understands.
Bottom Line: Nassim Taleb’s popularity notwithstanding, there are still intellectuals who take their knowledge too seriously, confuse luck and randomness with skill and foresight, and pontificate with inappropriate levels of certainty in an uncertain, complex world.
###
Seth Roberts takes Elizabeth Kolbert to task for putting faith in scientists over science. Seth writes about “practically all science journalists”:
They take the consensus view too seriously. In case after case — so many that it’s hard not to draw sweeping conclusions — the consensus view about difficult topics is more fragile than an outsider would ever guess. It’s not necessarily wrong, just less certain.
Nice post Ben. In my experience, people that are unable to explain their (often very complex) work in simple terms that everyone can understand are using complexity & jargon to hide work that’s not very sound.
The one potential counterargument to keep in mind is the possibility that mesquita coulllld do a genuinely good job of explaining himself, but that he doesn’t want to reveal his secrets because they make him millions of dollars.
That’s the claim, anyways. He doesn’t publish academic papers on his method, analogous to opting for the trade secret rather than the patent. He could decide to go this route either because he’s a fraud or because he’s the real deal; neither can be eliminated from our perspective.
If he IS for real, this might be the smartest way for him to handle things. But we are right to level on him the skepticism he signs up for by making such a choice.
Sounds just like Hari Seldon of Asimov’s “Foundation” series. See http://en.wikipedia.org/wiki/Psychohistory_%28fictional%29
Saw him at LSE a while ago. Have the book sitting in my too read pile. But this is very similar to how I felt. The bravado far outpaced the substance of his talk.
Someone actually asked him a question about Taleb. I’m kind of annoyed that I can’t remember the question or answer.
I didn’t get this though:
“Do analysis and make assertions as an informed pundit, like everyone else, not some mathematically gifted prophet whose models only the CIA understands.”
Are you criticizing just his persona, or the actual methods here?
His persona and methods as I understand them. He doesn't talk very much
about his methods, though. That's part of the problem.
Huh? Am I missing something, or hasn’t he been published papers on his method for over twenty years?
Most of his work has been in political science in general; predictions are
just one part of what he does, and the least refereed and published. His
computer model is not public, it's proprietary, so there's no way to check
that. And my post was in response to his public presentation, not an
exhaustive evaluation of his written work.
This is going to come out differently from how I mean it (I swear, legitimate question, not being dismissive), but how much time have you actually spent looking at the methodology/math behind his work?
Not that I have. But I’m kind of looking forward to that if it’s there in Predictioneer. Which I doubt it will be.
Game theory is sort of an interest of mine, so I’m curious to find out what the meat here is. At this point, I’m sort of reserving judgment. I mean, a position at NYU and endorsements from Nobel Prize economists don’t often come easily.
As for his persona, I’m in complete agreement.
Very little. I could do more. Though the real meat of the matter is the
computer model, which he won't make public.
Oh. Saw your post above. That basically answers my question.
I’m not surprised that there’s an “expert” who’s claiming this stuff. What I’m surprised about is how much press he’s been getting and people aren’t calling him out more vigorously. I made this Delicious note when I first read about him in NYT Magazine in August: A highly suspect modeling system to predict human-decision outcomes. The telltale line: “Bueno de Mesquita does not express his forecasts in probabilistic terms; he says an event will transpire or it won’t.” Unfortunately, the NYT’s Clive Thompson does not express nearly enough suspicion despite the obvious red flags.
I’m glad you’re drawing attention to his suspect system. The mainstream media are being fooled.
It seems like he’s using agent-based models based on game-theoretic rule sets. That, in itself, is increasingly common in the social sciences. He seems to also add unique parameters to each specific model gathered from interviews of domain experts, not just through derived assumptions. It’s not a bad system for trying to analyze and understand complex human social systems. It is not a prediction system, however, and saying so is what he should definitely be criticized for.
I allowed myself a smile after reading this post.
Most advocates of game theory, even its card carrying members admit openly that the complex models often lead to expected outcomes merely by accident. Or they occur at random as you say. Now if some *expert* comes up and kind of guarantees future outcome, then he is my idea of God – which clearly is formless, stateless and occupies the wider expanse of the universe that Taleb contrived as *randomness*.
I have been told that one of the reasons the astronomers of the world cooperate is the fact that there is no one nation from which the entire sphere of the sky can be seen. Perhaps there is in that fact a parable for game theorists, whose predictive horizons are all too often limited by national /political proclivities – that is loosely evident from the attribution to Mother Teresa as selfish. I live in India and I know her work. This man is nuts.
Reminds me of the work I did back in undergrad with the Brunswick Lens Model. Basically it comes back to Rumsfeld’s famous line: “As we know, there are known knowns. There are things we know we know.
We also know there are known unknowns.
That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don’t know
We don’t know.”
Basically you can choose to act on less information in the chance that you’ll be wrong (Bush administration). Or you can choose to wait until you have more information at the expense of something like 9/11 happening (Clinton administration). So although this guy is 90% accurate supposedly, it still leaves that 10% known and unknown unknowns. It’s a philosophical administrative decision as to where on the lens you want to be…each with their own advantages and disadvantages.
Following up on your quoting Taleb, yet finding that the group seemed taken in by Mesquite, in Robert Burton’s On Being Certain, he describes our often non-rational need to think we know something to be true in spite of evidence that we are not.
On a similar theme, all anecdotally, I see my people chucking out answers to a question they don’t know the answer to. It’s like humans have to have an answer to everything (even when we think we are wrong!) and being handed an explanation on a plate from an expert is an easy way to get ‘an’ answer.
Survivorship bias may figure prominently in Bueno de Mesquita’s predictions, and perhaps “ground squirrels and vampires” use voodoo decision theory as well.
Who am I to say?
But being hired repeatedly by the CIA is not an endorsement that should sway rational people, and Clive Thompson’s phrase “enthralled by the idea of rendering the messy business of politics and history into precise, logical equations” made me laugh.
I want some of what these guys are smoking.
Mesquita is obviously not a good mathematician. One of his published models is wrong — in fact very wrong. The embarrassing reference to n! in his TED lecture is really pathetic.
Regarding the 90% success rate: This figure is provided by a (former) CIA analyst and should not be taken seriously. For example, Mesquita regards his prediction on the future of Hong Kong as a “success” while experts on this topic dispute this claim.
The extensive empirical tests reported on in
D. Scott Bennett and Allan C. Stam
The Behavioral Origins of War
University of Michigan Press, 2004
do not paint such a rosy picture.
In any case, the main difficulty in dealing with his claims is that we do not have access to the models that he uses, so there is no way to independently verify the validity of the predictions.
More on this at
http://decision-making.moshe-online.com/criticism_of_bueno_de_mesquita.html
Ben,
John Gall’s Systemantics would back up your criticism.
Some may interpret your comments as a beat-down on all of his material, which is surely not the case. His work with George Downs on how authoritarian regimes keep democracy from developing is superb: (http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=mesquita+downs+coordination+goods)
hi ben, didn’t like the presentation either. Daniela showed me your post. If you like more on mistakes generated from randmness, you may like the book “The Drunkard’s Walk: How Randomness Rules Our Lives” fom Leonard Mlodinow. I really liked the book. Cheers, Diego
I don’t know what kind of presentation this was, exactly, but I take it that it was not to a bunch of other academics. In my experience, the kinds of presentations academics give for an academic audience versus an audience of laypeople (even highly intelligent, well-read, and well-informed laypeople) is very different. I don’t know anything about the work he does, but I would guess he kept saying it was “very complex” because he had no intention of explaining the actual numbers to an audience he assumes won’t understand. It’s crappy, but that’s how intellectuals are.
This guy is clearly a whack-job, but he’s getting attention because he represents certainty in very uncertain times. A lot of people seek refuge in experts because they want to feel like “it’s all under control”. It’s not just a harmless rouse, I think it’s actually pretty insidious stuff. Mesquita’s implicit worldview is that we’re all determined creatures with no free will. Doesn’t that fly in the face of any possible morality?