background circlebackground circlebackground circlebackground circlebackground circlebackground circlebackground circle



People use imprecise words to describe the chance of events all the time — “It’s likely to rain,” or “There’s a real possibility they’ll launch before us,” or “It’s doubtful the nurses will strike.” Not only are such probabilistic terms subjective, but they also can have widely different interpretations. One person’s “pretty likely” is another’s “far from certain.” Our research shows just how broad these gaps in understanding can be and the types of problems that can flow from these differences in interpretation.






We created a fresh survey with a couple of goals in mind. One was to increase the size of the sample, including individuals outside of the intelligence and scientific communities. Another was to see whether we could detect any differences by age or gender or between those who learned English as a primary or secondary language.

Here are the three main lessons from our analysis.

Lesson 1: Use probabilities instead of words to avoid misinterpretation.






Lesson 2: Use structured approaches to set probabilities.




How do you set probabilities intelligently?



Now imagine a jar full of 25 green marbles and 75 blue marbles. You close your eyes and select a marble. If it’s green, you receive $1 million, and if it’s blue, you get nothing. You know you have a one in four chance (25%) to get a green marble and win the money.

Now, which would you prefer to bet on: the launch failure or the draw from the jar?

If you’d go for the jar, that indicates that you think the chance of winning that bet (25%) is greater than the chance of winning the product-failure bet. Therefore, you must believe the likelihood of your competitor’s product launch failing is less than 25%.

In this way, using an objective benchmark helps pinpoint your subjective probability. (To test other levels of probability, just mentally adjust the ratio of green and blue marbles in the jar. With 10 green marbles and 90 blue ones, would you still draw from the jar rather than take the product-failure bet? You must think there’s less than a 10% chance the product won’t launch.)

Lesson 3: Seek feedback to improve your forecasting.

Whether you’re using vague terms or precise numbers to describe probabilities, what you’re really doing is forecasting. If you assert there’s “a real possibility” your competitor’s product will launch, you’re predicting the future. In business and many other fields, being a good forecaster is important and requires practice. But simply making a lot of forecasts isn’t enough: You need feedback. Assigning probabilities provides this by allowing you to keep score of your performance.

Opinion writers and public intellectuals often talk about the future, but typically they don’t express their convictions precisely enough to allow for accurate performance tracking. For example, an analyst might speculate, “Facebook will likely remain the dominant social network for years to come.” It’s difficult to measure the accuracy of this forecast because it is subjective and the probabilistic phrase suggests a wide range of likelihoods. A statement like “There is a 95% probability that Facebook will have more than 2.5 billion monthly users one year from now” is precise and quantifiable. What’s more, the accuracy of the analyst’s forecast can be directly measured, providing feedback on performance.

The best forecasters make lots of precise forecasts and keep track of their performance with a metric such as a Brier score. This type of performance tracking requires predicting a categorical outcome (Facebook will have more than 2.5 billion monthly users) over a specified time period (one year from now) with a specific probability (95%). It’s a tough discipline to master, but necessary for improvement. And the better your forecasts, the better your decisions. A few online resources make the task easier. The Good Judgment Open (founded by Tetlock and other decision scientists) and Metaculus provide questions to practice forecasting. Prediction markets, including PredictIt, allow you to put real money behind your forecasts.

The next time you find yourself stating that a deal or other business outcome is “unlikely” or, alternatively, is “virtually certain,” stop yourself and ask: What percentage chance, in what time period, would I put on this outcome? Frame your prediction that way, and it’ll be clear to both yourself and others where you truly stand.


Have questions or feedback for us? Write to us at
author image
Andrew Mauboussin is a data scientist at Twitter in San Francisco.
author image