Philip Tetlock spent 30 years exploring what makes someone a superforecaster (not the ones you see on TV). In his book Superforecasting, he distills his work into several commandments on how we can improve our judgment without any complex algorithm.
1. Triage: Focus on questions where your hard work is likely to pay off
Tetlock wrote “Don’t waste time either on easy questions (where simple rules of thumb can get you close to the right answer) or on impenetrable questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.”
In investing, you want to spend a majority of your time on the 2 to 3 most important variables that determine 80% of the outcome. Why only 2 to 3 variables? Because success rate falls exponentially when an investment has many moving parts. If an investment idea requires 6 variables to work out to be profitable, the success rate quickly plummets to 53% (0.9^6) even when you’re confident that each variable carries a 90% likelihood of happening.
You also want to avoid questions that are too hard to answer. Let’s say you are researching an O&G company. You figure that the oil price is an important variable but what’s your probability of predicting it correctly 3 to 5 years out? That’s an impenetrable question where you won’t do much better than flipping a coin. Instead, another important yet solvable variable is to find out the cost structure of the company and its sustainability. Given that most O&G players are a price taker, the most efficient company sitting at the bottom of the variable-cost curve will reap the most benefits regardless of the oil price level.
2. Breaking seemingly intractable problems into tractable sub-problems
George Polya, a Hungarian mathematician, once advice “If you can’t solve a problem, then there is an easier problem you can solve: find it.”
Is this stock a buy? That is a big hairy problem. One way to solve it is to break it into sub-problems. Every time you break down a big problem, you’re asking “What needs to happen for this to be true?”. In this case, what needs to happen to qualify this stock as a buy? Let’s say you break it into 3 sub-problems:
3. Strike the right balance between inside and outside views.
The inside view looks at specific circumstances surrounding a situation, while the outside view tied circumstances to an appropriate reference class by asking what happens when others encounter something similar?
It is easy to engross in the specifics surrounding a company, such as the growth story and extrapolate based on what we see. But a good forecast also requires the outside view or base prediction. If a company is forecasted to grow 15% annually over the next 5 years (inside view), while its competitors have a growth rate closer to 9% (outside view), then you need a good reason to explain the difference. More often than not, your revised growth rate will likely fall somewhere in between those numbers. We shouldn’t dismiss the inside view entirely, of course. What’s unique to a company i.e culture, could sometimes turn out to be a good predictor of success. But at the same time, the outside view tames overconfidence and avoid base rate neglect so we don’t miss the forest for the trees. As a rule of thumb, start from the outside view before adjust towards the inside view to avoid anchoring bias and overestimation.
4. Strike the right balance between under- and overreacting to evidence.
If a company reported two consecutive lower quarterly earnings due to a challenging environment, should you take that as a canary in the coal mine for more bad news or just a temporary setback that is unlikely to affect the company’s future? There’s no easy answer because there is a lot of unknown. But one way to weigh evidence appropriately is to use Bayes rules to update belief. Tetlock wrote, “The best forecasters tend to be incremental belief updaters, often moving from probabilities of, say, 0.4 to 0.35 or from 0.6 to 0.65, yet superforecasters also know how to jump, to move their probability estimates fast in response to diagnostic signals.”
To apply Bayes rules to this scenario, you need 3 things:
5. Look for the clashing causal forces at work in each problem
For every good argument, there is always a counterargument. Your role as an investor is to synthesize all the positive and negative evidence and make a decision. If a company won a multi-million dollar contract which improves the future prospect of the business, also think about the other side of the argument on what can be negative: prolonged delay in project completion, cost overrun, contract termination, unattractive return etc. If you believe investing using margin is risky, think about circumstances when it can be useful. If you’re a true believer in owning only quality companies in a concentrated portfolio, consider circumstances when this strategy becomes unfavorable, and vice versa if you prefer owning cheap mediocre stocks. This can be applied to any general argument from operating leverage, company debts, profit margin to situation specific to an investment.
The lesson here is you have to have the ability to hold contradictory information in your head to improve forecasting skills. All great investors are great synthesizer—the ability to blend both sides of things into a single worldview and make decisions accordingly.
6. Strive to distinguish as many degrees of doubt as the problem permits but no more
If you say “I’m fairly confident that this company can increase its profit over the next 3 years.”, what does fairly mean? If the company downgraded their earnings 6 months later, would you still remember how you feel about the investment 6 months prior? And do you reduce your fairly to likely, possibly or something else? Using words is not informative, not to mention you probably have no idea what that word means as time goes by. Even if you wrote it down.
A better way is to translate them into numeric probabilities. If you think the probability of an increase in profit over the next 3 years is 75%, that would be the prior probability. And when you receive the news of a profit downgrade, you can use Bayes rule to revise and update your belief. Thinking in probability also allow you to think in expected value and alternative outcomes, which are the first step to think clearly and improve accuracy.
7. Strike the right balance between under- and overconfidence, between prudence and decisiveness
Good forecasting requires a good calibration between confidence and accuracy. If confidence gets ahead of accuracy, you risk going into an investment that falls outside your circle of competence, where you think you know when in fact you don’t know; If accuracy outruns confidence, you risk missing out on great opportunities by pondering too long. A good calibration means know when to sit down and do nothing as well as knowing when to swing for the fences. A good grasp on the edge of your circle of competence is critical to improving calibration, and an easy way to do that is to read a lot.
8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases
Learn from your mistakes is a great way to improve forecasting. Mistakes are not necessarily ones you’ve made a loss; you can make a big gain through pure luck. So it is critical to look at all of your investments and focus on the thought process rather than the outcome. This underlines the importance of having a decision journal so you can refer back to your decision-making process. It also prevents hindsight bias.
9. Master the error-balancing bicycle
The best way to become a better forecaster is to practice it. But first, you need to create a feedback loop so you can learn and improve as you go along. A good feedback loop should have these criteria:
1. Triage: Focus on questions where your hard work is likely to pay off
Tetlock wrote “Don’t waste time either on easy questions (where simple rules of thumb can get you close to the right answer) or on impenetrable questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.”
In investing, you want to spend a majority of your time on the 2 to 3 most important variables that determine 80% of the outcome. Why only 2 to 3 variables? Because success rate falls exponentially when an investment has many moving parts. If an investment idea requires 6 variables to work out to be profitable, the success rate quickly plummets to 53% (0.9^6) even when you’re confident that each variable carries a 90% likelihood of happening.
You also want to avoid questions that are too hard to answer. Let’s say you are researching an O&G company. You figure that the oil price is an important variable but what’s your probability of predicting it correctly 3 to 5 years out? That’s an impenetrable question where you won’t do much better than flipping a coin. Instead, another important yet solvable variable is to find out the cost structure of the company and its sustainability. Given that most O&G players are a price taker, the most efficient company sitting at the bottom of the variable-cost curve will reap the most benefits regardless of the oil price level.
2. Breaking seemingly intractable problems into tractable sub-problems
George Polya, a Hungarian mathematician, once advice “If you can’t solve a problem, then there is an easier problem you can solve: find it.”
Is this stock a buy? That is a big hairy problem. One way to solve it is to break it into sub-problems. Every time you break down a big problem, you’re asking “What needs to happen for this to be true?”. In this case, what needs to happen to qualify this stock as a buy? Let’s say you break it into 3 sub-problems:
- Market valuation
- Business characteristics
- Portfolio hurdle rate
3. Strike the right balance between inside and outside views.
The inside view looks at specific circumstances surrounding a situation, while the outside view tied circumstances to an appropriate reference class by asking what happens when others encounter something similar?
It is easy to engross in the specifics surrounding a company, such as the growth story and extrapolate based on what we see. But a good forecast also requires the outside view or base prediction. If a company is forecasted to grow 15% annually over the next 5 years (inside view), while its competitors have a growth rate closer to 9% (outside view), then you need a good reason to explain the difference. More often than not, your revised growth rate will likely fall somewhere in between those numbers. We shouldn’t dismiss the inside view entirely, of course. What’s unique to a company i.e culture, could sometimes turn out to be a good predictor of success. But at the same time, the outside view tames overconfidence and avoid base rate neglect so we don’t miss the forest for the trees. As a rule of thumb, start from the outside view before adjust towards the inside view to avoid anchoring bias and overestimation.
4. Strike the right balance between under- and overreacting to evidence.
If a company reported two consecutive lower quarterly earnings due to a challenging environment, should you take that as a canary in the coal mine for more bad news or just a temporary setback that is unlikely to affect the company’s future? There’s no easy answer because there is a lot of unknown. But one way to weigh evidence appropriately is to use Bayes rules to update belief. Tetlock wrote, “The best forecasters tend to be incremental belief updaters, often moving from probabilities of, say, 0.4 to 0.35 or from 0.6 to 0.65, yet superforecasters also know how to jump, to move their probability estimates fast in response to diagnostic signals.”
To apply Bayes rules to this scenario, you need 3 things:
- Prior probability. This is the initial estimate on what probability would you assign to the business’s earning power being impaired before the company reported 2 consecutive lower earnings. Let’s say given the company’s strong track record, you estimate there is a 15% probability. Put it another way, you believe there is an 85% probability the business is going to do well in the long-term. (Hence you bought the shares).
- Next, we need the probability that the hypothesis being true—2 consecutive lower quarterly earnings is indeed a sign that the earning power is impaired. Let’s say based on your understanding of the company’s competitive position in the market, you think it is unlikely. But at the same time, the fact that the company has never reported any lower earning prior to this event is a cause of concern. You place the probability for the hypothesis to be true at 50%.
- Last, the probability that the hypothesis is false. If the earning power remains intact, what could explain those lower earnings? It could be a shift in product mix to lower margin items, lower volume sold due to supply constraints and so on. Let’s say you put this at 28% after considering all alternative outcomes.
5. Look for the clashing causal forces at work in each problem
For every good argument, there is always a counterargument. Your role as an investor is to synthesize all the positive and negative evidence and make a decision. If a company won a multi-million dollar contract which improves the future prospect of the business, also think about the other side of the argument on what can be negative: prolonged delay in project completion, cost overrun, contract termination, unattractive return etc. If you believe investing using margin is risky, think about circumstances when it can be useful. If you’re a true believer in owning only quality companies in a concentrated portfolio, consider circumstances when this strategy becomes unfavorable, and vice versa if you prefer owning cheap mediocre stocks. This can be applied to any general argument from operating leverage, company debts, profit margin to situation specific to an investment.
The lesson here is you have to have the ability to hold contradictory information in your head to improve forecasting skills. All great investors are great synthesizer—the ability to blend both sides of things into a single worldview and make decisions accordingly.
6. Strive to distinguish as many degrees of doubt as the problem permits but no more
If you say “I’m fairly confident that this company can increase its profit over the next 3 years.”, what does fairly mean? If the company downgraded their earnings 6 months later, would you still remember how you feel about the investment 6 months prior? And do you reduce your fairly to likely, possibly or something else? Using words is not informative, not to mention you probably have no idea what that word means as time goes by. Even if you wrote it down.
A better way is to translate them into numeric probabilities. If you think the probability of an increase in profit over the next 3 years is 75%, that would be the prior probability. And when you receive the news of a profit downgrade, you can use Bayes rule to revise and update your belief. Thinking in probability also allow you to think in expected value and alternative outcomes, which are the first step to think clearly and improve accuracy.
7. Strike the right balance between under- and overconfidence, between prudence and decisiveness
Good forecasting requires a good calibration between confidence and accuracy. If confidence gets ahead of accuracy, you risk going into an investment that falls outside your circle of competence, where you think you know when in fact you don’t know; If accuracy outruns confidence, you risk missing out on great opportunities by pondering too long. A good calibration means know when to sit down and do nothing as well as knowing when to swing for the fences. A good grasp on the edge of your circle of competence is critical to improving calibration, and an easy way to do that is to read a lot.
8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases
Learn from your mistakes is a great way to improve forecasting. Mistakes are not necessarily ones you’ve made a loss; you can make a big gain through pure luck. So it is critical to look at all of your investments and focus on the thought process rather than the outcome. This underlines the importance of having a decision journal so you can refer back to your decision-making process. It also prevents hindsight bias.
9. Master the error-balancing bicycle
The best way to become a better forecaster is to practice it. But first, you need to create a feedback loop so you can learn and improve as you go along. A good feedback loop should have these criteria:
- Time-specific. If you make a forecast without a timeframe, your forecast is untestable. Set a date to score your forecast.
- Quantifiable. Translate words into numbers whenever possible. “83% probability” instead of “very likely”; “22% ROC” instead of “high ROC”. Always ask if your forecast will leave any room for interpretation or arguments. As a general rule, it is more useful to forecast business performance than the share price.
- Measurable. You can’t improve what you don’t measure. Brier score is a good tool to measure the accuracy of your forecast. The lower the score, the more accurate you are. Superforecaster tends to have a Brier score of around 0-0.25.
