Analyzing Successful Predictions

We are consistently reading about predictions from weather forecasts, to the stock market, to the cost heating oil this winter. In every major field there is someone that is making a prediction and trying to convince us based on past historical trends, alignment of the planets, or that caterpillars have hairy coats this fall. Statistical analysis of these forecasts has shown at best having about 50 percent chance of being correct while others rank with winning the lottery.

Most mutual fund managers don’t perform any better or as well as the SP index [1]. In fact most correct predictions are correct only by luck or the original prediction was incorrectly stated in the beginning.[2]

The World Weather Organization reports that the global weather forecasting budget to be approximately $4B per year. Yet long range weather forecasts are at best 50% correct. What has been shown is as the lag time between the prediction and the event decreases the greater the accuracy of the prediction. [2]

So what makes one prediction better than the next? In what sectors do predictions tend to be forecasters of the future? This question is an age old question that has been pondered over the decades. As with any prediction it should be viewed with caution and judged on the merit the prediction is founded on. If prediction is based on the readings of sacrificed goat entails it should be viewed with much caution. If it is based on hard scientific research then maybe more thought should be given it. In “The Fortune Sellers” the author lists 5 questions that should be asked: [2]

1. Is the forecast based on hard science?
2. How sound are the methods?
3. Does the forecaster have credible credentials?
4. Does the forecaster have a proven track record?
5. To what extent is my belief in a particular forecast influenced by my personal beliefs and wishful thinking?

Even after reviewing these questions it is still important to ask yourself if current events have been factored into the prediction. They may be predicting a mild winter so the expected price of heating oil may be forecasted to be low, however if there is a reported oil refinery closure due to a fire, the end resulted may be high heating oil prices. The longer lag time on a predication, the high the probability that an external event may impact the prediction.

These questions should apply to any prediction, whether it appears in a local news paper or is part of a strategic report from Gartner that your company paid for. Be leery of predictions based on “group think”. Just because a group of analysts’ support a given predictions, doesn’t mean that it is correct. They may all read the same market research report. Look for supporting data from different research sources.

In an attempt to test this approach I have selected a couple of predictions that have been proven with time to see if this model is supported. The two predictions are Moore’s Law and Elaine Garzarelli prediction of the 1987 stock market crash.

If we look at the circumstances around Gordon Moore’s famous prediction of circuit density doubling every 18 months we will find an interesting story. He had been approached by electronics magazine to predict semiconductor progress over the next 10 years. The present state of the industry was 30 circuits per chip and he knew the prototypes he presently had in the lab where at 60 circuits. Taking an educated guess he expected the density in 10 years to be 600,000 circuits, a thousand-fold increase which turned out to very precise.[3] There appears to be good amount of luck in this forecast, but it was extrapolated from the best data available at that time. If we apply the 5 rules we previously discussed, steps 1 & 2 fail, step 3 is met, step 4 would probably fail (insufficient information), and step five would probably not be a weighting factor. So to summarize 1 pass, 3 fails, and 1 unknown, however the prediction was true and still holds true after 42 years. In the case of Gordon Moore’s prediction, he was an expert in his field, he did have credible data to extrapolate from, and he had a good share of luck.

For the next case to analyze I draw from a case study cited in “The Fortune Tellers”. Elaine Garzarelli had developed a model to predict the direction of the market as either being bullish or bearish. After tracking the stock market for 17 years her model was showing 92% bearish, the most negative it had ever indicated. She went on Cable News Network’s “Money Line” predicting an “imminent collapse of the market”. Four days later the DOW crashed. She was immediately declared a “Market Guru”. However, she was able to predict the market crash, but was not able to predict its recover. For the preceding 9 years after her successful prediction she was only able to correctly predict the direction of the market 5 times out of 13 calls. If we look at the steps we can say that the model had been developed and tweaked for 17 years. It was based on accepted scientific methods, but not necessarily proven theorems. So we would have to say that step 1 failed, steps 2 & 3 passed, step 4 is unknown, and step 5 would depend on your feeling about the health of the market. So again the 5 step evaluation model sends an unsure message if to accept or reject the prediction.

To summarize, the five step model failed to accept the two predictions even though the predictions where true. The first prediction was not only correct, but has been able to stand the test of time of the past 42 years. The second prediction was correct, but could not be sustained. If these are typical successful predictions, then the model should be amended to the following conditions*:

1. The forecasters must have credible credentials
2. They must have current data
3. The lag between the prediction and time to occur must be short
4. Is the forecast base on proven theorems
5. How sound are the methods
6. Does the forecaster have a proven track record

*Note: A larger sampling would be required to prove or disprove the model with any level of confidence.


Reference:

[1] Richard A. Ippolito, “Efficiency with Costly Informational A Study of Mutual Funds in the Period 145-1964,” Journal of Finance 23(2) (1968): 389.

[2] William A. Sherden, “The Fortune Sellers: The Big Business of Buying and Selling Predictions” John Wiley & Sons, (1998)

[3] InfoWorld CTO Forum, San Francisco, (2002), http://www.intel.com/pressroom/archive/speeches/gelsinger20020409.htm