In the first part of this review we spoke of how people make predictions all the time. The Signal and the Noise: Why So Many Predictions Fail- but Some Don’t has more to offer than some generic input on predictions, it has a path for us to walk about the models and statistics we can use to make better predictions.

All Models are Wrong but Some are Useful

Statistician George Box famously said, “All models are wrong, but some are useful.” The models that we use to process our world are inherently wrong. Every map inherently leaves out details that shouldn’t be important – but might be. We try to simplify our world in ways that are useful and that our feeble brains can process. Models allow us to simplify our world.

Rules of thumb – or heuristics – allow a simple reduction of a complex problem or system. In this reduction, they are, as Box said, wrong. They do not and cannot account for everything. However, at the same time, they can be useful.

The balance between underfitting and overfitting data is in creating a model that’s more useful and less wrong.

Quantifying Risks

Financial services, including investments and insurance, are tools that humans have designed to make our lives better. The question is, making whose lives better? Insurance provides a service in a world where we’re disconnected and we don’t have a community mentality where we support each other. In Hutterite communities – which is a division of the Anabaptist movement like the Amish and Mennonites – all property is owned in community. In a large enough community, the loss of one barn or one building is absorbed through the community. However, that level of community support doesn’t exist in many places in the modern world.

Insurance provides an alternative relief for catastrophic losses. If you lose a house or a barn or something of high value, insurance can provide a replacement. To do this, insurance providers must assess risk. That is, they must forecast their risk. The good news is that insurance providers can write many insurance policies with an expected risk and see how close they get to calculating the actual risk.

Starting with a break-even point, the insurance company can then add their desired profit. For those people and organizations that believe there’s good value in the insurance, their assessment of risk or willingness to accept risk is such that they want the insurance buy it. Given that people are more impacted by loss than by reward, it’s no wonder that insurance is a booming business. (See Thinking, Fast and Slow for more on the perceived impact of loss.)

The focus then becomes on the ability of the insurance company to quantify their risk. The more accurately they can do this, and take reasonable returns, the more policies they can sell and the more money they can make. Risk, however, is difficult to quantify, ignoring for the moment black swan events (see The Black Swan for more). You still must first separate the signal from the noise. You must be able to tell what is the rate of naturally-occurring events, and which events are just normal random deviations from this pattern.

Next, the distribution of the randomness must be assessed. What’s the probability that the outcome will fall outside of the model? When referring to the stock markets, John Maynard Keynes said, “The market can stay irrational longer than you can stay solvent.” The same applies to insurance: you must be able to weather the impact of a major disaster and still stay solvent. Whether it’s a particularly difficult tornado season or a very bad placement of a hurricane, the perceived degree of randomness matters.

Then you have the black swan events, the events that you’ve never seen before. These are the events that some say should never happen. However, many of the times when this has been used, the risk was well-known and discussed. A hurricane hitting New Orleans was predicted and even at some level prepared for – though admittedly not prepared for well enough. This is not a true black swan, or completely unknown and unpredictable event. It and other purported black swan events were, in fact, predicted in the data.

When predicting risks, you have the known risks and the unknown risks. The black swan idea focuses on the unknown risks, those for which there’s no data that can be used to predict the possibility. However, when we look closely, many of these risks are predictable – we just choose to ignore them, because they’re unpleasant. The known risks – or, more precisely, the knowable risks – are the ones that we accept as a part of the model. The real problem comes in when we believe we’ve got a risk covered, but, in reality, we’ve substantially misrepresented it.

Earthquakes and Terrorist Attacks

Insurance can cover the threat of earthquakes and the threat of terrorist attacks. However, how can we predict the frequency and severity of both? It turns out that both obey a similar pattern. Though most people are familiar with Edward Richter’s scale for earthquake intensity, few realize that it’s an exponential scale. That is, the difference in magnitude between a 4.1 and a 5.1 earthquake isn’t 25% more energy released, it’s 10 times more. Thus, the difference between a magnitude 6.1 and an 8.1 earthquake is 100 times more energy released.

This simple base-10 power rule is an elegant way to describe the release of energy that can be dramatically different. What’s more striking is that there is a line that moves from the frequency of smaller earthquakes to larger ones on this scale. It forecasts several large earthquakes for a given period of time. Of all the energy released in all the earthquakes from 1906 to 2005, just three large earthquakes—the Chilean earthquake of 1960, the Alaskan earthquake of 1964, and the Great Sumatra Earthquake of 2004—accounted for almost half the total energy release of all earthquakes in the world. They don’t happen frequently, but these earthquakes make sense when you look at the forecast along the line of frequency of smaller earthquakes.

Strikingly, terrorist attacks follow the same power law. The severity rises as frequency decreases. The 9/11 attacks are predictable with the larger framework of terrorism in general. There will be, from time to time, larger terrorist attacks. While the specific vector from which an attack will come or the specific fault line will cause an earthquake will be unknown, we know that there’s a deceasing frequency of large events.

Industrial and Computer Revolutions

If you were to try to map the gross domestic product by person, the per-person output would move imperceptibly up over the long history of civilization, right up to the point of the industrial revolution when something changes. Instead of all of us struggling to survive, we started to produce more value each year.

Suddenly, we could harness the power of steam and mechanization to improve our lives and the lives of those we care about. We were no longer reduced to living in one-room houses as large, extended families and began to have a level of escape from the threat of death. (See The Organized Mind for more on the changes in our living conditions.) Suddenly, we had margin in our lives to pursue further timesaving tools and techniques. We invested some of our spare capacity into making our lives in the future better – and it paid off.

Our ability to generate data increased as our prosperity did. We moved from practical, material advances to an advance in our ability to capture and process data with the computer revolution. After a brief dip in overall productivity, we started leveraging our new-found computer tools to create even more value.

Now the problem isn’t capturing data. The Internet of Things (IoT) threatens to create mountains of data. The problem isn’t processing capacity. Moore’s law suggests the processing capacity of an individual microchip doubles roughly every 18 months. While this pattern (it’s more of a pattern and less of a law) is not holding as neatly as it was, processing capacity far outstrips our capacity to leverage it. The problem isn’t data and processing. The problem is our ability to identify and create the right models to process the information with.

Peer Reviewed Paucity

The gold standard for a research article is a peer-reviewed journal. The idea is that if you can get your research published in a peer reviewed journal, then it should be good. The idea is, however, false. John Loannidis published a controversial article “Why Most Published Research Findings Are False,” which shared how research articles are often wrong. This finding was confirmed by Bayer Laboratories when they discovered they could not replicate two-thirds of the findings.

Speaking as someone who has a published peer-reviewed journal article, the reviews are primarily for specificity and secondarily for clarity. The findings – unless you make an obvious statistical error – can’t be easily verified. While I have done thousands of pages of technical editing over the years where I would verify the author’s work, I could test their statements easily. For the most part, being a technical editor means verifying that what the author is saying isn’t false and making sure that the code they were writing would compile and run.

However, I did make a big error once. We were working on a book that was being converted from Visual Basic to Visual C++. The book was about developing in Visual Basic and how Visual Basic can be used with Office via Visual Basic for Applications. There was a section in the introduction where search and replace done by the author said that there was Visual C++ for Applications. Without anything to verify, and since the book was working on a beta of the software for which limited information was available, I let it go without a thought. The problem is that there is no Visual C++ for Applications. I should have caught it. I should have noticed that it wasn’t something that made sense, but I didn’t.

Because the ability to validate wasn’t easy – I couldn’t just copy code and run a program – I failed to validate the information. Peer-reviewed journals are much the same thing. It’s not easy to replicate experimental conditions. Even if you could replicate experimental conditions, you’re likely to not get exactly the same results. So, consequently, reviewers don’t try to replicate the results, and that means we don’t really know whether the results can be replicated – particularly, using the factors that the researcher specifies.

On Foxes and Hedgehogs

There’s a running debate on whether you should be either a fox – that is, know a little about many things – or a hedgehog – that is know a lot about one thing. Many places like Peak tell of the advantages of focused work on one thing . The Art of Learning follows this pattern in sharing Josh Waitzkin’s rise to both chess and martial arts. However, when we look at books on creativity and innovation like Creative Confidence, The Medici Effect, and The Innovator’s DNA, the answer is the opposite. You’re encouraged to take a bite out of life’s sampler platter – rather than roasting a whole cow.

When it comes to making predictions, foxes with their broad experiences have a definite advantage. They seem to be able to consider multiple approaches to the forecasting problem and look for challenges that the hedgehogs can’t see. I don’t believe that the ability to accurately forecast is a reason to choose one strategy over another – but it’s interesting. Foxes seem to be able to see the world more broadly than the hedgehogs.

The Danger of a Lack of Understanding

There’s plenty of blame to go around for the financial meltdown of 2008. There’s the enforcement of the Community Reinvestment Act (CRA) and the development of derivatives. (I covered correlation and causation and the impact on the meltdown in my review of The Halo Effect.) The problem that started with some bad home loans ended with bankruptcies as financial services firms created derivatives from the mortgages.

These complicated instruments were validated with ratings agencies, but were sufficiently complex that many of the buyers didn’t understand what they were buying. This is always a bad sign. When you don’t understand what you’re buying, you end up relying on third parties to ensure that your purchase is a good one – and when they fail, the world comes falling down, with you left holding the bag.

The truth is that there is always risk in any prediction. Any attempt to see if there’s going to profit or loss in the future is necessarily filled with risk. We can’t believe anyone that says that there is no risk.

Bayes Theorem

I’m not statistician. However, I can follow a simple, iterative formula to continue to refine my estimates. It’s Bayes theorem, and it can be simplified to:

You can use the theorem over and over again as you get more evidence and information. Ultimately, it allows you to refine your estimates as you learn more information. It is, however, important to consider the challenge of anchoring, as discussed in Thinking, Fast and Slow and How to Measure Anything.

The Numbers Do Not Speak for Themselves

Despite the popular saying, the numbers do not, and never do, speak for themselves. We’re required to apply meaning to the numbers and to speak for them. Whatever we do, however we react, we need to understand that it’s our insights that we’re applying to the data. If we apply our tools well, we’ll get valuable information. If we apply our tools poorly, we’ll get information without value. Perhaps if you have a chance to read, you’ll be able to separate The Signal and the Noise.

People make predictions all the time. They predict that their team will win the Super Bowl, or they’ll win the lottery. These predictions are based on little more than hope. The Signal and the Noise: Why So Many Predictions Fail- but Some Don’t seeks to set us on the right path to understanding what we can learn from data, what we can infer from data, and what we can’t. By looking at the power and weaknesses of statistics, including both using the wrong model and supplying bad data, we can see how statistics has the power to improve our lives through productive forecasts and predictions.

In this part of the two-part review, we’ll look at predictions.

Forecasts and Predictions

Sometimes in our rush to be amazed at something, we simplify the questions we ask. We fail to recognize that our brain has simplified the thing that we’re trying to sort out (see Thinking, Fast and Slow for more on substitution). In the case of looking into the future, what we really want is prediction, and what statistics gives us most frequently is a forecast. Forecasts necessarily have a certain amount of error and involve statistical relationships. Forecasts become predictions when they become specific and precise.

Each day when we look at the weather, what we want is a soothsayer to predict what the weather will be like. However, what they offer us is a forecast based on models that result in a chance of rain somewhere between zero and 100%. We look at economists and seek the answer about whether we’ll make more money next year – or not. We want to know whether a risky investment will be worth it. However, economists and meteorologists are subject to the same rules as any other statistician.

While it’s true that statistics can predict – as long as we’re using this in a general sense of the word – events that are to happen in the future, there must always be some level of uncertainty as to whether the event will happen – or not. Predictions are just an attempt to refine forecasts into specific, tangible probable outcomes. Sometimes that process is successful but often it is not.

Falsifiable by Prediction

Karl Popper suggested that every forecast should be falsifiable via prediction. To test a model, you needed to be able to make some sort of a prediction with it that then could be proven false. In this way, you could create a test to ensure that your model was accurate and useful. A model that doesn’t forecast appropriately and that you can’t make a prediction from doesn’t do much good.

Everything Regresses to the Mean

One thing about statistics is that it can tell you with relative authority things you want to know with less precision than is useful. Statisticians can forecast the economy but not predict whether you will get a raise or not. The Black Swan artfully points out the challenges of statistics and modeling when the sampling size is insufficient. Until you’ve seen a black swan, you’ve not sampled enough to make the statistical models work. Until you’ve sampled enough, the noise will dramatically pull your results askew.

With large sample sizes, everything regresses to the mean. We no longer see the outlier, even as something that is distinct and that does happen, rather it gets lost in the law of averages. Tragic events like 9/11 are never forecast using the wrong model. They’re not perceived as possible if they’re averaged into the data. It’s like the proverbial statistician drowning in a river that is, on average, only 3 feet deep – all the depth of the data was averaged out.

Right Model, Right Results

Perhaps the most difficult challenge when working with data is not the data collection process. Collecting data is tedious and needs to be done with meticulous attention to detail; however, it’s not necessarily imaginative, creative, or insightful. It’s the work that must be done to get to the magic moment when the right model is uncovered for working with data. Though statisticians have ways of evaluating different models for their ability to predict the data, they must see some inherent signal in the noise.

For a long time, we couldn’t find planets outside of our solar system. One day, someone identified a detection model – that is, they discovered a theory for the strange oscillations in the light frequency from distant stars. The theory proposed that super-massive planets in close orbit were causing the star to move. This created a Doppler effect with the light from the star causing what we perceived as light frequency oscillations. Consensus coalesced, and the scientific community agreed that this was indeed what was happening. We had found the first extra-solar planet. Almost immediately, we found nearly a dozen more.

These super-massive planets were hiding in the data we already had. We had already captured and recorded the data to indicate the presence of other planets, but we didn’t have a model to process the data that we had to allow us to understand it.

There were plenty of ideas, thoughts, theories, and models which were tried to explain the light variations, but it wasn’t until the consideration of a super-massive planet that we settled on a model that was right.

The Failure of Predictions

We got lucky finding extra-solar planets. The right idea at the right time. It was a good fit model. It wasn’t a specific prediction. With predictions, our luck is very, very poor. The old joke goes, “Economists have predicted nine of the last six recessions.” They predicted a recession where none happened. Earthquakes and other disastrous cataclysmic events are predicted with startling frequency. It seems that everyone has some prediction of something. Sometimes the predictions are harmless enough, like whose team will win the super bowl. Sometimes the consequences are much direr.

Disease

When you think in systems, delays are a very bad thing. Delays make it harder for the system to react to a change in circumstances. In the case of the SR-71 Blackbird, the delays in a mechanical system made engine unstarts a regular occurrence. Reduce the delay with electronic controls and the unstart problem is dramatically reduced. (See The Complete Book of the SR-71 Blackbird for more.) In the creation of vaccines, the delay is great. To scale up production and get enough doses for the country, it takes six months.

What makes the vaccination “game” worse is that vaccines are designed to target specific viral strains. If the virus mutates, the hard work of creating the vaccine may be wasted, as it may become ineffective at protecting against the new strain. Each year, the vaccine makers attempt to predict which variations of influenza will be the most challenging. They start cooking up batches of vaccines to combat the most virulent.

What happens, however, when you get noise in the identification of the influenza that will be the most impactful? From 1918 to 1920, swine flu afflicted roughly one-third of humanity and killed over 50 million. So when there was an apparent outbreak of a strain of it at Fort Dix, who can blame President Ford for encouraging the vaccine industry to create a vaccine for it and encouraging every American to do their part in preventing the spread of the disease by getting vaccinated – and hopefully increasing the herd immunity?

It turns out it was all a bad call. Issues with the vaccine caused Guillain–Barré disease in some. The virus strain turned out to not be that virulent. The noise at Fort Dix that had produced the scare wasn’t a result of the virus’s potential but was instead a result of environmental and cultural factors that allowed the disease to spread at Fort Dix but weren’t generalizable to the population.

SIR

A classic statistical way of modeling diseases is the SIR model, which is an acronym for susceptible, infected, and recovered. The assumption is made that everyone who is recovered is not susceptible again, and everyone has an equal level of susceptibility. This simplified model works relatively OK for measles, but fails to account for natural variations in susceptibility in humans. More importantly, the model fails to account for the connections that we have with each other. It fails to account for how we interact.

Another classic example of disease was cholera in London, but it didn’t seem to have any connections. There was no discernable pattern – that is, until John Snow discovered a connection in the Broad Street well and removed the pump handle. The disease slowly dissipated, as Snow had correctly identified the root cause. However, his job wasn’t easy, because people who were far away from the pump were getting sick. Those who weren’t close to the Broad Street pump had hidden connections. Sometimes they lived near the pump in the past and still used it for their main water source; in other cases, they had relatives close by. The problem with forecasting diseases is the hidden patterns that make it hard to see the root cause. To correctly forecast, we need to find and then use a correct model.

An Inconvenient Truth

It’s an inconvenient truth that, in the decade when An Inconvenient Truth was released, there was no substantial change in temperatures across the planet – in truth, there was an infinitesimal reduction in temperature from 2001 to 2011. However, Gore wasn’t the first to claim that there were problems. In 1968, Paul and Anne Ehrlich wrote The Population Bomb. It was 1974 when Donella Meadows (who also wrote Thinking in Systems), Jorgen Randers, and Dennis Meadows first published Limits to Growth. (It’s still on my reading list.) These books both sought to predict our future – one with which the authors were most concerned. Of course, population is increasing, but it’s far from a bomb, and we’ve not yet reached the feared limits to growth.

These predictions missed what Everett Rogers discovered when working with innovations. In
Diffusion of Innovations, he talks about the breakdown of society created by the introduction of steel axe heads in aboriginal tribes in Australia. They missed the counter-balancing forces that cause us to avoid catastrophe. However, presenting a balanced and well-reasoned point of view isn’t sensational, and therefore doesn’t sell books, nor does it make TV exciting. The McLaughlin Group pundits’ forecasts about political elections are not at all well-reasoned, balanced, or even accurate – but that doesn’t stop people from tuning into what amounts to be a circus performance every week.

So the real inconvenient truth is that our predictions fail. That we overestimate, and we ignore competing forces that attempt to bring a system into balance. In fairness to Gore, the global temperature on a much longer trend seems to be climbing at 1.5 degrees centigrade per year. It’s just that there’s so much noise in the signal of temperatures that it’s hard to see – even over the course of a decade. We need to be concerned, but the sky isn’t falling.

Watching the Weather

If you want to find a prediction that’s guaranteed to be wrong, it’s got to be the weather. The oft quoted remark “What job can you be wrong most of the time and still keep your job?” refers to meteorologists. However, in truth, forecasts are substantially better than they were even a decade ago. They’ve done a startlingly good job of eliminating the problems with the mathematical models that generate weather forecasts. Increases in processing power has made it more possible to create more accurate and more precise forecasts. And they’re still frequently wrong. A wise weatherman goes outside and looks at the sky before going on air to share their predictions, because they know that the computer models can be wrong.

The problem isn’t the model. The problem isn’t our ability to model what will happen with the forces of nature. The problem is in our ability to measure precisely the inputs for the model and the inherent dynamic instability of the systems. It was Lorenz that first started the conversation about the butterfly effect. That is, a butterfly in Brazil can set off a tornado in Texas. That’s a mighty powerful butterfly – or the result of an inherently unstable and dynamic system. A very small change in input has a very large change in output.

As a quick aside, this is where the hash algorithms have their roots. We use hash algorithms to ensure that messages aren’t tampered with. They work by small changes in input resulting in large changes in the output.

The problem with predicting the weather, then, isn’t that we don’t know how to process the signal and arrive at the desired outcome. The problem is that we can’t get a precise enough signal to eliminate all the noise.

Overfitting and Underfitting

In attempts to find the models that perfectly describe the data, we run the risk of two sides of the same coin. On the one hand, we can overfit the data and try to account for every variation in the dataset. Or we can look for mathematical purity and simplicity and ignore the outliers – this is “underfitting.”

“Overfitting” mistakes noise for signal. An attempt is made to account for the randomness of noise inside the signal we’re trying to process. The result is that our ultimate predictions try to copy the same randomness that we saw in our sample data. In other words, we’ve mistaken the noise for the signal and could not eliminate it.

Underfitting, on the opposite side of the coin, is the inability to distinguish the signal in the noise. That is, we ignore data that is real signal, because it looks like noise. In a quest for mathematical simplicity, we ignore data that is inconvenient.

Brené Brown speaks of her scientific approach to shame and vulnerability as grounded theory and the need to fit every single piece of data into the framework. (See The Gifts of Imperfection for more.) When I first read this, it stood in stark contrast to what I saw with scientists ignoring data that didn’t fit their model. It seems like too many scientists are willing to ignore the outliers, because their theory doesn’t explain it. In other words, most scientists, in my experience, tend to underfit the data. They are willing to allow data to slip through their fingers for the elegance of a simpler model. Brown and those who follow the grounded theory approach may be making the opposite error in overfitting their data.

Statistical Models

In the next part of this review, we’ll talk about models and statistics.