Simpson's Paradox: And How it Could Kill You!

Simpson’s Paradox is an amazing result that can cause us to make catastrophic errors under certain circumstances. It is so counter-intuitive it can even make us question our ability to think at all when we first hear about it, so we better learn about it right away!

Here is a seemingly simple problem: suppose you have a serious disease called “x syndrome” and you have to decide between going to the Big City Hospital or the Small Country Hospital.

You find some data online showing survival rates for patients with “x syndrome” and the data looks like this:
Big City Hospital – had 1000 patients and 800 survived which is a survival rate of 80%.
Small Country Hospital – had 100 patients and 90 survived which is a survival rate of 90%.

The best decision is obvious, right? It’s so easy to see what the best decision is that we hardly have to think about it at all. At the Small Country Hospital 90% of patients survive and at the Big City Hospital only 80% of the patients survive. We should get to the Small Country Hospital as fast as we can!

That seems so obviously like the right decision it appears it could not possibly be wrong. But in our example it is wrong. Terribly wrong. Maybe fatally wrong.

If we had dug deeper we would have found data that breaks down the cases of patients with “x syndrome” into two groups: serious cases and simple cases. If we look at the data for serious cases and simple cases separately we suddenly see results that look very different from what we first saw and we realize that our first hasty decision was wrong.

Big City Hospital
900 serious cases – 705 survived for a survival rate of 78.3%
100 simple cases – 95 survived for a survival rate of 95%

Small Country Hospital
10 serious cases – 7 survived for a survival rate of 70%.
90 simple cases – 83 survived for a survival rate of 92.2%

We see now that the Big City Hospital has better results with serious cases: 78.3% survive, while at the Small Country Hospital only 70% survive.

Then we see that the Big City Hospital also does better with simple cases: 95% survive, while at the Small Country Hospital only 92.2% survive.

So we originally decided to go to the Small Country Hospital because they had a higher overall survival rate than the Big City Hospital. But now we see that, whether we consider serious cases or simple cases, the Big City Hospital has better survival rates for BOTH, so obviously we should go there, which contradicts our first decision.

How can this be?

This paradox is based on the fact that we first considered only what happens at the different hospitals because we thought that was the only important factor. In the second analysis we considered what happens at the different hospitals and ALSO what happens with different severities of the disease. It turns out that severity of the disease is also an important factor in determining survival rate. By ignoring that factor in our first analysis we drew a false conclusion that seemed so true it could not possibly be wrong!

People with serious cases of “x syndrome” have a lower survival rate, and the Big City Hospital has many more patients with serious cases, and this is what drags the overall survival rate for that hospital down.

People with simple cases of “x syndrome” have a higher survival rate, and the Small Country Hospital has many more patients with simple cases, and this what pushes the overall survival rate of that hospital up.

This paradox comes up in many areas of life so always keep it in mind when looking at data about groups. Don’t assume that one factor explains the outcome you see. Look for other factors that might be involved and check out what the results are if you consider those different factors separately.

Is a private school really better than a public school? Or does the private school just start out with better students?
Is Adam really doing better than Abigail in school? Or is Abigail just taking harder courses?

The most important lesson of Simpson’s Paradox is this: don’t assume that something OBVIOUSLY true is REALLY true. Dig deeper and it might turn out to not be true at all.


Copyright © 2016 by Joseph Wayne Gadway

***

Interesting and humorous book on the art of persuasion.

If you want to support "Anything Smart" just click on book links like the one below to buy your books. "Anything Smart" will receive a commission. Thanks!