Survivorship Bias in UX Research: Why You Only Hear From the People Who Stayed

What is survivorship bias? Survivorship bias is the mistake of studying only the people or things that made it through, and forgetting the ones that did not.

The ones that did not make it are quiet. They are gone, so they cannot answer your questions. What is left sounds like the whole story.

In short

• What it is: drawing a conclusion from the people or things that survived, when the ones that did not are out of sight.
• Why it matters: the missing group often has the answer to your question, such as why people cancel.
• How it shows up: surveys of current users, app store reviews, interviews with loyal customers, and case studies of products that succeeded.
• The fix: ask “who is missing from this data?” before you trust a finding, then go and find them.

---

Why does it happen?

There are three reasons.

1. Survivors are easy to find. They are still using the product and still answering email.
2. People who left have no reason to talk to you. They moved on.
3. We like success stories. Winners get written about. Failures are forgotten.

Imagine a teacher who asks the students still in the room, “How was the course?” Everyone says it was great. The students who dropped out in week two were not there to answer.

---

A famous example: the planes

During World War II, the US military studied bomber planes that came back from missions. They marked where each plane had been hit by bullets. The holes were on the wings, the tail and the middle of the body. The first idea was to add armor where the holes were.

A mathematician named Abraham Wald, who worked with the Statistical Research Group at Columbia University, saw the problem. The data came only from planes that had returned. Planes hit in other places, such as the engines, did not come back, so those holes were missing from the data. The clean places on the returning planes were the places that needed armor.

The story is told in several versions, and the details differ. The reasoning is the same in all of them: the planes you can study are the ones that survived.

Show

Planes that came back Every plane that was hit

Only the planes that came back: 38 holes, on the wings, the tail and the middle of the body, and none on the engines or the cockpit. It looks as if the wings need armor, but a plane hit on a wing can still fly home.

Switch between the two views. With only the planes that came back, the engines look safe. Add the planes that did not come back and the engines are where the damage was.

---

Where it hides in UX research

• Surveys of current users. People who left are not on the list.
• App store reviews. They come from people who care enough to write, and who are still around.
• Interviews with loyal customers. They are the ones who stayed.
• Funnels that count only the last step. The people who dropped out in the middle are not counted.
• Case studies of successful products. “Copy what the winners do” ignores the losers that did the same.
• Usability tests with existing users. They already got past the hard part.

---

See it with 100 sign-ups

Here is a model with made-up numbers. 100 people signed up. The slider is how many of them left. Of the people who stayed, 90% are happy. Of the people who left, 20% are happy. Choose who you ask.

You ask

Only people still using it Everyone who signed up

People who left, of the 100: 60

You asked only the 40 people who stayed. 36 of them are happy, which is 90%. If you had asked all 100 who signed up, 48 would be happy, which is 48%.

Leave the slider at 60 and ask only the people who stayed. Nine in ten say they are happy. Now ask everyone who signed up and the answer is 48%. The product did not change. Only the group you asked did.

Move the slider and watch the first number. It stays near 90% however many people leave. A survey of survivors cannot tell you that people are leaving.

---

How to catch it

1. Ask “who is missing?” about every data set. Write the missing group down.
2. Find the people who left. Use an exit survey, an email, or an interview with someone who cancelled.
3. Count the whole funnel. Report how many started, not only how many finished.
4. Study the failures. If you copy a successful product, look for products with the same features that failed.
5. Recruit from outside your user base. Include new, lapsed and never-converted people.
6. Write down your sample. Say who it covers and who it leaves out.

---

Try it yourself: who is missing?

The question is: why do people cancel? Pick the sources you will use. Then see whom you will hear from.

Your sources

Survey of current users Support tickets from current customers App store reviews Exit survey when people cancel Interviews with people who cancelled

You will hear from no one who left. Any reason you find comes from people who stayed, so it cannot explain why people leave.

If you hear from no one who left, any “why” you find comes from people who stayed. It cannot explain why people leave.

---

Survivorship bias and other biases

• It is a close cousin of selection bias. Selection bias is about who gets picked for your study. Survivorship bias is what happens when the group is picked by what happened to it.
• It feeds confirmation bias: when everyone you hear from is happy, you find it easy to believe the product is fine.
• It is the opposite habit to usability testing with real, fresh people, which brings the missing voices back in.
• The bandwagon effect can make it worse: you copy what winners do, and never see the losers that did the same.

See the map of cognitive biases for how these fit together.

---

Common mistakes

• Trusting a high score. A score from survivors is high by design.
• Reading only five-star reviews. Read the other ends too, and remember that both are only the people who chose to write.
• Counting only the end of the funnel. The middle is where people leave.
• Learning only from winners. Some losers did exactly what the winners did.