Chapter 4: Turn notes into decisions, lesson 4 of 7
Read the numbers before you trust them
A total can go up while every part goes down. Read the numbers by group, ask who is missing, watch out for targets, and use people to explain the numbers.
Warm-up
From last time. Does each thing belong on today's journey map, or in a test?
-
Right, it's something for today's map. It's what happens today. It's the evidence for the fix.
Not quite. It's something for today's map. It's what happens today. It's the evidence for the fix.
-
Right, it's something for a test. It's a fix. It goes in a test, not on today's map.
Not quite. It's something for a test. It's a fix. It goes in a test, not on today's map.
-
Right, it's something for today's map. It happens today, and it's serious. Keep it on the map.
Not quite. It's something for today's map. It happens today, and it's serious. Keep it on the map.
Last time, you mapped Anna’s day and chose what to fix. That came from people. Now the team gets a number, and numbers can fool you in ways people can’t.
Nora Kim, the growth manager, sends this to the product team.
- From
- Nora Kim, Growth manager
- To
- Product team (you're on it, as the designer)
- Subject
- The checkout redesign worked
Hi all,
Great news. We launched the new checkout on 1 September, and conversion went from 20% to 26.4%. Same 2,000 visits in the first two weeks, 128 more orders.
I'd like to use the same design on the menu page next. Can we plan it for October?
Nora
Think first
Nora, growth manager: Conversion went from 20% to 26.4%. Same 2,000 visits, 128 more orders.
Would you agree the new checkout worked? What would you check first?
You
You
What this means
Before you trust a total, split it into groups and look at each one. Here’s the same data, split by device.
The same numbers, split
| Device | August (old checkout) | September (new checkout) |
|---|---|---|
| Computer | 40% (160 of 400 visits) | 36% (432 of 1,200 visits) |
| Phone | 15% (240 of 1,600 visits) | 12% (96 of 800 visits) |
| All visits | 20% (400 of 2,000) | 26.4% (528 of 2,000) |
Think first
On computers, 40% became 36%. On phones, 15% became 12%. In total, 20% became 26.4%.
What happened?
You
You
What this means
The new checkout made both devices worse, by 4 points on computers and 3 on phones. The total rose because of who arrived. In September, a LinkedIn ad for office computers started, and phone ads were cut back.
So what should the team tell Nora? The new checkout probably hurt. Judge it inside each group, not by the total.
When the mix of people changes, the total can move the opposite way from every group inside it. That’s segment mixing. Statisticians call it Simpson’s paradox. It’s not rare, and it’s not noise.
Who’s missing from the numbers?
Nora has another idea. She wants to survey current group cart users about what they love. What’s wrong with that?
Think of Dana. She used the group cart from May to July, then stopped. She’s not a current user, so she’d never get the survey. The people with the best reason to complain are the ones the survey can’t reach.
Studying only the people who stayed is survivorship bias. Before you trust a number, ask who it leaves out. (There’s a full article on survivorship bias (opens in a new tab).)
When a number becomes a target
Here’s one more trap, hiding in Nora’s email. Say the team is rewarded for conversion. The easiest way to raise it isn’t a better checkout. It’s cutting the phone ads, which is what happened in September by accident.
When a number becomes a target, people find ways to hit it that miss the point. That’s Goodhart’s law. Watch for it whenever a number is someone’s goal.
Can this number fool you?
Practice
Which trap could fool you with each number?
-
Right, it's segment mixing. Each office may order the same as before. The mix changed.
Not quite. It's segment mixing. Each office may order the same as before. The mix changed.
-
Right, it's survivorship bias. Only people who stayed are users. The unhappy ones already left.
Not quite. It's survivorship bias. Only people who stayed are users. The unhappy ones already left.
-
Right, it's a number that became a target. The number is hit. The problem isn't solved.
Not quite. It's a number that became a target. The number is hit. The problem isn't solved.
-
Right, it's survivorship bias. You heard from the people who stayed. What about those who left?
Not quite. It's survivorship bias. You heard from the people who stayed. What about those who left?
Numbers say what. People say why.
The split shows the checkout got worse. It doesn’t show why. For that, you need people. I learned this the expensive way.
Shirishfrom my own work
The checkout that wasn't the problem
Early in my career, I relied purely on analytics. The data showed a huge drop-off at checkout. I spent weeks redesigning it. Cleaner, simpler, more obvious.
Launch. No improvement. The drop-off stayed exactly the same.
Then I watched five people try to check out. The problem wasn’t the interface. Users didn’t trust us with their card details. Analytics couldn’t tell me that. Only watching users could.
Use both. Numbers tell you what happened, and how much. People tell you why.
Words from this lesson
- Segment mixing
- The total moves because the mix of people changed, not because each group changed. In this lessonConversion rose from 20% to 26.4%, while computers and phones both fell.
- Survivorship bias
- Studying only the people who stayed, and missing those who left. In this lessonSurveying current users, so you never hear from Dana.
- Goodhart's law
- When a number becomes a target, it stops being a good measure. In this lessonRaising conversion by cutting phone ads, not by fixing the checkout.
Back to your first answer
Your first answer
The question
Your answer
Your answer will show here after you write one at the top of this page.
Your bet: what do most readouts leave out?
Your bet
It’s Thursday. Mia has to give Peter one page by Friday: everything the team learned, and what to do next.
Which line do most research write-ups leave out? Make your bet, and say how sure you are.
You'll find the answer in Lesson 4.5: Write a readout that carries a decision.
Try this during the week: find a number someone is proud of at work or in the news. Ask what changed in the mix, and who it leaves out.
Sources
- Edward Simpson, The Interpretation of Interaction in Contingency Tables, 1951. The paradox now named after him.
- Charles Goodhart, 1975, on monetary targets. The plain-words version is Marilyn Strathern’s, 1997.
- The checkout story is from my own work. Tiffin, Nora and Dana are made up for this course.
Written by