Chapter 2: Find out if you're wrong, before you build, lesson 5 of 7
Run a test in a day
Five cheap tests you can run in a day, and what each can and can't tell you. Plus two questions to ask first: could it fail, and who agreed to it?
Warm-up
From last time. The team had weak proof for group ordering. Could weak proof carry each of these steps?
-
Right, it's a step weak proof can carry. A sketch is cheap, and you can throw it away.
Not quite. It's a step weak proof can carry. A sketch is cheap, and you can throw it away.
-
Right, it's a step that needs strong proof. Hiring is costly and hard to undo.
Not quite. It's a step that needs strong proof. Hiring is costly and hard to undo.
-
Right, it's a step weak proof can carry. It's a date to look again. That's what a decision on weak proof needs.
Not quite. It's a step weak proof can carry. It's a date to look again. That's what a decision on weak proof needs.
Last time, the team decided not to build yet. Mia, Tiffin’s research lead, got one day to test the belief at the bottom of the stack. That belief is: placing the order is the hard part.
You made a bet on how to spend that day. Mia picks two things, both for Monday, 28 September:
- In the morning, she reads every help request offices have sent Tiffin since July.
- At lunchtime, she sits with an office manager who orders lunch for her team, and watches. Start to finish, from the menu to the money.
Why not just ask managers if ordering is hard? Because the question points them at ordering, so they’ll talk about ordering. Watching lets them show you where it hurts. And a link would take weeks to build. A day isn’t enough.
Think first
On Friday, Mia asks you to write down one thing before Monday.
Mia, research lead, to you: If we only decide what counts afterwards, we’ll see what we want to see.
What result on Monday would prove the belief wrong: placing the order is the hard part?
You
You
What this means
Mia writes it down on Friday. If ordering takes a few minutes, and most of the work is elsewhere, the belief is wrong.
It’s lesson 2.2 again. Decide what counts as wrong before you look.
Monday morning: what the help requests say
Peter, the head of sales, said Tiffin gets help requests about groups all the time. Nobody had read them. Mia reads all 40, from July to September.
Practice
Here are six of them. What is each one really about?
-
Right, it's about paying and getting paid. Ordering works. Paying for twelve people, every day, is the trouble.
Not quite. It's about paying and getting paid. Ordering works. Paying for twelve people, every day, is the trouble.
-
Right, it's about placing the order. This one is about placing the order. It's slow when you add every meal yourself.
Not quite. It's about placing the order. This one is about placing the order. It's slow when you add every meal yourself.
-
Right, it's about paying and getting paid. She wants to stop collecting money. The order isn't the problem.
Not quite. It's about paying and getting paid. She wants to stop collecting money. The order isn't the problem.
-
Right, it's about something else. It's about delivery. Group ordering wouldn't change it.
Not quite. It's about something else. It's about delivery. Group ordering wouldn't change it.
-
Right, it's about paying and getting paid. They do want a link. But a link to pay, not a link to order.
Not quite. It's about paying and getting paid. They do want a link. But a link to pay, not a link to order.
-
Right, it's about placing the order. Placing the order went wrong here. It's a bug in the group cart.
Not quite. It's about placing the order. Placing the order went wrong here. It's a bug in the group cart.
Of the 40 help requests, 10 were about groups. Seven of those 10 asked about money: paying for your own meal, or getting your share back. Three were about placing the order.
That’s rung 2: what people wrote. It shows what hurt enough to write about. It can’t show what people do, or what they gave up on without writing.
Monday lunchtime: Anna shows how she does it
Anna is the office manager at a software company. She orders lunch for about 11 colleagues, through the team’s group chat. Mia asked first, and Anna agreed to be watched and recorded.
At 10:15, Anna posts the day’s menu. People reply with what they want, and she writes it all in a notebook. By eleven, she calls the order in. That part takes about ten minutes.
Then comes the money. Some people send it in a payment app, and she screenshots it. Some give cash, which goes in an envelope in her drawer. She ticks them off in the notebook. She chases the rest.
Mia asks how much she’s owed right now. Anna opens the notebook. This month, she’s $157 short.
Then Anna mentions something else. In July, she tried Tiffin’s group cart once, for a farewell lunch. Adding 14 meals took her 20 minutes, and she paid almost $200 on her card. Two people still haven’t paid her back. Two of the help requests Mia read that morning were Anna’s.
Anna, office manager: The ordering takes ten minutes. The money takes my whole week.
What Peter believed
Placing the order is the hard part.
So a link that makes ordering easier will bring more orders.
What Monday showed
Getting paid back is the hard part.
Ordering takes Anna ten minutes. The money takes her week, and she's $157 short.
Think first
Mia wrote this on Friday. If ordering takes a few minutes, and most of the work is elsewhere, the belief is wrong.
Does Monday prove Peter’s belief wrong?
You
You
What this means
For Anna, the belief is wrong, and the help requests agree. That’s strong enough for a small step. Mia will talk to five more people this week, starting tomorrow.
And look at what changed. Peter asked for group ordering. What offices need is to stop being owed money. The idea didn’t die. It changed shape, because someone tested the belief at the bottom first.
Who pays if a test goes wrong?
Mia asked Anna before she watched her. It sounds like a small thing. Here is why it isn’t.
Shirishfrom my own work
The test I almost ran on a teacher
Back to my idea for teachers. It showed how a student’s code was made, so the teacher could judge what students really know.
I wanted to know if a teacher grades by the work, or by who wrote it. Here was my first plan, in my own words: I wouldn’t say anything to him.
Some students’ names would show on their code, and some wouldn’t, at random.
Then I’d see if he asked why some names were missing.
Think first
The plan: hide some students’ names at random, during real grading, without telling the teacher. Then see if he asks about the missing names.
What’s wrong with this test?
You
You
What this means
It had two problems. It couldn’t fail, because a teacher needs names to give grades. And real students’ grades depended on a test nobody agreed to.
So I changed it. I’d ask the teacher to set up a pretend assignment, with pretend grades. He’d see only the code and its history, with no names. And he’d know it was a test.
Before you run any test, ask two questions:
- Could the result prove the belief wrong? If not, it’s the 1, 2, 3 of the 2-4-6 game.
- Who agreed to it, and who pays if it goes wrong?
Five tests you can run in a day
Mia used two of these on Monday. There are five worth knowing. Each one is cheap, and each one has limits.
- Read the help requests. Sort what people already wrote to you. It shows what hurts enough to write about. It can’t show what people gave up on quietly. Rung 2.
- “Show me how you do it now.” Sit with someone while they do the task, start to finish. It shows where the time and trouble really go. It can’t show if a new design would fix it. Rung 3.
- A five-person test. Watch five people try a sketch or an early version. It shows where they get stuck. It can’t show if they’d use it for real. Rung 3.
- A fake door. Show a button for something that doesn’t exist yet, and count the clicks. Then tell people right away it’s coming. It shows how many want it enough to click. It can’t show if they’d keep using it. Rung 4.
- A concierge test. Do the service by hand for a few customers, before you build it. It shows if people use it, and pay. It can’t show if it works for every office. Rung 3 or 4.
A fake door has a cost: the people who click are let down. So tell them at once, and keep it short. That’s who pays, for this test.
Pick the test for the belief
Practice
After Monday, the team has new beliefs about paying. Which test, from the list above, could prove each one wrong?
-
Right, it's a five-person test. Watch five managers try the sketch. If most can't find the button, the belief is wrong.
Not quite. It's a five-person test. Watch five managers try the sketch. If most can't find the button, the belief is wrong.
-
Right, it's a fake door. Put a Split the bill button on the office page for a week and count the clicks. Then tell people it's coming.
Not quite. It's a fake door. Put a Split the bill button on the office page for a week and count the clicks. Then tell people it's coming.
-
Right, it's a concierge test. Mia collects the money by hand for two offices for a week. If they stop using it, the belief is wrong.
Not quite. It's a concierge test. Mia collects the money by hand for two offices for a week. If they stop using it, the belief is wrong.
Words from this lesson
- Fake door
- A button for something that doesn’t exist yet, to count how many people want it. In this lessonA Split the bill button that counts clicks, then says it's coming soon.
- Concierge test
- Doing a service by hand for a few customers, before you build it. In this lessonMia collects the money by hand for two offices for a week.
Back to your first answer
Your first answer
The question
Your answer
Your answer will show here after you write one at the top of this page.
Your bet: what goes wrong in a first interview?
Your bet
Tomorrow, Mia talks to five more people. Ryan Brooks, the product manager, will run one of the talks. It’s his first.
Ryan, product manager, to Mia: What’s the mistake I’m most likely to make?
What’s the most common mistake in a first interview? Make your bet, and say how sure you are.
You'll find the answer in Chapter 3, lesson 3.3: Follow the thread people hand you.
Try this during the week: pick one belief from your own work. Which of the five tests could prove it wrong in a day? And who would need to agree to it?
Sources
- The fake door and the concierge test are common in product teams. Both are described in David J. Bland and Alexander Osterwalder, Testing Business Ideas, 2019.
- The five-person test comes from Jakob Nielsen, Why You Only Need to Test with 5 Users, Nielsen Norman Group, 2000.
- The teacher test is from my own work. Tiffin, Peter, Mia, Anna and Ryan are made up for this course, and so are the help requests.
Written by