Chapter 2: Find out if you're wrong, before you build, lesson 2 of 7

Turn a belief into a test

A belief becomes a test when a result could prove it wrong. Six plain questions, asked one at a time, turn any belief into a test with a number and a date.

By 4 min read, 15 min of exercises

Warm-up

From last time. Each of these is something the team could look at. Could it prove Peter wrong, or can it only agree with him?

  1. Five offices say they would like group ordering.

    Right, it's something that can only agree. People say yes to a free feature. It's 8, 10, 12 again.

    Not quite. It's something that can only agree. People say yes to a free feature. It's 8, 10, 12 again.

  2. Most managers who get the link still type every order.

    Right, it's something that could prove him wrong. It's something they do, and it goes against Peter's belief.

    Not quite. It's something that could prove him wrong. It's something they do, and it goes against Peter's belief.

  3. Office managers tell you the idea sounds great.

    Right, it's something that can only agree. It's a kind word, not a result. It can't go against him.

    Not quite. It's something that can only agree. It's a kind word, not a result. It can't go against him.

Last time, Peter asked for group ordering. You pulled out the beliefs his idea rests on. And you found that asking offices if they’d like it can only agree with him.

Now Mia, Tiffin’s research lead, wants one of those beliefs turned into a real test. Something that could prove Peter wrong, she says, by Friday.

Think first

Peter, head of sales, replies to the team: Group ordering will get us more orders.

Rewrite Peter’s line so that some result could prove it wrong.

You

Which of these is closest to your answer?

Shirish

Good, a number. Now picture the day you check it. Orders are up 20%. Was it group ordering, or a new menu, or the weather?

And when do you check?

Shirish

That’s a feeling, and feelings are hard to prove wrong. Offices can love an idea and never use it.

What would offices do, that you could count?

Shirish

Yes. That can come out against Peter, and it’s something you can count.

Two things are missing. By when? And why half, not a third?

You

What this means

A belief becomes a test when you add the result that would prove it wrong, decided before you look.

That’s a lot to hold in one sentence. So Mia doesn’t ask for one. She gives you six plain questions instead.

Write the test, one question at a time

Here are Mia’s six questions. Answer them one at a time, for one of Peter’s beliefs. Your answers save as you type. You’ll use them again in the chapter’s project.

Your turn

What you're making: the test your team runs before anyone builds group ordering. The first two answers come from Peter's email, so they are filled in for you. Each question also shows how Gmail's attachment check, from lesson 1.1, would answer it.

Turn this belief into a test, one question at a time:

One of Peter's beliefs about group ordering Office managers will send colleagues a link, and each person will add their own meal.

Question 1 of 6

What am I adding or changing?

Name the one thing you would build or change.

How Gmail's attachment check answers this

A check that asks "Did you forget the file?" when an email says "attached" but has no file.

Question 2 of 6

Who is it for?

Name the people whose behaviour you are testing, not "users".

How Gmail's attachment check answers this

Everyone who sends email.

Question 3 of 6

What should people do differently?

Something you could see them do. Not "be able to".

Pick the strongest answer. You can change the words after.

How Gmail's attachment check answers this

Add the file before they press Send, instead of sending a second email.

Question 4 of 6

Why would they?

The reason your change works.

Pick the strongest answer. You can change the words after.

How Gmail's attachment check answers this

The check catches the slip at the moment it happens.

Question 5 of 6

What result would prove it useless?

Count the thing that matters, and pick the number now.

Pick the strongest answer. You can change the words after.

Why that number? Pick the strongest reason.

How Gmail's attachment check answers this

Second emails with the file drop by less than a quarter.

Why that number: Below that, the check stops people more often than it helps them.

Question 6 of 6

On what date do I decide?

A real day. On that day you decide, whatever the numbers say.

How Gmail's attachment check answers this

Four weeks after launch.

Your hypothesis

This is what people mean by a hypothesis: every answer, together. You wrote each part of it.

The two questions most tests get wrong

Look back at questions 5 and 6: the number that would prove you wrong, and the date. They are where most tests go wrong. I got both wrong myself.

Shirishfrom my own product

How I got the number and the date wrong

My pop-up asks Are you sure? before a project is deleted. I was asked to write a test for it, as one sentence with gaps to fill. I couldn’t. I said, I don’t understand what to put in the square brackets.

Asked as plain questions, it was easy. How many people in 100 would need to press Cancel for the pop-up to be worth keeping? I said 20.

Then: for how long would I watch? I said, One month minimum. That was my mistake. At least a month has no end. The test can run for a year, because the number never looks quite good enough.

A number needs a reason, or it’s just a guess.

Think first

I said my pop-up is worth keeping if 20 in every 100 people press Cancel.

Which reason makes 20 a number worth deciding on?

You

Pick the answer closest to yours.

Shirish

It is round. But would 25 be worse? Or 15? A round number is easy to say, and just as easy to change later.

What would 20 in 100 mean for the people who see the pop-up?

Shirish

Yes. That reason is about people, so the team can argue with it. That’s the point.

Someone might say losing weeks of work is so bad that even 5 in 100 is enough. What would you say to them?

Shirish

Other apps have other users and other costs. Their number was set for their reasons, if they had any.

What does 20 in 100 mean for your own users?

You

What this means

A good reason says what the number means for people. Pick the number, then write why. If you can’t say why, you’re guessing.

And a date has to be a real day.

Practice

Which of these ends a test on a real date? Sort each one.

  1. We decide on Friday, 16 October.

    Right, it's a date to decide. It names the day. On that day you decide, whatever the numbers say.

    Not quite. It's a date to decide. It names the day. On that day you decide, whatever the numbers say.

  2. We'll watch it for at least a month.

    Right, it's an open door. "At least" has no end. It can run for a year.

    Not quite. It's an open door. "At least" has no end. It can run for a year.

  3. We'll decide when we have enough data.

    Right, it's an open door. Nobody agrees on "enough". The test never ends.

    Not quite. It's an open door. Nobody agrees on "enough". The test never ends.

  4. Four weeks after launch: 30 October.

    Right, it's a date to decide. It names the length and the day.

    Not quite. It's a date to decide. It names the length and the day.

  5. We'll keep going until the numbers look good.

    Right, it's an open door. This one only ends if you're right. It can never prove you wrong.

    Not quite. It's an open door. This one only ends if you're right. It can never prove you wrong.

What you just built

What you built with Mia’s six questions has a name: a hypothesis. It’s a belief, written with the result that would prove it wrong. The result you picked in question 5 is the kill criterion. The date in question 6 is the timebox.

People often teach a hypothesis as one long sentence with gaps to fill. You built it the other way round, one plain question at a time. That’s easier, and it’s the same thing.

Words from this lesson

Hypothesis
A belief, written with the result that would prove it wrong. In this lessonWe're wrong if fewer than half the managers use the link twice.
Kill criterion
The result, chosen before you look, that means stop. In this lessonFewer than half the managers use the link twice in two weeks.
Timebox
The date you decide on, whatever the numbers say. In this lessonWe decide on Friday, 16 October.

Back to your first answer

Your first answer

Your answer

Your answer will show here after you write one at the top of this page.

You've read the lesson. Would you change your answer?
Good. Which part of this lesson backs your answer?

Your bet: which belief would you test first?

Your bet

On Friday, Mia reads your test. Good, she says. But Peter’s idea rests on four beliefs, and we only have time to test one this week.

Which one would you test first? Make your bet, and say how sure you are.

Your guess
How sure are you?

You'll find the answer in Lesson 2.3: Test the riskiest belief first.

Try this during the week: take one belief from your own work. Answer the six questions for it, and put a real date on it. The free hypothesis builder keeps your answers.

Sources

  • Karl Popper, Conjectures and Refutations, 1963. A claim is useful when some result could prove it wrong.
  • The one-sentence hypothesis is common in product teams. It is often taught from Jeff Gothelf and Josh Seiden’s Lean UX, 2013. This course asks the same things as plain questions.
  • The delete pop-up is from my own product. Gmail’s check is real; its numbers here are made up for the example. Tiffin, Peter and Mia are made up for this course.
Portrait of Shirish Shikhrakar

Written by

Shirish Shikhrakar

Shirish is a Google-certified UX designer and UX engineer based in Kathmandu, Nepal. He started as a software developer, moved into product design for Silicon Valley startups, and has taught UX foundations to hundreds of designers since 2019.

AboutLinkedIn