Chapter 2: Find out if you're wrong, before you build, lesson 4 of 7

Match the decision to the evidence

Some decisions can't wait for proof. Learn how strong your evidence is. Then size the step to match it, and set a date to look again if the proof is weak.

By 5 min read, 15 min of exercises

Warm-up

From last time. Mia had time to test one of Peter's beliefs this week. Should each of these be tested now, or not now?

  1. Placing the order is the hard part.

    Right, it's one to test now. It's the lowest belief with no proof. Two others rest on it.

    Not quite. It's one to test now. It's the lowest belief with no proof. Two others rest on it.

  2. Offices order lunch as a team.

    Right, it's one to leave for now. Nora's records already show it. 34 of 60 offices use the group cart every week.

    Not quite. It's one to leave for now. Nora's records already show it. 34 of 60 offices use the group cart every week.

  3. It will bring more orders than it costs.

    Right, it's one to leave for now. It rests on the beliefs under it. If those fall, it can't come true.

    Not quite. It's one to leave for now. It rests on the beliefs under it. If those fall, it can't come true.

Last time, Mia, Tiffin’s research lead, picked the belief to test first: placing the order is the hard part. Her test runs next week.

But Peter, the head of sales, doesn’t want to wait. Three offices asked him for group ordering, and he wants to give them an answer.

Think first

Peter, head of sales, at the team meeting: Three offices asked for it. Can’t we just start building?

Ryan Brooks, the product manager, guesses that building group ordering would take the team about ten weeks.

What would you tell Peter?

You

Which of these is closest to your answer?

Shirish

It’s real that they asked. But asking costs an office nothing, and people ask for lots of things they never use.

On the ladder from lesson 1.2, where does they asked sit?

Shirish

That’s safe. But waiting is a decision too. Peter has offices waiting for an answer, and a rival, LunchRun, might ship first.

Is there anything small the team could do now, that would be easy to undo?

Shirish

Yes, that’s the shape of it. Something now, and a time to look again.

What makes a step small enough? Is it the cost, or how easy it is to undo?

Shirish

That’s the right thing to ask. Say it took one afternoon. Would three offices asking be enough?

And if it took ten weeks?

You

What this means

Two things decide it. How strong is the proof? And how hard is the decision to undo?

Three offices asking is weak proof. Ten weeks of building can’t be given back. Those two don’t match. So first, how weak is the proof?

How strong is the proof for group ordering?

In lesson 1.2, you met the evidence ladder. It shows how much you can trust a piece of evidence. Here it is again.

  1. 0 Opinion I think people forget files.
  2. 1 One story One person tells you they forget files.
  3. 2 What people say Several people say they forget files.
  4. 3 What you saw You watched 5 people. 2 forgot a file.
  5. 4 What you counted The log shows 3 in 100 emails are sent again with a file.
  6. 5 It happened again Next month, the count is the same.
  7. 6 A fair test You added the check, and those second emails dropped.
The evidence ladder. The further down, the more you can trust it. You will use it in every chapter.

Practice

Here is what the team knows about group ordering so far. Where does each piece sit on the ladder?

  1. Three big offices asked Peter for group ordering.

    Right, it's what people say, rung 2. They told Peter what they want. That's rung 2. Asking is free, so people ask for lots of things.

    Not quite. It's what people say, rung 2. They told Peter what they want. That's rung 2. Asking is free, so people ask for lots of things.

  2. Nora's records: 34 of 60 offices use the group cart every week.

    Right, it's what people do, rung 3 or 4. It's counted from what offices did. That's rung 4.

    Not quite. It's what people do, rung 3 or 4. It's counted from what offices did. That's rung 4.

  3. Peter's guess: it could double office orders.

    Right, it's an opinion, rung 0. It's Peter's guess. Nobody has seen it happen. That's rung 0.

    Not quite. It's an opinion, rung 0. It's Peter's guess. Nobody has seen it happen. That's rung 0.

  4. Ryan: building it would take about ten weeks.

    Right, it's an opinion, rung 0. It's a guess too, from someone who knows the team. Guesses like this often run long.

    Not quite. It's an opinion, rung 0. It's a guess too, from someone who knows the team. Guesses like this often run long.

The demand for group ordering rests on rung 2: three offices asking. Peter wants to spend ten weeks on it.

Decisions can’t always wait

So should the team wait for strong proof? I faced the same question on an idea of my own.

Shirishfrom my own work

One teacher, and a decision that couldn't wait

My idea for teachers showed how a student’s code was made: built up over days, or pasted in whole. I planned a test with one teacher. He’d grade ten pretend assignments, with names hidden and the code history showing. If he used the history for 8 of the 10, I’d call it a yes.

Then came the question: how far can one teacher take me? My answer was honest: I can’t decide from rung 1 alone. I wanted a count across many teachers. But doing that at scale isn’t easy.

So I said, Sometimes a decision needs to be made and I need to live with it.

That’s true for every team, Tiffin’s too. You’ll often decide before the proof is strong. The trick is to choose a step the proof can carry.

Size the step to the proof

Practice

The team has weak proof for group ordering: three offices asking. Which of these steps could that carry?

  1. Sketch the link screen and show it to the three offices.

    Right, it's a step that weak proof can carry. A sketch takes a day, and you can throw it away.

    Not quite. It's a step that weak proof can carry. A sketch takes a day, and you can throw it away.

  2. Spend ten weeks building group ordering for every office.

    Right, it's a step that needs strong proof. Ten weeks can't be given back. A step this big needs proof from what people do.

    Not quite. It's a step that needs strong proof. Ten weeks can't be given back. A step this big needs proof from what people do.

  3. Tell the three offices you're looking into it, and will reply by 23 October.

    Right, it's a step that weak proof can carry. It promises an answer, not a feature. That's easy to keep.

    Not quite. It's a step that weak proof can carry. It promises an answer, not a feature. That's easy to keep.

  4. Promise the three offices they'll have it by December.

    Right, it's a step that needs strong proof. A promise to customers is hard to take back. If the proof falls, you break it.

    Not quite. It's a step that needs strong proof. A promise to customers is hard to take back. If the proof falls, you break it.

The small steps have two things in common. They’re cheap, and they’re easy to undo. That’s what weak proof can carry.

There’s one more thing a decision on weak proof needs: a date to look again. Without one, live with it turns into forever. You’ve seen this before. It’s the date from lesson 2.2, used on a decision instead of a test.

So the team decides. This week, Ryan sketches the link screen, and Peter tells the three offices they’ll hear back by Friday, 23 October. Mia runs her test. On 23 October, the team looks again.

What you just did

You matched a decision to its proof. That rule has two halves:

  1. A decision can’t be surer than its proof. Three offices asking is rung 2. It can’t justify ten weeks.
  2. When you must decide anyway, size the step to the proof. Weak proof can carry a step that’s cheap and easy to undo. That’s a reversible step. Give it a date to look again.

Peter’s mistake has a name too. Three offices asking felt like a lot, because they were all he could see. Tiffin has 60 office accounts, and nobody heard from the other 57. The psychologist Daniel Kahneman called this what you see is all there is. We judge from what’s in front of us, and forget what’s missing.

Words from this lesson

Reversible step
A step you can undo cheaply if the proof turns out wrong. In this lessonSketching the link screen and showing it to three offices.
What you see is all there is
Judging from the evidence in front of you, and forgetting what you haven’t seen. In this lessonThree offices asked, so Peter forgot about the other 57.

Back to your first answer

Your first answer

Your answer

Your answer will show here after you write one at the top of this page.

You've read the lesson. Would you change your answer?
Good. Which part of this lesson backs your answer?

Your bet: which test would you run in one day?

Your bet

Mia has one day next week to test Peter’s belief: placing the order is the hard part.

Mia, research lead, to you: One day. What should we do with it?

Which test would tell her the most? Make your bet, and say how sure you are.

Your guess
How sure are you?

You'll find the answer in Lesson 2.5: Run a test in a day.

Try this during the week: think of a decision your team made on weak proof. Was the step easy to undo? And did anyone set a date to look again?

Sources

  • Daniel Kahneman, Thinking, Fast and Slow, 2011. What you see is all there is is his name for judging only from what’s in front of you.
  • The evidence ladder is this course’s own tool, from lesson 1.2.
  • The teacher test is from my own work. Tiffin, Peter, Mia, Nora and Ryan are made up for this course.
Portrait of Shirish Shikhrakar

Written by

Shirish Shikhrakar

Shirish is a Google-certified UX designer and UX engineer based in Kathmandu, Nepal. He started as a software developer, moved into product design for Silicon Valley startups, and has taught UX foundations to hundreds of designers since 2019.

AboutLinkedIn