Hypothesis Driven Development In Action
The concept of Hypothesis Driven Development was an absolute epiphany for me and influences basically every aspect of how I think about website development today. From ideation, to development, and even design, I’m thinking of everything in a hypothesis and research backed format.
But what is this process?
Hypothesis Driven Development in a Nutshell
The general approach I personally take with hypothesis driven development is to simply turn “I know” to “I think”. And if you think something might be true, then how would you test it to make sure you do know.
Or to formalize that a little more:
I believe that hypothesis goes here .
We’ll know we’ve validated this when success metric goes here .
The beautiful part of breaking it down this way is we can execute on what we know instead of blindly executing on a bad idea, designing something that doesn’t work, or programming a feature no one uses.
Let’s apply this to a fictional physical product. Jetpacks! Jetpacks are clearly the future of transportation and we finally have the technology to build them!
Idea Hypothesis Testing
Assumption: “If we build a Jetpack, people will buy it.”
Hypothesis: “We believe that if Jetpacks were available to purchase, people would buy them.”
Now, here is where you need to be clever. You need to define a way to test this hypothesis. The best way to do this is to define your perfect customer and test if they are interested. Define this customer well and be very specific. The goal here is legitimizing your market and demand for your product. If the person you’ve defined as absolutely perfect for your product isn’t interested…then time to rethink this whole endeavor.
Test: “We’ll create a single landing page that describes what we’re building and ask people for their email to keep them up to date on what we’re doing. This will be one way we prove there is demand.”
Success/Failure Metric: “We’ll know we’ve validated this when the landing page converts 10% of visitors.“
If your idea was proven correct and you’re thinking, “Yeah I knew this already and I just wasted all this time validating and when I could’ve been building!” You’re thinking about it the wrong way.
Not only do you have more confidence that you’re building something people want with this approach but once you’ve built the product you’ll have an instant list of people who want to buy. If you ever need outside investment to push the last few weeks of development or marketing, then you’ll have some ammunition when showing your product is actually sought after to investors.
Mint.com for example had more than 20k emails collected before they launched their product. How would you like to have that many people ready to buy on your launch day?
Sidenote: Testing ideas with a hypothesis is well covered in the Lean Startup. I’ve recommended this before. It’s a great introduction into building the correct product.
Design Hypothesis Testing
Ok, well that’s marketing you might be thinking…that doesn’t fit into design. So wrong. A designer who can prove their design is superior to another is a dangerous designer.
You’ll commonly see designs go head to head with what’s called A/B testing. That is a great way to test which design is completing the objective for the page. In this case the objective is to collect visitor’s emails
Expanding on the Jetpack landing page, let’s pretend you’re a designer trying to prove your worth to this jetpack startup company and the CEO just doesn’t see much value in graphic design or user experience. They made a crap landing page and you know you can do better.
Assumption: “My landing page design is better than what is currently used.”
Hypothesis: “If we implement my design on the jetpack landing page, we’ll see an increase in email conversions.”
Test: “We’ll create a secondary landing page using my design to see if it converts better.”
Success/Failure Metric: “We’ll know we’ve validated this when my landing page consistently converting more visitors.“
Sidenote: A/B testing is fantastic for emails too. Most major email campaign software has it backed in. This is another space where a designer can add major value.
Hypothesis Driven Development
Absolutely huge impact can be made with testing a hypothesis in development. Development is expensive so it’s important to be setting tasks for development teams that are as close to correct as possible.
Let’s say the Jetpack CEO comes to the development team and says,
“EVERYONE is asking for our jetpacks to be customizable! Unique colors, sizes, speeds! We NEED a customization walkthrough process on the checkout!”
So if you’re paying attention, there are some assumptions here. The tip offs are words like, “Everyone” and “Need”.
Most developers, myself included, can’t wait to design and build something like this. It’s a subtly complicated feature of the checkout, especially if it’s completely automated and syncs with the manufacturing systems directly.
Since so many sites have a customization process in their checkout it might seem like an easy ask to the CEO but developers know better. But the real question is if no one uses it..was it worth building?
Let’s just test our CEO’s first assumption, the EVERYONE claim.
Assumption: “EVERYONE wants customization in checkout and they will use it if available.”
Hypothesis: “If we build a robust customization feature, people will use it.”
Test: “We’ll add a text area field where customers can write in their customization requests.”
Success/Failure Metric: “We’ll know people want more robust customization if we see more than 30% of orders with text customization requests.“
The beauty of this is that we built a way for customers to make customization requests, so we have a simple version of the feature already out the door even if it’s not exactly what the CEO had in mind.
Furthermore, we finished developement end to end in one week, not 5-6 weeks. And now we’re able to measure actual usage and gather customer feedback as we build version 2.
Sidenote: This is a great way to show CEOs, clients, and product managers their ‘insight’ might need some more polishing and you’re not just a worker bee.
Explaining Hypothesis Driven Development to Clients
Creating and testing various hypothesis you may have for yourself or your internal projects may be one thing but explaining this concept to clients might not always work out as you intended. They are hiring you to build their idea, not challenge it.
However, if you build something and it doesn’t yield the results they want, you will be the first to blame. Not their lack of research, promotion, or marketing.
The dangerous pit so many rookie entrepreneurs, both technical and non-technical, fall into is limited market and customer research on their end. The assumption is that once a website is built the sales will just start rolling in. I assure you this is not the case.
Testing each assumption the client has made is a great way to validate their ideas, define a limited scope to keep costs down, and build trust between both parties.
My client projects only follow two types of paths currently:
1) Hypothesis Driven Development and starting small.
2) Development based on well defined tasks that removes me from ‘success of project’ beyond a functional website/feature.
I’ll gladly program for anyone willing to pay but success online involves a lot more than just having a website.
Hypothesis Driven Development in Action
Hopefully you have a crisp idea of what I’m getting at but let’s dig in with some more concrete examples. If you read some more in-depth articles on product development you’re seeing words like ‘assumptions’, ‘testing’, and ‘hypothesis’. They are doing hypothesis driven development whether it’s on a small scale or large scale. Here are some examples from large and smaller companies:
- Netflix – Hypothesis Driven Development at Netflix
- Groove – 3 Early Fails that Nearly Killed Our Startup
- Jason Shah – Shutting Down Instead of Pivoting
- Magoosh via Intercom – Magoosh Increased Conversion 17% with Onboarding Experiment