Testing Business Ideas with Customers: Going Beyond the Data

By Noel Sobelman


Early in the innovation process, when solution uncertainty is highest, it’s common practice to run experiments with target customers to test the most critical aspects of a transformative new idea.  But even the most well-designed experiments can produce misleading results.  Innovation teams must go beyond the numbers and ask themselves, “Why did the customer behave this way?”  It often takes keen observation, multiple experiments, and follow-up with customers to capture truly meaningful insights.  No single experiment or data set holds all the answers.


One of my favorite examples that highlights the difference between experiment results and insights comes from Netflix back in the days when they were trying to figure out the impact next-day DVD delivery would have on their customers.  As co-founder Marc Randolph describes in “That Will Never Work,” Netflix hypothesized that the move to next-day delivery would lead to improved customer retention.  However, the data told a different story.  An analysis of the data showed no correlation between delivery time and customer retention.  So, instead of building an entirely new distribution network based on a potentially flawed hypothesis, Netflix decided to learn what was going on by running a low-cost experiment.


For several months, a Netflix employee drove inventory to a nearby test city to overnight mail the previously ordered DVDs (a Wizard of Oz experiment).  What they learned surprised them.  Next-day delivery did not impact customer retention.  However, they noticed an increase in customer sign-ups.  This was puzzling since they did not promote next-day delivery to non-subscribers.


Further investigation revealed the true insight.  Word-of-mouth was the behavior change triggered by next-day delivery.  Customers in the test city were so delighted with overnight delivery that they were telling their friends about the service. This insight gave leadership the confidence to roll out next-day delivery nationwide — a game changer for scaling Netflix.

In new venture creation, the goal of experimentation is to produce evidence in the form of a measurable customer action or behavior that proves you are progressing toward a profitable, scalable business.  Proof in the form of data plus insights will inform you if customers desire your solution, are engaged when they use it (enough to tell their friends) and remain engaged over time.  So don’t stop with the experiment results.  Talk to your customers and find out why they behaved the way they did.

About the Author

Noel Sobelman is a researcher, writer, and corporate advisor on innovation effectiveness. His experience includes senior-level corporate roles, new venture creation, and executive advisory. He is widely recognized for bringing a practical and applicable approach to companies looking to accelerate growth from innovation.