Why experiment?

In the past (and still very common today) the way of developing new products was top down. Leadership would set the strategy, management would build the business case, create a brief then either procure an internal team or hire an agency to do the execution. Many steering committee meetings later and a bit of pre-launch testing - your product was ready to be launched unto an unsuspecting world. Sometimes it would be successful, but in the overwhelming majority of cases, it would be an expensive failure.

There are always inherent risks when creating something new. Any project where the outcome has a high amount of uncertainty, is loaded with risks. These come in 3 general ‘flavours’: Market, Execution, External risks.

Market risk

Uncertainty around the idea. You need to know whether there are customers out there that have the problem you’re trying to solve and are willing to pay for your solution. Then you need to assess whether the customer base is big enough to build a sustainable business on with a repeatable mechanism to reach them

Execution risk 

Uncertainty around your ability to build the solution. Here’s where you need to be sure that you’re able to deliver the thing that customers want. Factors such as available expertise, budgets and underlying complexities of the technology required will impact the ability to bring your product to the market.

External risk

Uncertainty around outside factors. Here, you don’t have direct control over these factors but need to accommodate them into your thinking early on. Things like legal, governmental and societal shifts that could impact the way you build or sell your product, especially if it is right on the frontier of current understanding. 

With each of these risk factors, the more innovative the underlying concept, the higher the risks involved. So it is only prudent to tackle each risk independently and break down the larger questions into smaller challenges that can be proven.

What is an experiment? 

In its most basic form, an experiment is an activity designed to capture data that helps answer a question - ideally set up to be a true or false outcome. Normally seen as the domain of scientists, experiments are in fact part of everyday decision making. The “science” part is the rigour in first defining what you want to find out, understanding the factors which you need to control and recording the outcomes in a way that will help you to progress (or change) your idea. 

A Hypothesis

This is a belief or assumption that if certain conditions are met a stated outcome will be observed. The experiment is then used to prove that hypothesis it true or false. In order to test the hypothesis, there needs to be a variable, something that you can change that will have an impact on the outcome. For example, a simple experiment might be to change the image used in an online ad and measure if more people click on it as a result. 

It’s important to only change one variable per experiment to ensure that you know what is causing the change of outcome. Taking the ad example - you could also change the headline, the call to action or the demographic to test conversion rates - but again, they would have to be done one at a time. Companies such as Amazon will run thousands of these experiments daily.

Why are they useful?

Experiments are cost effective ways to prove key business assumptions before investing a lot of time, money and reputation on building a solution that has a high chance of failure. It is possible to run multiple experiments in parallel to quickly get to a point where you can decide whether the idea is worth taking further

When it comes to product development, the experiments can be divided into 3 main categories based on what sort of questions you’re trying to answer.

Desirability 

These will be questions related to the market, e.g. are customers interested in your solution? These are the foundational questions you’ll need to address, ideally before you even go to raise funds or invest anyone's time in building the solution.

Viability

These questions relate to your ability to generate sustainable revenues from your idea. So pricing plays an important role here,  as does the need to really test that customers are willing to pay to get value from your solution. If you find that you're unable to deliver value to the customer and derive a profit from that activity, then the business model needs to be reworked or abandoned.

Feasibility

These questions relate to you or your teams ability to build the right solutions. This could be down to the resources available or access to partners required to deliver certain aspects of your solution. You would typically only be answering these questions AFTER you’ve proven that customers (a) want your product and (b) are willing to pay for it. 

So, what types of experiments can you start with today? 

There are an infinite number of ways to test whether customers will be interested in your product or service. Some take longer to set up than others and the costs can vary wildly. At Makette, we focus purely on experiments that can be executed digitally and set up within a window of 2 weeks. This is because it’s important to be able to measure and monitor your outcomes and quickly change your variables. It’s much easier to do this on digital platforms as trying to replicate real world experiments opens you up to a lot of variables and externalities that cannot be easily mitigated or measured.