Doing nothing is the biggest risk of all

Your experimentation programme needs a risk profile

Risk can be a competitive advantage in your experimentation programme.

If you’re taking more calculated risks than your competitors, you’re going to get better results. But to do that, you need to understand your risk profile.

Now some stakeholders will think that any experiment – no matter how small – is a huge risk. But for most A/B tests, your risk is limited. There’s the cost of building the test and occasionally a potential drop in performance during the test. But normally that’s it.

In fact, there’s actually much more risk in making changes to the website without testing them.

Your experimentation programme already has a risk profile.

Every experiment you run is low risk, high risk, or somewhere in between. And if you’re not consciously managing that balance, you’re probably not getting the full benefit of experimentation.

In this post, we’ll look at how to define risk in your experimentation programme – as well as three techniques to create better high-risk experiments.

What are low- and high-risk experiments?

Experiments can be low risk. You might have run similar tests in the past, and be pretty confident that this one will work. (Or at least confident it won’t break anything.) Here’s an example.

They can be medium risk. You might be trying out a completely new untested hypothesis. It could work – or you might have wasted time building the test, and lost money running it.

Or the experiments can be high risk. This is when you test disruptive ideas. Experiments like this are high risk because you’re risking the cost of building it and the potential loss of money while the experiment is live.

But it goes further.

There’s also a risk outside of the experiment.

It might affect the audience in the experiment long after you stopped it – or it might have implications on the brand as a whole.

Low-risk experiments exploit, medium ones explore, and high-risk disrupt

Low-risk experiments are typically iterative – you’re building on an already-proven concept. Their role is to exploit: you’ve validated a lever and are now looking to maximise its impact across the customer journey. The only potential loss is the cost of building the experiment (just because it worked once, doesn’t mean it’ll work again).

Medium-risk experiments are typically innovative – you’re testing out new concepts (but not necessarily radical ones). Their role is to explore: you want to understand what drives customer behaviour, and an experiment will inform that understanding. As before, the potential loss is the cost of building the experiment – but you may also lose money running the experiment, if it lowers performance.

High-risk experiments are disruptive – not only are you testing out something new, there’s a chance that it could fail miserably. These are the concepts that your competitors are probably too nervous to test – but they could deliver you a significant competitive advantage if they work.

Their role is to expand – to widen your approach by testing radically different ideas. But the risk is greater too. There’s potential for non-controlled impact – essentially, where the damage doesn’t stop when the A/B test stops.

Experimentation allows us to test anything we want – and to limit the fallout. It derisks innovation.

Take the screenshot above from Wistia’s pricing page. Testing a new pricing structure is a high-risk experiment: it could significantly increase revenue, or it could lower it. And potentially it could affect customers who aren’t in the experiment, it could be reported on social media or wider, and so on.

But often these high-risk experiments come with the highest reward. These are the ones that help you move beyond competitors.

Work out your experimentation risk profile

Look at the experiments you’ve run in the last 6 or 12 months, as well as your backlog of upcoming experiments. Then rate each as low, medium or high risk.

Of course, the definition of risk in your organisation will be different to mine. So come up with a simple format that works for you.

If you like, you can try a series of questions like this:

• What type of change are you making? eg UI, functionality, pricing, product.
• Have you tested a similar hypothesis before?
• If you have, was it successful?
• What’s the cost needed to build the experiment?
• What percentage of online revenue does the experiment affect?
• Might it change the behaviour of users in the experiment even after it’s stopped?
• Might it change the behaviour of users not in the experiment?

We’ve put this in a simple spreadsheet. You can answer all the questions and get a risk score straight away. Of course, you’ll want to adapt the questions and variables and scoring before you start. This is just an example:

Or if you want an even simpler alternative, just ask yourself this question about each experiment: “If I couldn’t run an A/B test, would I still make this change?”

If you’d still make the change, it’s almost certainly low or medium risk. If you wouldn’t, it’s probably high risk.

The importance of high-risk experiments

If we only test changes we’d make anyway, we’re wasting the opportunity of experimentation.

This is one of the most common mistakes people make in experimentation. They only run tests on changes that they’d make anyway.

It starts with an idea: “This seems like a good idea. Let’s test it and see just how right I am.”

Now there’s a good reason to test these changes. You might be wrong. Or some audience segments may respond differently. And if it is successful, it’s good to know the size of the impact – not just whether it’s positive or negative. This insight will help you come up with new hypotheses and prioritise your roadmap.

If I couldn’t run an A/B test, would I still make this change?

But it’s just as important to test changes that make you nervous. Disruptive experiments allow you to make bigger bets. “This experiment might crash and burn, but if it works…”

If you’re only testing best practice or patterns you see on competitor websites, you’re not going to be getting a competitive advantage. You’re going to be limiting yourself to the local maxima.

Experimentation allows us to test anything we want – and to limit the fallout. It derisks innovation.

Creating your risk strategy

If you used the Google Sheet above, it’ll show you what your risk profile looks like visually:

In this example, you’ll see that most experiments are blue (medium risk), with an equal balance of low- and high-risk experiments.

There’s no perfect answer for what your risk profile should be. Ideally, you’d have a balance of all three – and it should change over time.

So right now, your risk profile might look like this:

You’ve got an even balance of innovative and iterative experiments, with occasional radical experiments included to allow for greater leaps forward.

But if you’re in peak season, it might look like this:

You increase the iterative experiments to reduce the risk. Because iterative experiments have a higher win rate, you’re going to have a safer programme during peak season. That means you increase revenue without risking revenue at peak. And you might hold back on radical experiments altogether.

But if you’re just starting your experimentation programme then it might look like this:

You have an even balance across all three. You don’t invest too heavily in iterations, since you haven’t tested too much yet. And you balance innovative and radical ideas to get quick feedback as you develop your product and marketing strategy. (Of course, having this many disruptive experiments is dependent on having the right culture.)

How COVID-19 changes your risk profile

Right now is the best time to be thinking about your risk profile. COVID-19 has changed everything.

Some companies – food delivery, e-learning, home retail have seen a surge in demand. They should adopt the peak risk profile above, unless they’re still relatively new to experimentation.

But other companies have seen demand drop off a cliff. That means they could be more aggressive:

With demand dropping, doing nothing is the biggest risk of all.

Instead of doing nothing, or just iterating on the experiments that you’ve run previously, now’s the perfect time to try out the ideas that you were too nervous to do before.

This article by Stephen Pavlovich first appeared on Conversion.com, GO Group Digital’s exclusive partner and service provider for the U.K. and Ireland. Reach out to Conversion’s CEO and founder, Stephen, to learn your experimentation programme’s risk profile. Or get in touch!

Contributing Author(s)

• 
  Stephen Pavlovich Strategic Partner, GO > United Kingdom & Ireland