Market view: Testing the waters

Conducting experiments can be a useful technique for companies to find out what services their customers want and what they are willing to pay for them, says James Harvey.

One of the Competition and Markets Authority’s (CMA’s) remedies from its energy probe is a recommendation that Ofgem should establish “…an ongoing programme to identify, test (through randomised controlled trials, where appropriate) and implement measures…” to address weak customer response.

A randomised controlled trial (RCT) would involve Ofgem working with suppliers to establish a control group who are given a retail ‘placebo’ (that is, current bill presentation) and one or more treatment groups who are given a retail ‘drug’ (new bill presentations). You then watch what each group does to see if it has any effect on customer engagement.

Although the use of RCTs is not new, they are only just finding their way into the competition and regulatory toolkit/parlance proper. Ofwat has stated it expects water companies in PR19 to consider “…thinking about more innovative and frontier-shifting approaches to customer engagement…”

Example: an online experiment in relation to energy retail

Around 3,000 respondents were given a time limit to choose the cheapest tariff from four options. One of the options was preselected and respondents had to decide whether to stay on the preselected tariff or choose another. The annual cost of the four tariffs did not change from one respondent to another, but different respondents were shown different labels and descriptions of the preselected tariff. The labels and descriptions are listed below.

• Labels: current tariff; default tariff; out-of-contract tariff.
• Descriptions: blank; “you could make savings by choosing an alternative tariff”; “other people like you have made savings by choosing an alternative tariff”; “you could be missing out on savings by not choosing an alternative tariff”.
• The main results of the experiment were as follows:
• 38 per cent of respondents did not choose the cheapest tariff;
• Alternatives 1 and 2 were the cheapest tariffs and had the same annual cost – but nearly 30 per cent more people chose alternative 2 than alternative 1 (35 per cent versus 27 per cent). This could be because respondents were drawn to the mention of the direct debit discount;
• The label given to the preselected tariff made a material difference to how many people decided to stay on it – 57 per cent more people stayed on it when it was labelled “current tariff” than when it was labelled “out-of-contract tariff” (22 per cent versus 14 per cent);
• However, the extra switching from the out-of-contract tariff would not necessarily be a good thing for those customers if they made the same decision in real life – the extra switchers tended to go to the dearest of the four options.
• The description given to the starting tariff had little effect on switching rates.

Customer research techniques

There is a wide range of techniques that sit on a “what people do – what people say” spectrum. At the far end of the “what people do” spectrum are RCTs. At the far end of the “what people say” spectrum is a stated preference survey.

To date, relatively little has been said about the usefulness of experiments. Experiments sit between surveys and RCTs and, in doing so, share some of the benefits of both:

• Like surveys, experiments do not require the actual implementation of a service or communication change. Therefore they avoid some of the costs and challenges of making such changes, and arguably, allow more radical options to be tested. Also like surveys, they can be implemented face to face in a lab environment or online.
• Like RCTs, they involve defining control and treatment groups so that cause and effect can be thoroughly investigated and including incentives that are intended to go some way to mimic the incentives that customers face in real life and therefore encourage real life-type decision-making (for example, finding the best deal for them, but without spending too long on it).

The experiment was motivated by the CMA’s recommendation that Ofgem and suppliers should undertake RCTs to work out how to increase customer engagement. It shows there is a viable and useful middle ground between standard surveys and RCTs. This middle ground should be considered by Ofgem and companies to help narrow down the range of options for trialling in the market and in doing so help to manage some of the direct and indirect costs of RCTs. Other companies, such as new entrants, could consider using similar techniques to work out how best to communicate with potential customers to attract them to their services.

In the water sector, we anticipate much discussion about what is the right tool for the job in the lead-up to PR19, particularly in the context of estimating customers’ willingness to pay for investments and setting outcome delivery incentives. In this context, RCTs are unlikely to represent a sensible research strategy for many service areas, because they require the investment under consideration to take place first. But companies should consider whether experiments can and should be used.

Another key issue in the water sector is the introduction of competition to the non-household market. The lessons from the energy sector suggest that a key determinant of the success of this move will depend on how engaged customers are and this, in part depends on how companies engage them.

It is positive that regulators and companies are actively considering new techniques to understand what services their customers want and what they are willing to pay. The number of options for companies is greater than ever and the benefits of experiments mean they should definitely be on the radar.