Demand reduction measured in last winter’s DFS is ‘too optimistic’

The Electricity System Operator’s methodology for determining demand reductions from customers participating in last winter’s Demand Flexibility Service was “too optimistic”.

According to analysis of the results of Octopus Energy’s Saving Sessions by its research facility, Centre for Net Zero, demand reductions from the supplier’s customers were “overestimated” by 13%.

The Demand Flexibility Service was first introduced by the Electricity System Operator (ESO) in autumn last year as one of a series of contingency measures to address concerns over the security of fuel suppliers for gas generators. Octopus Energy marketed the service to its customers in the form of Saving Sessions.

The ESO said around 1.6 million households and businesses participated in the service and they officially delivered more than 3.3GWh of demand reduction. Of this, Octopus Energy’s customers accounted for 1,863MWh, according to Centre for Net Zero.

There were 22 events in total – 20 tests and two live events. Octopus Energy participated in 13, including both of the live events.

The ESO set a minimum price for payments to suppliers of £3/kWh, which was also the clearing price for all test events. Of this, Octopus Energy paid £2.25/kWh to customers participating in the test events. Prices were higher for the two live events. Across all the events, the ESO paid suppliers a total of £11 million, of which Octopus Energy received £6 million.

Under the ESO’s methodology for determining demand reductions, each customer’s unadjusted baseline consumption was calculated by averaging their usage during the same half-hour period on the 10 most recent workdays prior to an event, excluding days on which there was another event. This was then adjusted based on their consumption in the three-hour period prior to the start of the event in question. Their consumption during the event was subtracted from their adjusted baseline to calculate the amount of demand reduction they delivered.

The Centre for Net Zero instead sought to determine the amount of demand reduction delivered by Octopus Energy’s customers by comparing the consumption of different groups.

In total, Octopus Energy invited 1.384 million customers to sign up to participate in its Saving Sessions. Of these, around 375,000 signed up by the first event. Once signed up, customers were sent notices in advance of events asking them to explicitly agree to participate in individual saving sessions.

A number of customers were excluded from the analysis for reasons including a lack of sufficient data. The consumption of three sets of groups were compared:

• Around 332k Octopus Energy customers who were invited and signed up by the first event on 15 November 2022 versus 654k Octopus Energy customers who were invited but never signed up
• Around 332k Octopus Energy customers who were invited and signed up by the first event on 15 November 2022 versus around 12k Octopus Energy customers who were invited but signed up after the ninth event on 30 January 2023
• Around 1.137 million Octopus Energy customers who were invited versus around 198k Bulb customers who were not invited due to Octopus’ acquisition of the supplier in December 2022

The analysis found that signing up to the Saving Sessions reduced customers consumption during the events by around 26%, while opting-in reduced consumption by 40%.

Importantly, the study said these rates suggest Octopus Energy’s customer base reduced their consumption by a total of 1,642MWh across the events, 13% less than the ESO’s measurement. It said this indicates an “upward bias” in the ESO’s “before-after” methodology, which is “too optimistic”.

Maria Jacob, one of the report’s authors, told Utility Week the main reason for this difference is consumers who had actually increased their consumption during the sessions according to the ESO’s methodology were instead recorded as having delivered no reduction so “they were not getting penalised not being able to turn down. In that way, you’re overestimating the amount of reduction that’s coming from the domestic sector.”

Earlier this year, Utility Week reported that some customers were earning large sums of money per session by artificially raising their consumption during the in-day adjustment period to game the service. The ESO subsequently decided to remove the in-day adjustment for the second iteration of the service to prevent this from happening.

Jacob said the Centre for Net Zero’s analysis did not find any evidence of gaming having a measurable impact on the demand reduction seen during last winter’s events.

The analysis also explored the effect of different notice periods and incentives on demand reduction during two particular sessions.

For the event on 13 February 2023, a technical fault meant some customers received opt-in notices on the day of the event, rather than the day before as was usually the case. Although the group that received late notice was not random – depending on how long they had been an Octopus Energy customer – the Centre for Net Zero used the resulting data for a “quasi experiment.”

Those who received the notice overnight were excluded from the comparison, but the demand reduction from the other customers who received an opt-in notice on the day of the session was a quarter less than was otherwise typical for the session. The analysis also found that for customer receiving opt-in notices between 8am and 1pm on the day, every hour closer to the event, their demand reduction was on average 10% lower.

This was followed by a “field trial” in which around 651,000 customers were sent opt-in notices on the day of the event on 14 March 2023. A subset of around 19,000 customers were additionally sent a “head-up” notice the day before, whilst another 19,000 were also told they would receive a one-off bonus payment of £1.25 for participating.

This experiment also found that short notices periods dampened demand reduction. The reduction from customers who did not receive a heads up email was 7% lower than those that did. Their opt-in rate was also lower. The customers who were offered a bonus payments reduced their consumption by an additional 4%.

National Grid ESO held its first test event for this winter’s service on Thursday last week (16 November).

The day beforehand Octopus Energy revealed that one million customers had already signed up for its Saving Sessions. The equivalent figure for last winter’s service was around 409,000 (note: this includes customers who signed up without being invited, who ultimately numbered 138,000).

Participation in Octopus Energy’s Saving Sessions