ESO to develop AI solar forecasting service

National Grid Electricity System Operator (ESO) has partnered with the non-profit startup Open Climate Fix to develop a “first-of-a-kind” near-term forecasting service for solar generation using artificial intelligence.

The ESO said the “nowcasting” service will allow its control room to minimise the amount of fossil fuel generation held in reserve to cover swings in solar output as clouds move across the sky, lowering both emissions and costs.

ESO head of innovation strategy and digital transformation Carolina Tortora said: “Accurate forecasts for weather-dependent generation like solar and wind are vital for us in operating a low carbon electricity system. The more confidence we have in our forecasts, the less we’ll have to cover for uncertainty by keeping traditional, more controllable fossil fuel plants ticking over.

“We’re increasingly using machine-learning to boost our control room’s forecasts, and this latest nowcasting project with Open Climate Fix – whose work could have real impact for grid operators around the world – will bring another significant step forward in our capability and on our path to a zero-carbon grid.”

Jack Kelly, Open Climate Fix co-founder and a former researcher for Google’s artificial intelligence company DeepMind, said their work will borrow ideas from fields such as “natural language processing, where you get machines to do things like translate from English to French, and from protein folding, where you try to get computers to understand the 3D structure of proteins from the genetic sequence.

“Weirdly enough, the techniques that have been developed in those domains are very likely to be applicable to energy forecasting.”

He said one of these ideas is “self-attention, where the model learns the relationships between the different inputs.”

“In energy forecasting, we hope that technique will be really helpful, for example, for identifying which clouds in the sky are having the most impact on the sunlight that’s hitting the surface,” he explained.

Their model will learn how to read satellite images updated every five minutes to work out how and where clouds are moving in relation to the solar arrays below and how this impacts their output.

“Somewhat counterintuitively, it turns out it’s not just the cloud that’s immediately in the line of sight between the solar system on the earth’s surface and the sun,” Kelly told Utility Week.

“Because the sunlight scatters around the atmosphere a lot, other clouds that aren’t in that direct line of sight can have quite a large impact on now much energy the solar system can actually produce.”

Kelly said the service will seek to provide for forecasts for up to a few hours ahead. He said conventional weather models struggle to provide accurate forecasts at such short notice: “One of the issues with these big conventional weather forecast models is that even on huge supercomputers they take in the order of a few hours to run.”

This means by the time they have produced a forecast it is already slightly stale: “It won’t have actually looked at the state of the real world for two hours. And that’s fine for things like temperature and probably even wind speed but certainly for clouds, they can change quite a lot over two hours so one big advantage of machine learning is that it should take in the order of a few minutes to run, rather than a few hours.”

He said improved near-term forecasting will allow the ESO to better manage its reserves: “We need lots of that spinning reserve because maybe a cloud will come along and cover Cornwall and wipe a gigawatt of solar generation.

“That gigawatt has to be replaced almost instantly and its mostly up to that spinning reserve to fill in that gap. And so, if we have better forecasts, even for a few hours ahead, National Grid ESO could trim back the amount of spinning reserve that they have on the system.”

Kelly estimated that improved solar forecasting could save the ESO between £2 million and £10 million per year and cut carbon emissions by around 100,000 tonnes annually – an amount that will “go up considerably as we get more and more solar on the system.”

He said the service could additionally benefit operators of solar farms and batteries, helping the former to sell their power on the intraday markets and the latter to also schedule charging.

Kelly said all of their work will be made freely available to others: “Everything will be completely open – all the codes and the model – and the reason here is if we can bring about as step-change improvement in forecasting accuracy, then the best way to spread that technology across the globe is to enable other forecasting companies to take technology and integrate it into their existing forecasting products.”

He said he is also interested in using the same techniques to forecast wind generation and demand, and has already begun working on the latter as consultant to the ESO, with machine learning being able to cut forecasting errors by around half: “And that’s still using a fairly small amount of input data, so we’re optimistic that if you come along with huge amounts of satellite data and the full fire hose of data that comes off the back of Met Office’s supercomputer, we can do even better at forecasting demand.”

In August, the ESO launched a project to improve its day-ahead forecasts for reserve requirements using machine learning.