Open data – why it matters and why it’s hard

Candid discussion at a recent virtual round table debate hosted by Utility Week, in association with Talend, showed a growing will to embrace open data principles among utilities is being stymied by doubts over internal data quality and consistency.

Key points from the debate:

• Open data is a must for an effective response to climate change
• Open data can help unlock step changes in operational efficiency
• Open data must be underpinned by a common understanding of data definitions and categorisation as well as high standards for data quality and consistency
• Without open data, the full value of digitalisation will not be accessible to utilities

The term “open data” is trending highly in the utilities sector today. Last year it came to prominence in the energy sector with the publication of the Energy Data Taskforce’s recommendations for the creation of system-wide asset registry and adoption of “presumed-open” principles for asset data. And earlier this month, Ofwat underlined its will to see open data principles adopted in the water sector in its updated consultation on the introduction of a new £200m innovation fund.

This surge of interest in open data is driven by a growing realisation that monumental challenges facing utilities, including climate change and supporting continuity of supply to growing and urbanising populations cannot be tackled with traditional operating models or the limited insights available from data held by individual organisations.

And, on a more pragmatic level, many utilities are also waking up to the fact that the step changes in operational efficiency expected by regulators during upcoming price controls are only likely to be accessed through greater collaboration between asset owners and a willingness to collectively revisit well-worn use cases for asset data.

This realisation has been buoyed by successful open data initiatives in other sectors, such as banking where FinTech innovators have been empowered by open data to transform operating models and, in turn, consumer value from the products and services on offer.

Meanwhile sectors such as agriculture and transport have also made moves to embrace “data for good” and reaped innovation benefits.

Build trusted foundations to unlock open data benefits

For utilities, as with many other sectors, responding to the net zero challenge is going to require some very different thinking and action than has been the norm in the past.

But as the writer William Pollard once said , “Without change there is no innovation, creativity, or incentive for improvement. Those who initiate change will have a better opportunity to manage the change that is inevitable”…Read the rest of this commentary from Sean Crouch, sales director at data integration specialist Talend, here.

The opportunity for utilities to follow the same path, accelerating their transition to future operating models, accessing regulatory rewards and bringing about social good in the process is enormous. But as a recent virtual meeting of technology and businesses highlighted, it’s an opportunity that not all in the sector are ready to share in.

The virtual round table event hosted by Utility Week, in association with cloud data integration specialist Talend, highlighted that while most technology and data specialists in the sector understand the potential value of open data, many do not feel their organisations are ready for it.

This lack of confidence comes down to common concerns about the quality, consistency and completeness of data held within their organisations – and concerns about the readiness of suitable data governance processes for exposing data to third parties.

The discussion, which included participation from electricity, gas and water network operators, emphasized that system-wide benefits from open data must be underpinned by some essential ground work within individual utilities to create confidence in the veracity of data, or risk major “kickback” from third parties seeking to “make informed decisions and develop use cases”.

What also quickly became clear as experiences and insights were shared among the group is that some companies and sector segments are considerably further on with this groundwork than others.

In the electricity distribution space for example for sector-wide drive to transition to so-called “DSO” [Distribution System Operator] business models has provided a foundation for collaborative work, coordinated by Electralink, on an open data portal.

The purpose of this is to pool industry data on assets, markets and flexibility provision in one place and in a standardized format in order to facilitate innovation and easier interaction with an increasingly complex set of system stakeholders.

The impression from other round table participants however, was that this kind of work is significantly ahead of anything in the gas and water sectors, where in the main companies are still grappling with more fundamental data quality, consistency and governance challenges which make them nervous of the idea of placing large data sets out in the open.

Many participants also admitted that they are struggling to put together business cases for open data initiatives which can gain board level backing. Even for the power networks involved in the debate the challenge of gaining executive buy into the “speculative benefits” of open data can still be tricky.

One participant from a power distribution operator commented: “Getting that senior level support is a challenge because while we can see some potential use cases in opening up certain data sets, you have to be able to build a business case. It’s hard to create a case for them to spend money on expensive system configuration for open data and undertake risks, in the expectation that maybe a use case will emerge that will outweigh the costs incurred.

The same participant continued to say that boards are “beginning to get their heads around this – we’ve come a long way, but there is a long way to go.”

Such observations led to vigorous discussion of the need to tie open data benefits for regulated utilities to specific regulatory performance mechanisms and incentives. Several water companies agreed that their business plans for the new price control included strategies for the application of data to help them meet Ofwat’s stretching ODIs [Outcome Delivery Incentives].  One company said it has submitted a strong case for open data as part of its approach to accessing regulatory rewards.

Meanwhile, energy network representatives also talked about the importance aligning open data business cases with clear regulatory performance benefits, from customers satisfaction to improved performance on interruptions and customer minutes lost. While cross industry performance against regulatory measures for these things is high, it’s been made clear that Ofgem expects the industry to uncover room for significant further improvement and efficiency during RIIO2 in order to “make room on customer bills” for the additional costs which will inevitable be incurred by investments in aid of the low carbon transition.

To accelerate progress towards open data maturity and all the benefits it could entail for companies and society, many participants in this discussion hoped that greater use of regulated innovation funds to support open data initiatives will soon be seen.

In energy, power distribution operators have recently collaborated on a submission for Network Innovation Competition funding for a collective open data project (linked to the above-mentioned open data portal). The submission is the first of its kind made under the scheme, but it was felt there will likely be more projects soon, especially as the gas sector moves to catch up with power network peers.

There was a keen interest from water sector participants in learning from the experience of energy sector contemporaries in this area in order to plan robust bids for support of open data programmes under the new Ofwat innovation fund.

While such bids are formulated however, this discussion showed clearly that there is a great deal of work many firms can focus on progressing internally in order to tackle people, process and technological barriers to open data. In short, there is work to be done to ensure that when more companies “lift the lid on the information we hold” that it is in an appropriate state and that incoming data can also be easily received.

Only when this has been achieved will companies be in a position to reap the full benefits of digitalisation and trigger the innovation needed to take access hard to reach efficiency gain and generate crucial social and environmental value.

Participant quotes:

“Opening data open doors to innovation that we could never imagine. There are people out there who are interested in our data even if they don’t have clear idea what they want to use it for. They can take or data with no preconceptions about how it can or should be used and come up with methods and use cases that we could never conceive.”

“Utilities are not at the cutting edge of data management practices. We’re fortunate in that we deal with very structured data within established systems of record. But most of us have complex systems architectures and the data architecture associated with that can be complex…that present challenges when you want to open data up.”

“Opening up data is not just about being altruistic and supporting the innovation of others. Opening up data so that utilities have greater information and insight into each others networks has big benefits – interoperability of utilities is key to whole system awareness and efficiency.”

“If we don’t get definitions and categorisation right to start with then the data will be all over the place. Open data portals are great to aspire to but many companies have a lot of leg work to do on getting data into the appliance and quality mode required.”