Cybersecurity for Water Utilities

With the digitalization of the water sector come unbounded opportunities. However, the introduction of digital solutions is not without new and profound challenges.

Controlling assets, processes and whole systems digitally, and in many instances remotely, requires the existence of networks which, by their nature, can be vulnerable to malice.

Threats ranging from relatively benign (albeit unwanted) intrusion to far more serious acts of sabotage call for the development and implementation of protocols and systems to secure assets against interference. Such measures fall within the field of cybersecurity and their adoption is rapidly becoming a must-have for utilities charged with providing one of society’s most essential of services.

An Evolving Landscape of Cyber Threat

Kevin Morley, American Water Works Association (AWWA) manager for federal relations, works closely with a variety of organizations tasked with advancing the security and preparedness of U.S. critical infrastructure, including DHS/FEMA, EPA, and the Water Sector Coordinating Council.

Having facilitated the development of water sector standards and guidance for security and preparedness across a number of initiatives, Morley recently led the development of a water sector-specific approach to support voluntary use of the NIST Cybersecurity Framework.

“We can view the risk as function of threat, vulnerability and consequence. Those parameters are dynamic, and we’ve seen significant shifts across each in recent years. In my view the threat landscape has evolved dramatically as utilities’ threat surface has gone global,” he said.

“In the sense that many operating systems now have some degree of automation, this opens up exposure to attack from intruders who need not be physically present to cause either economic or physical disruption,” he added.

ICS providers are strengthening their offerings through collaboration with security experts. For example, global leader in engineering and industrial software AVEVA signed a partnership agreement with Virsec, a leader in cybersecurity innovation. Photo courtesy AVEVA.

Threat surface, a combined measure of points of vulnerability, includes industrial control systems (ICS), such as supervisory control and data acquisition (SCADA) systems, RTUs, operational technology (OT) and information technology (IT). Increasing reliance on these tools, along with adoption of industrial Internet of Things (IIoT), all represent new routes to interference.

“New technologies are certainly increasing threat surface and also heightening potential consequence of attack, particularly with devices that have direct control over processes,” Morley said. “Advanced metering infrastructure and energy management systems bring great benefits, but it’s critical to acknowledge the implications of these technologies in terms of supply-chain security.”

The simplest and most frequent threats remain economically motivated, explained Morley, highlighting ransomware for instance; but with new opportunities for hackers come new consequences.

“The big leap is that the attack space has evolved from enterprise, or IT, to include the targeting of OT systems — that’s game changing. Disrupting those processes is when things get serious, it’s not just emails at stake.”

Morley noted that specific vulnerabilities and incidence levels are difficult to speak about. “We never want to play into the hand of hackers,” he said. “Incidents, despite a rise in attempts, aren’t always reported, since they may be countered effectively. However, I know from speaking with utilities that there is an increasing amount of unwanted, potentially malicious, traffic.”

A corresponding sentiment is held by Michael Salas, chief information and digital officer for SUEZ North America, who said: “Cybersecurity is certainly being taken seriously by the water community. There’s no doubt amongst peers I speak with, and here at SUEZ, that water utilities are vulnerable and becoming more so.”

Contributing to that increased risk, according to Salas, is the digital transformation of the water sector. “More digitalization, more connected tools in the hands of service operators, field crews and customers — all of this means that a utility’s digital footprint is increasing and with that comes exposure.”

At SUEZ, as with many other utilities, growing awareness of the cybersecurity landscape is reflected in mandatory reporting on breaches, in assessments, talks with regulators and boards, and, as we shall see, investment.

Morley also acknowledged a growing awareness of a clear and present threat: “Ten years ago I would say the recognition of cyber threats was fairly abstract, but there’s now a significantly greater appreciation for the capability and reality of the risk. Cybersecurity has moved up in terms of areas of concern in utility leaders, with the majority indicating they acknowledge it as a critical issue.” In response, Morley said, AWWA has developed resources to support the implementation of recommended best practices.

ICS providers are strengthening their offerings through collaboration with security experts. Photo courtesy AVEVA.

Responding to Threat

Designated as critical infrastructure, the water sector has garnered attention from national bodies concerned with expediting cybersecurity, including DHS, EPA, and the National Institute of Standards and Technology (NIST). Bodies like AWWA, too, have been proactive in encouraging cybersecurity solutions, including facilitating appropriate changes in business culture, geared toward navigating the contemporary threat landscape.

For the water sector a key development has been the AWWA Cybersecurity Guidance & Tool, which is recognized by major water organizations, including the Water Sector Coordinating Council and EPA, as the sector-specific approach for implementing the NIST Cybersecurity Framework (CSF).

Introduced in 2017, the AWWA initiative provides a set of best practices and standards for improving the security posture of the water utility ICS. “The resource aims to help utilities understand what controls are most applicable to their operations and provides a baseline on where to begin,” Morley explained. “We put together use cases, for instance: Do you remotely manage your pump station? If so, these are the controls that should be in place to manage the risk.”

Morley described the tool as a low resolution, non-technical approach that enables utilities to review what controls are relevant to their operations, and identify the highest priority controls to have in place in a given use case.

Importantly, given utilities’ concern over financing of new systems which don’t inherently lead to a better bottom line, AWWA prioritizes the relative importance of controls, acknowledging when they might be implemented relative to budgets.

“This allows utilities to focus potentially limited resources on what will deliver the biggest return,” said Morley. “The goal is appropriate risk management not risk elimination.”

Embracing security

Hawaiian water utility Aqua Engineers has engaged with AWWA’s tool. David Paul, vice president and director of engineering at the company, said, “We see the threat as growing and evolving. I think it’s been overlooked in the past and [is] now gaining much more attention, although it’s still new to utilities. If we can keep the cybersecurity issues at the forefront of water and wastewater system concerns, we’ll move our industry to a safer place.”

Paul acknowledged that addressing cybersecurity was overwhelming at first. “We sought out consultant help and found that we didn’t always assess ourselves correctly,” he said. “Once we assessed properly and identified vulnerabilities in our SCADA system with aid of the AWWA framework, we created a list of prioritized system improvements.” As a result, the utility was able to reduce its attack surface and limit opportunities for hackers to do harm.

Paul noted that improving security while retaining the ability for operations staff to remotely access the wastewater treatment plant SCADA system (from their work computers, personal mobile phones, personal tablets, home computers, etc.) was one challenge they encountered.

“That access creates a lot of opportunity to attack our SCADA system in addition to the SCADA server’s direct access to the Internet,” he explained. “We want our operators to have remote access so they can assess alarms and situations quickly, but we have to find a way to enable that ease of access with more security.”

Part of the solution was to redirect all remote access through a single, secure authentication server that could also monitor and track activity. “In theory, this reduces the attack surface area because we reduced the number of possible entry points,” said Paul, adding that the AWWA cybersecurity tool “certainly creates foundations for sound cybersecurity practices and guides us toward opportunities for improvement and steps to improve security.”

At SUEZ, investments have been made in line with the NIST CSF, but Salas described other actions taken by the company too. “Something we’ve done differently at SUEZ is to have the one cybersecurity team managing both corporate IT and operational technology, or ICS. In many utilities, those two fields are managed independently.” Bringing it all together under the one umbrella, he said, means resources and expertise can be focused and the challenge addressed holistically. “That’s significant, as solutions require expertise that can exceed those of many site operators.”

With so many potential vulnerabilities and courses of action to take, how to allocate investment warrants careful consideration. Salas described the SUEZ approach as risk-based. “Wherever the risk is highest, that’s where we invest money and resources — regardless of whether it’s a new site or an existing one. Much of our investment goes into protecting the perimeter of both corporate IT and OT.”

Low-Cost, High-Gain

Morley rightly noted that even basic security measures, coming at little cost, can deliver substantial gains. “A notification from DHS earlier this year on Russian intrusion into critical infrastructure revealed substantial amounts about the threat landscape,” he said, “but perhaps most revealing of all, if you look at how intrusion occurred, there are some very low-cost measures that could have prevented it. Security doesn’t always mean major capital investment.”

It starts with staff. “Employees are your first line of defense,” said Morley. “Water utility security culture has to evolve — security cannot just be IT’s problem. Everyone with some kind of connectivity is part of the threat surface.”

That philosophy is echoed in the approach taken at SUEZ. “Another significant SUEZ investment, more in terms of time than money, has been in awareness training for staff,” said Salas. “I believe it’s key to protection.” Here, SUEZ has partnered with an outside company to provide staff with monthly, bite-sized cybersecurity awareness training packages based around short videos. “We’ve seen huge take-off in compliance rates since initiating the scheme around September 2017, and the feedback has been great.”

Awareness is crucial, he said. “One can invest a lot in protecting the perimeter, but at the end of the day if you don’t create awareness, employees may still make mistakes that undermine those other investments.”

Morley highlighted another low-cost, high-gain outcome grounded in business culture: mapping your system infrastructure. “As systems and networks expand, new features and potential vulnerabilities might not be mapped. But maintaining a clear system architecture is at the core of managing threat exposure,” he said.

However, when higher costs do enter into the equation, Morley contended that security is a necessary expense with a simple rationale. “Consider the cost of SCADA going down and what that means to mission continuity. If I can spend, say, tens of thousands to secure it to offset potentially millions of dollars in damages, is that a worthwhile investment?” Given the millions invested in it to begin with, Morley said the answer is almost certainly yes.

SUEZ has invested in employee security training facilitated in part by animated cybersecurity training materials developed by Ninjio, providers of gamified security awareness training. Image courtesy of Ninjio.

Upstream and Third-Party Solutions

While it’s clear that utilities must assume responsibility for their cybersecurity, it’s encouraging to note that stakeholders are being supported not only by guidance provided through the likes of AWWA but also via developments of upstream solutions actors.

ICS providers, for instance, are strengthening their offerings through collaboration with security experts. For example, in August, global leader in engineering and industrial software AVEVA signed a partnership agreement with Virsec, a leader in cybersecurity innovation, expected to deliver AVEVA customers improved cyber protection for ICS and SCADA systems.

Morley noted the trend, saying: “With multiple critical infrastructure sectors stressing the need for security, and embedded controls in commonly used systems in the supply chain side, there is more upstream ICS development in terms of high security systems.”

Salas acknowledged these developments too. “ICS vendors are taking the lead in providing new systems that readily enable good security,” he observed. These include easier software patching and fixing vulnerabilities more quickly. “However,” he added, “I think it’s going to remain the case that there’s another layer of high-level, cybersecurity-specific vendors providing additional security systems, whether it’s firewalls, intrusion detection, event monitoring, etc.”

Despite perceived assurance of on-site servers, cloud-based solutions from providers like Amazon and Microsoft are gaining traction. While such big-name players may be a safe bet in terms of security assurance, plenty of other third-party server providers might not be as reliable.

“Are third parties involved in your ICS systems also secure? I would urge utilities to check service level agreements with their cloud providers,” Morley cautioned.

According to Paul, Aqua Engineers is currently engaged in precisely this. The utility operates fifteen remote pump stations managed via a cloud-based SCADA service provided through a third party. “We’re currently seeking to develop a service level agreement with the vendor that provides more assurance,” he said.

All told, good cybersecurity not a single product, or even a secured system, but rather an operational disposition to be incorporated, holistically, into business planning and operations. Naturally there is a clear role for robust, physical security systems, but these must be bolstered by continuous and proactive engagement with maintaining security through business practice and employee culture. Above all, as threats evolve, so too must defenses.

A final word of caution: don’t be lulled into a false sense of security because you’re a small utility in the middle of nowhere. “From a physical threat security perspective, a utility’s obscurity or smallness may be a perceived benefit,” said Morley. But when it comes to cybersecurity, “the reality is, [it] makes no difference. If you’re online, you’re online — opportunistic hackers don’t care who you are, where you are, or what you’re doing. Obscurity is a false pretense in cybersecurity.” WW

About the Author: William Steel is a freelance reporter covering renewable energy, water and cleantech industries.