By Angela Godwin
Today's water utilities are realizing great benefits from technological advancements in automation. SCADA systems, PLCs, RTUs — a veritable alphabet soup of modern conveniences — help water and wastewater utilities maximize resources, conserve energy, monitor operations, and track trends. But the very tools that hold so much potential for operational improvement are also a source of growing apprehension, surely keeping more than a few plant operators awake at night.
Industrial control systems like SCADA, PLCs, and RTUs help water and wastewater utilities maximize resources, conserve energy, monitor operations, and track trends.
The disquiet stems from the increasing awareness of inherent weaknesses in industrial control systems (ICS), weaknesses that could be — and have been — exploited by individuals with malicious intent.
One oft-cited incident occurred in Australia in 2001. A disgruntled former employee of a SCADA software vendor successfully hacked into a Queensland wastewater treatment plant's system, releasing 264,000 gallons of raw sewage into local rivers and parks.
In another incident in 2006, a foreign hacker found his way into the system of a Harrisburg, PA, water treatment plant through an employee's remote Internet access to the system. He was attempting to use the system to distribute malware.
It's not difficult to imagine the havoc that could be wrought by a motivated 'hacktivist' with keys to the water kingdom. But, acknowledging the threat of cyber attack is not the issue; it's how to protect against it that inspires much debate...and confusion.
Challenges
Industrial control systems are meant to ensure reliability, facilitate interoperability, and increase safety.
"All of those go in the opposite direction of security," said Joe Weiss, managing partner at security consulting firm Applied Control Solutions. "So if you want to have a secure system, that by its very nature makes the system less flexible and less usable."
One of the challenges, Weiss explained, is that ICS security is often left to the discretion of the utility's Information Technology (IT) department. IT personnel are well versed at securing business systems against cyber threats, installing Microsoft patches, updating virus definitions. They are keenly aware that their business users need simultaneous access to the network, the Internet, the email client, and other systems — and they have strategies and protocols for managing that. But industrial control systems are different and, as such, require a different approach to security.
"The concept of control systems is not to provide complete access everywhere," said Grant Van Hemert, an automation and control applications engineer for the Schneider Electric Water & Wastewater Competency Center. "But it is to manage the flow of information traffic and even to prevent it from going to certain places."
In recent years, a number of ICS vendors have been associated with vulnerabilities in the "back doors" into their systems. Take for example the highly publicized Stuxnet virus, an extremely intelligent piece of malware that was transmitted via USB. The virus was programmed to look for a very specific type of programmable logic controller (PLC) on a specific plant floor running a specific type of procedure. If it didn't find it, it would lie dormant. When it did find it — in this case at an Iranian uranium enrichment facility — it sped up the plant's centrifuges to the point of early failure. Nearly a thousand centrifuges failed before the virus was discovered.
"A PLC manufacturer like ourselves, we have to close these gaps," Van Hemert said. "There's no two ways about it. But at the same time, cyber security in the control system needs to be a layered approach."
Defense-in-Depth
The layered approach he referred to is called the "Defense-in-Depth" strategy. It's one of the security tactics suggested in the 2008 Roadmap to Secure Control Systems in the Water Sector, developed by the Water Sector Coordinating Council (WSSC) Cyber Security Working Group (CSWG) with the support of AWWA and the Department of Homeland Security (DHS).
Physical security, such as locking control panels in a cabinet, can help reduce the possibility of intentional or accidental cyber incidents.
Defense-in-depth takes into account the fact that no single security product can adequately protect an ICS. Rather, a properly configured combination of security technologies, controls, and policies is required.
"You have to think of cyber security as a chain and it's only as strong as its weakest link," said Bill Phillips, a senior control systems technologist with water and wastewater consulting firm CH2M HILL. "That's where the defense-in-depth approach comes from."
But for many water utilities — particularly smaller systems with little funding and limited personnel — building a strong cyber security chain is a daunting goal. There are myriad sources of information available, but they are spread out and fragmented, making it very difficult to know where to start.
Assess the System
Phillips is often tapped to advise his clients on ICS security.
"The first thing you do — the number one recommended step in improving cyber security — is to assess the current system," he said.
It's important to identify what the weaknesses are, as well as the impact of an attack on those weaknesses.
"Maybe it's something we could deal with even if it happened every day. Or maybe it's something that would put us out of business for two weeks," he said.
It's important to understand the level of risk you're facing. To do this, Phillips recommends CSET, a free tool created by the Department of Homeland Security's (DHS) National Cyber Security Division's Control Systems Security Program (CSSP).
CSET, the Cyber Security Evaluation Tool, is a downloadable executable file that's continually updated and available online (www.us-cert.gov/control_systems). It guides users through a step-by-step process to assess their control system and IT network security practices against recognized industry standards from organizations such as National Institute of Standards and Technology (NIST), North American Electric Reliability Corporation (NERC), International Organization for Standardization (ISO), U.S. Department of Defense (DoD), and others.
Based on information provided by the utility, the tool generates a prioritized list of recommendations for improving the organization's industrial control cyber systems. The recommendations come from a database of cyber security standards, guidelines, and practices and each one is linked to a set of actions that can be applied to enhance cyber security controls.
"We've worked really hard with the National Cyber Security Division to promote the CSET tool," said Kevin Morley, AWWA's Security and Preparedness Program Manager.
Granted, the tool isn't perfect: a specific unanticipated threat will likely not be in the domain of what's considered in the standards database.
"But it's a good starting point," he said. "And we advise our members to use it and have included it in guidance and things of that nature."
An ICS should allow an operator to tell a pump to start or stop but should not allow him to change a control strategy.
And CSSP also offers onsite training and guidance at no cost to utility owners.
"DHS has made funding available to actually assist the utilities in conducting these assessments," said Phillips. They provide varying levels of assistance, but as a minimum, they'll spend a full day with the utility.
"So if you've gone through and used their tool and done a good job of documenting it, they can sit down with you for a day and go through it and actually give you a good quality review and endorse the results or make recommendations of other things you can do."
Phillips said it's something that isn't done often enough, but really should be. He's hopeful that more utilities will take advantage of the service as they learn of its availability.
Segment the Network
Internet protocol (IP) provides the ability to connect anything to anything, meaning network equipment includes all the features for doing that: ports, drives, switches.
"So every device you connect to an EtherNet port or EtherNet device is vulnerable to whatever is going on at the other end of that wire," said Phillips.
Industrial control systems are different from business systems and, as such, require a different approach to security.
He advises his clients to segment their networks. "Make sure you've got the traffic organized on the network so that the control system components are all talking to each other because they need to, but they aren't chatting it up with Facebook, or business applications, or people on the network who don't even know what a control system is."
Within the network, you limit the conversations to specific topics, he explained. If the SCADA HMI (human machine interface) is talking to a PLC, there are very few things they need to be talking about. The HMI asks the PLC how things are going; the PLC responds back. The HMI relays the operator's command.
"Commands from the HMI and status from the PLC. That's it," he said. "If there are other things going on, there's probably a problem."
Limit Access
Access control goes hand in hand with segmentation, Phillips said. "Whenever there's a connection to your network from another network, you want to know who it is and why they want to connect."
It's also important to have access control for the people that use the system.
"If an operator is sitting at an HMI computer, then you want the operator's actions to be limited to the things operators do," he said. The operator should be able to tell a pump to start or stop, or initiate a controls strategy, but he shouldn't be able to change the control strategy or change the operations.
"You want role-based access control within the organization," Phillips said. "And you also want to provide role-based access control to anyone who accesses it from outside."
Strong Authentication
It may sound elementary but creating a strong password can go a long way toward strengthening your first line of defense.
"If your password is 'Password1,' you're going to be vulnerable," said Morley. As in any business, passwords should be sufficiently complex and changed regularly.
"But it's got to be integrated into the business culture," Morley said. "You can provide all the guidance you want but it if somebody doesn't apply it, it doesn't do much good."
Hardened Components
Hardening network components means locking down functionality to prevent unauthorized access. It means disabling any unused functions and ensuring that configurable options are set to their most secure levels.
"This is something the International Society of Automation is supporting with its new ISASecure program," said Phillips. Like the ubiquitous UL label, the ISASecure designation signifies that an ICS product adheres to industry-accepted cyber security specifications.
"They're going to put this logo on equipment so that when you pick it up, or when you write a specification, or when you get ready to install it, you can see it's been certified," he said.
Physical Security
Along with hardening components, there is a certain amount of physical security that should be considered for ICS.
"You should think about, literally, taking the computer that's on the floor in your control room and locking that off in a cabinet," said Van Hemert. Similarly, you can block off USB ports, put padlocks on control panels, and incorporate EtherNet switch control and mapping to manage traffic.
"If you do that, then there's a good chance your security procedures may be able to protect against a threat that hasn't even occurred yet," said Van Hemert.
These are just a few of the basic precautions that water and wastewater utilities can take to begin strengthening their cyber security. But perhaps the most important strategy — the one that sets the stage for all the others — is also the most difficult.
Culture Shock
"There's a certain amount of culture that needs to be integrated into any business — whether a bank, or a home network, or a water facility," Morley said. To truly become a priority in the water sector, cyber security must become part of the culture.
Phillips likened it to the evolution of safety.
"Basically over the years, safety has evolved into a common practice, engrained in business procedures," he said. Security and cyber security are they same way. "They need to be treated like safety," he said. But it's a painful, time-consuming process, and water utility personnel have plenty of other tasks to keep them busy.
"It's hard to get your head around all the cyber stuff," said Morley. "You can't see it."
Without the drive from above, it's difficult for water and wastewater utilities to justify diverting funds and resources to a problem they can't see.
"It all goes back to the C-level executives," said Weiss. "Things aren't likely to change until they mandate that the control systems must be as secure as the business systems," he said.
"Most of the clients that I talk to are keenly aware of the problems and they would love to address them," Phillips said, "but they have all these competing needs, a lot of them with more visible results than improving cyber security."
But it's still important to go through the process of developing cyber security procedures, he said. "And then once those policies and procedures are in place, they must be woven into human resources procedures so that it becomes part of the organization's culture."
Phillips also advised conducting ongoing training so that a) your staff will know what the issues are and b) they'll know that you're serious about it.
Control systems deployed in the water sector — just like control systems deployed in other critical infrastructure sectors — have provided enormous gains to water and wastewater utilities in terms of improving efficiency. But the reality is that there is no way to completely secure a control system or any other network. Whether it's connected to other networks or not, if someone is dedicated enough to get to it, they'll find a way.
"So you can't make it bulletproof but you can drastically improve it," said Phillips.
"There's a lot of power and a ton of benefits to putting your plant online," said Van Hemert. "And there is a lot of risk to be managed — but there are things you can do to manage it. That's the underlying theme."
In the Event
If a cyber threat is detected, do you know where to go for information? There are two main sources you need to be aware of: ICS-CERT and WaterISAC. These are widely recognized as the most credible sources of information which is extremely important as evidenced by last fall's highly publicized and wrongly identified "attack" on a pump system outside of Springfield, IL. Early and conflicting reports indicated that the utility's system had been hacked from an IP address in Russia. As it turned out, it was a vendor accessing the system for legitimate reasons while traveling abroad.
ICS-CERT is part of HSIN, the Homeland Security Information Network. It's a free resource managed by DHS under its National Cyber Security Division's Control Systems Security Program (CSSP). It's a general clearing house for all types of ICS cyber threats, not just those affecting water. So you may have to sift through some irrelevant information but if a credible cyber threat is detected, this is where it will be posted. To sign up, visit: www.us-cert.gov/control_systems/ics-cert/
WaterISAC is the Water Information Sharing and Analysis Center, a private, subscription service created and managed by the water sector to keep drinking water and wastewater utility managers informed about potential risks to the nation's water infrastructure from contamination, terrorism and cyber threats. There are two levels of service: WaterBASIC (free thanks to EPA funding) and WaterPRO (paid). The advantage to this resource is that it's water-specific but subscription fees vary depending on the type of user you are (public or private utility, population served, etc.) and may be cost-prohibitive. For more information, visit: waterisac.org.
Utility's Believe It or Not
"Not all cyber intrusions are malicious in intent," pointed out Van Hemert. The vast majority of cyber "attacks" are simply unintended consequences of other actions.
Take, for example, the story of the water utility employee working a long midnight shift manning the SCADA system. Out of boredom, he decides to load up a video game on his computer, which depletes his system resources and crashes the computer.
Although he did end up bringing down the system, it wasn't characterized by malicious intent. But, as Van Hemert points out, had the computer tower been locked away, he wouldn't have been able to load the computer game in the first place.
Industrial control systems are different from business systems and, as such, require a different approach to security.
Certified Secure
The ISASecure program was developed by the ISA Security Compliance Institute (ISCI) to accelerate industry-wide improvement of cyber security for Industrial Automation and Control Systems (IACS).
ISASecure Embedded Device Security Assurance Certification (ISASecure EDSA), the first ISASecure certification, focuses on security of embedded devices and addresses device characteristics and supplier development practices for those devices.
In November 2011, Honeywell was the first industry vendor to earn the ISASecure EDSA certification for an industrial control systems product, recognizing Honeywell's commitment to stringent cyber security standards and the security of Honeywell's control systems products.
Within the network, Phillips recommends limiting conversations to specific topics. "Commands from the HMI and status from the PLC. That's it," he said. "If there are other things going on, there's probably a problem."
More WaterWorld Current Issue Articles
More WaterWorld Archives Issue Articles
