Disasters such as Superstorm Sandy and Hurricane Katrina have taught water utilities that emergency planning and well-maintained backup systems are critical to limiting the devastating impact of plant failures. But what are the components of a good plan and maintenance program? We posed that question to water system managers who survived both Sandy and Katrina, and they provided us with some surprising answers. And though it’s typical to think of emergency standby generators as the foundation of a backup system, experts from a variety of utilities showed us that there’s much more to consider, such as pumps, logistics, storage, and staffing. We’ll look at those issues, and also provide some examples of funding solutions for upgrades to your backup systems.

The coast of New Jersey took a beating from Sandy that remains fresh in Kevin Kirwan’s memory. Kirwan, senior director of operations for the coastal region of New Jersey, American Water, recalls that storm surges ran as high as 14 feet above the average for low tide. The flooding and power outages overwhelmed sewer treatment facilities and caused an estimated 11 billion gallons of sewage to pollute rivers and bays. Kirwan was right in the middle of the mayhem and says that for a plan to work in such extreme circumstances, it needs to have some basic components.

The first step is to make sure your plan accurately represents your system. Accurate system data may seem to be an obvious assumption, but if there’s institutional knowledge that’s left the business due to retirement or other reasons, a utility could be vulnerable. “We have very sophisticated networks, with about 9,000 miles of transmission lines, just in New Jersey,” says Kirwan. “It’s very complicated and over the years we’ve been looking at the emergency response plans that every system should have, and exercising those plans.”

At a minimum, Kirwan suggests that emergency plans be reviewed and exercised once a year, although twice a year is better. These exercises should be more than just table-top simulations. For example, American Water’s training program for training plant operators includes putting the trainees in actual emergency situations. “We will have an operator demonstrate their capability to run a facility or system when there’s a vital component failure. So they might have to run the plant without the SCADA system, or with standby power generation,” says Kirwan.

In extreme weather events, SCADA systems can be points of failure, and often it’s not the system, but rather, the telemetry equipment. Trees can fall on antennas, or power outages can disable other forms of communication.

Knowing the exact location of a point of failure can be critical, and Kirwan advises managers to look at the advantages of a comprehensive GIS system. “Sometimes it’s not always a generator that will solve the problem. It might be a portable pump to get the water from point A to point B. “With well over 100,000 distribution valves and roughly 200,000 fire hydrants, the GIS system enables us to analyze the system for simple points of failure. We utilize the GIS to identify our assets and maintain accurate location information, so we can find them in an emergency. Also, the GIS describes a single point of failure and helps us prioritize our capital investments and build resiliency into the system. So, you can’t prepare for everything, but you have to be prepared to adapt. Every year we get better and better and that’s through utilizing the GIS and doing these tabletop exercises and soliciting the expertise of our folks in the field every day.”

Beyond the folks in the field, it’s also important to solicit the expertise of local, state, and federal emergency management agencies. “They have access to very large and vast networks of equipment and capabilities and resources that you may not be aware of,” says Kirwan.”An extreme example happened when we had to call the National Guard to bring fuel because there was about 8 feet of water surrounding an intake and we couldn’t get a fuel truck.”

During Sandy, American Water burned about 10,000 gallons of diesel fuel per day, and Kirwan notes that having relationships with fuel suppliers is critical to keeping generators running, as is regular testing and maintenance of standby generators. But even a well-maintained generator can fail, as was the case at an American Water treatment plant where a utility power failure triggered a 1-MW standby generator, but it failed. “That lesson taught us that even a backup can fail,” recalls Kirwan. “It was a critical facility, so we put in a bypass connection that allowed us to bring in another generator if the existing unit fails.”

There are many different aspects to the subject of standby generators. Let’s take a look at some lessons learned from utility managers that have seen some of the worst-case scenarios. We’ll begin by heading south from New Jersey to Tampa, to speak with Steven Kruger, vice president of operations for Veolia Water’s South Region. “I coordinated the recovery efforts after Hurricane Katrina, Rita, and Isaac, and one of the challenges here is that the landscape of the region is flat, so we have about 89 pump stations, and they are easily flooded by the rains that come with hurricanes,” explains Kruger.

Flooding and high winds knock out power feeds to the pump stations, so Veolia keeps 26 portable generators on hand to rotate in the field, and maintain flow at the stations. Keeping fuel reserves available in long-term storm situations can be a problem. Although Veolia has contracts with local suppliers, on more than one occasion, the company has trucked in fuel from other states when power failures kept local suppliers from delivering. During Katrina, Veolia had a sewage treatment plant under water, and had to have a helicopter fly in a portable generator and pump in order to remove the water.

That flooded treatment plant had a standby generator, but of course, it was submerged in the water, and couldn’t run. “We refresh our emergency plans after a disaster, and in that case we built a new, standby generator and administration building on stilts, to survive the next flood,” says Kruger. “The building is hardened to withstand a hurricane. The generators are tested once a month under full load, and we do both automatic and manual switchovers to make sure everything works. However, you could run a 24­-hour test and find that it works fine, but after operating it six days continuously you run into new problems. Those are things you find out the hard way, but that’s why you bring in your diesel mechanics and electricians—to keep these things running.”

We’ve looked at good design and maintenance strategies for standby generators, but there’s also merit to the idea of easing the load on a standby generator. In the case of the water utility at the City of Westerville, OH, elevated water storage is the answer, according to Richard Lorenz, water utility manager. “We don’t have to deploy our standby system very often because we have elevated storage,” says Lorenz. “We have enough stored water in the air to get us through many hours of having no pumps. Sometimes we actually shut our pumps off because power will be interrupted and that’s very hard on our equipment.”

Westerville’s water treatment plant has a 450-kW generator (currently being rebuilt). The pumps at the water treatment plant are hooked into the generator, and there’s a booster station to maintain pressure in the field. That booster station doesn’t have a backup power system, but a dedicated pump at the plant is hooked up to the generator and can bypass the booster station pump. “So instead of buying a second generator and having to maintain it off­site, we spent some money on pipe to reroute the transmissions line so there would beanother way to get water to that part of the city,” says Lorenz.

Using an alternative pipe routing strategy and renting portable pumps solved a major crisis for Avon Lake Regional Water, Avon Lake, OH. “We had an icing situation in 2014 at our intake site on Lake Erie,” says Todd A. Danielson, P.E., BCEE, Avon’s chief utilities executive. “At a staff meeting one of our collection and distribution people that remembered that we had rented pumps in the past to bypass problems, and suggested that we could use the same type of pumps to bypass the intakes to pump water directly into the plant. That was the solution that got us out of the woods. It was a practical idea that didn’t come from some Ph.D. who costs a lot of dollars.It was from the guys that are doing the day-­to-­day things.”

Avon is also building additional storage to deal with another problem that comes from drawing water from the Great Lakes—algae. The state of Ohio is offering zero-interest loans to help utilities combat the algae problem, and Avon is saving hundreds of thousands of dollars per year by qualifying for the zero-interest funding. Danielson sees additional value in elevated storage because it can work in a backup or blackout situation, so it’s a solution that can have multiple applications, similar to the intake’s emergency interconnection, which can be deployed against icing, a major water break, or contamination at Avon’s location on the lake.

Getting a maximum return on infrastructure investments can help a utility stretch its dollars, but even a stretched dollar has to come from somewhere, and with today’s tight local and state budgets, those dollars can be hard to come by. However, there are a number of strategies that can help. To start, look at reducing costs around the plant. For example, Danielson found significant cost savings by analyzing Avon’s electricity bill from the local utility. “If you’re looking at ways to reduce costs, one way is to participate in energy programs that the power companies offer. Peak shaving can save on power company surcharges, and by having water in the air you can cut back on power demands during peak shaving hours. We are very actively working on those strategies to address power costs. Rather than saying we’re just going to work out a contract to buy power at six cents a kilowatt hour or whatever it is, we are actively playing the market and doing what’s called a block index arrangement. We might buy blocks of power when the costs are cheap, such as next summer inJuly and August, when we would want to buy 5 MW of power. So we buy power when it’s cheap.”

Avon’s facilities consume about 2 MW of power, and by using the block and index program, the utility saved $275,000. Danielson notes that additional energy curtailment efforts saved on the capacity costs for the next year. “There are two components—there’s a generation cost and a distribution cost, and by shaving that during peak periods of summer you save on distribution costs for the next year. So we are saving $40,000 next year by cutting back on peak hours this summer.”

Participating in a peak shaving of “dispatch” program with an electric utility can also help to lower standby generator maintenance costs. For example, the City of Hillsboro, OR, Water Department Joint Water Commission has saved more than a million dollars by partnering with the local electricity utility. “We have a program here that’s unique and we work with the local utility so it can dispatch our standby generators,” explains Tyler Wubbena, P.E., engineering manager for the water department. “It’s a public-­private partnership whereby they can turn on the generator during peak demand periods, and they’re using the generators to put power back onto the grid. In this partnership the plant project received almost $1.4 million in construction aid, and then of course there’s the generator maintenance cost, such as field testing and oil testing. One of the beauties of the program is that once a month they turn on the generators to test them. In doing that we’re confirming their performance, so it gives me a lot of confidence that the generators will run when I want them to.”

Hillsboro’s treatment plant has two 2.5-MW generators and 15,000 gallons of fuel storage. A recently built reservoir pump station has an 800-kW generator. Variable frequency drive pumps contribute to energy savings.

Recycling biogas is another method of gaining energy savings, and by entering into an Energy Savings Contract (ESCO) it’s possible to have the savings pay for the entire project. At the Hay Road Wastewater Treatment Plant in the city of Wilmington, DE, a $35 million renewable energy project demonstrates how an ESCO can deliver significant savings.

As is typical of an ESCO arrangement, the biogas facility is one component of a plan to decrease energy costs and greenhouse gas emissions throughout the city. The city partnered with Honeywell to develop and execute the plan. The biosolids facility harvests biogas to generate electricity for the wastewater treatment plant, and it also provides thermal drying to substantially reduce the volume of sewage sludge the city had have trucked away. There’s enough gas to power generators that can produce up to 4 MW of electricity, which covers 90% of the treatment plant’s power.

Although the following funding resource is the last on the list, maybe it should have been the first. It’s EPA. Utilities can access an important resource for determining opportunities for federal funding with EPA’s Web­-based tool, Federal Funding for Utilities–Water/Wastewater in National Disasters (Fed FUNDS). Fed FUNDS summarizes funding from EPA’s State Revolving Funds, FEMA’s Public Assistance Program (used after presidentially declared disasters), FEMA’s Hazard Mitigation Grant Programs, the US Department of Housing and Urban Development’s Community Development Block Grant (used in low­-income areas), and USDA’s Emergency Community Water Assistance Grants (used by rural and small utilities). Utilities should also talk with their local emergency manager, town manager, and state hazard mitigation officer to determine the availability of these funding opportunities for power resilience projects. Also, the Clean Water State Revolving Fund can provide assistance in the form of low­interest loans for backup systems/redundancy at publicly owned waste­water treatment facilities.

On the subject of disaster survival, the EPA is publishing a Power Resilience Guide for Drinking Water and Wastewater Utilities. The guide covers seven areas related to power resilience: communication, power assessments, generators, fuel, energy efficiency, onsite power, and funding. The information, case studies, tips, and videos in the guide can help water utilities to assess, develop, and maintain their backup power capabilities. Find the guide at http://1.usa.gov/1NTA6EW.

Ultimately, from superstorm disasters to a minor power outages, keeping a water utility running requires an understanding of the workings of the system—and having reliable backup systems to provide power and pumping capabilities—to keep the water moving. Of course, there are many more details beyond the basics of power and pumps, and of course, those are all addressed in your emergency plan. Or are they? As we’ve seen from the experiences of our experts, disasters can test even the best of plans and equipment. Nonetheless, solid preparation remains the key. As Kirwan says, “So, you can’t prepare for everything, but you have to be prepared to adapt.”