The agency responsible for distribution of water and natural gas throughout the metropolitan area of Omaha, Neb., cannot tolerate failure in its SCADA system. Not only could a failure threaten the drinking water supply of over 400,000 residents, it might cost the district hundreds of thousands of dollars in penalties from the natural gas pipelines on which it depends. To protect against such failures, the district has installed a failsafe SCADA system that uses built-in redundancy to protect data and to ensure continued monitoring in case of hardware downtime.
The Omaha Metropolitan Utilities District (MUD) provides water and natural gas to the metropolitan area of Omaha, Neb. With rates for natural gas and water among the lowest in the Midwest, Omaha has attracted economic growth. The district provides drinking water to 164,000 customer-owners through a distribution system of 2,207 miles of mains.
Water comes from 158 MGD Florence Water Treatment Plant, which treats surface water from the Missouri River northeast of Omaha, and the 60 MGD Platte River Water Treatment Plant, which treats groundwater taken from the alluvium along the Platte River southeast of the metropolitan area.
Production of water from the plants, which have their own separate SCADA systems, is monitored and adjusted for predicted demand changes. Most water is pumped a second time at one of the 14 remotely operated pump stations. Total storage of 116 million gallons helps to balance production and demand.
Upgrade Planning
In late 1997 the district's engineering staff foresaw that it would need to upgrade the system used to monitor and control the distribution of both natural gas and water. They needed to improve operating efficiencies, contain maintenance costs and forestall any interruptions in service that might result from Y2K problems.
In place at that time was a Fisher Controls GV2000 Unix-based system with H-P RTAP (Real Time Application Platform) workstations. The version of Unix used was not Y2K compliant. In addition, support costs were expensive and rising because the district's system was essentially an orphan technology.
Genies allow operators to enter notes about status of individual control elements such as motors and pumps.
The district has a separate LNG (liquefied natural gas) operation that comes on line to balance gas loads during the winter months. This peak shaving operation had already selected and installed a Citect SCADA system. In its request for bids on the distribution SCADA system, the district specified that Citect would be required on any proposed solution.
"We decided to standardize on a common software package throughout the entire operation," said James K. Knight, who was then the superintendent, gas supply and production (since promoted to manager, energy acquisitions). "The district already had Citect in place at the LNG operation and we knew that it would work reliably. It also had the added benefit that our engineers and operators were already familiar with it."
The winning bid for the $200,000 project was submitted by Huffman Engineering, which had already done work for the district in engineering and upgrading the monitoring and control system at the Platte River and Florence water plants.
Project Scope
The M.U.D. SCADA communicates with 28 Fisher ROC RTU's (remote terminal units) in the field via 19 leased lines. Leased lines are used because they provide faster data polling (15 sec.) than would be permitted by dial up modems. Speed of polling is limited primarily by the internal modem in the RTU.
A new driver for the RTU hardware had to be written and installed by Ci Technologies to support the engineering effort by Huffman.
"This was a major obstacle to overcome in the project , and it would not have been solved without the team effort of MUD, Citect and Huffman Engineering," said HEI project manager Craig Malesker. "MUD was extremely patient and understanding during the driver testing phase and provided HEI and Citect with full support even though at times some of the testing was an inconvenience to MUD operators."
The leased line modems are connected to rack mounted fallback switches via RS-232; the switches are connected to the RS-232 ports on both I/O servers.
An Ethernet hub for the SCADA system connects both I/O servers, a display client for the water department operator, a display client for the gas department operator, a development/file server node, and the M.U.D. LAN router. Three manager nodes (display only) are connected via the M.U.D. LAN.
The I/O servers handle all I/O, alarms, trends, and reports, with the A unit being the primary and the B unit acting as the secondary node, prepared to take over transparently if the primary fails.
Although the field instrumentation has only 750 I/O points, measurement and control requires nearly 7000 tags:
- 5046 variable tags
- 275 trend tags
- 187 digital alarm tags
- 234 analog alarm tags
- 940 advanced alarm tags
Despite the relatively large number of tags in the system, the district needed to purchase only a small number of Citect licenses:
- Two 1500-point I/O servers
- Two display clients
- Three manager nodes
- One development node
Because the manager nodes are floating licenses, multiple users throughout the district can access the system from their office computers via the network. Five Pentium III 450 MHz PC's running Windows NT 4.0 were installed in the district control room. They are networked peer-to-peer, and connected to the district's Novell-based LAN.
There are 332 Citect accumulators configured in the project. Many water and gas flows are totalized on an hourly, daily, or monthly basis, with the accumulators updated every second. All operator commands (setpoint changes, pump start/stops, etc.) are logged for a historical sequence of events so that the information can be available for troubleshooting.
A single display with five or more windows is split across two separate monitors for the water distribution operator.
A supergenie popup screen with a variety of features was created for analog I/O points. When an operator clicks on any analog I/O point on any screen, an analog popup screen appears. It displays additional details such as the min and max range of the selected point. Properties of the point can also be viewed and modified from the analog popup screen. The selected analog point can be placed off scan and a manual value can be entered. The alarming for the selected point can be disabled, and an historical trend for the point can be displayed. The alarm setpoints and the point's zero cutoff value can be modified. And finally, a note for the point can also be entered from the analog popup screen.
Additional alarming details are worthy of note. The alarms are configured such that water alarms appear only on the water operator's screen and the gas alarms only appear on the gas operator's screen. There are 32 alarm categories configured. All alarms, disabled alarms, and alarms for each ROC can all be individually displayed.
Malesker points out that there was much custom Cicode written for this project, especially on the gas side where many complex calculations are required.
"This is worth noting," Malesker said, "as it describes the powerful functions available with Cicode. For example, the projection factor data is retrieved from a table each hour, and is then used in a calculation. This table contains a value for every hour of the day, for each day of the week."
System Operation
MUD management has concluded all principal objectives were accomplished by installation of the new SCADA system.
- Operational oversight. Departmental managers are able to view operations from their own offices in real time to plan for the production of water and the purchase or manufacture of natural gas in response to changing weather conditions. Both water and gas managers routinely import directly from the system into spreadsheets for operational analysis
- Failsafe system with redundancy. Computer downtime has been all but eliminated, and testing shows that switching between the primary and secondary I/0 server is seamless.
- Data sharing. Analog points and accumulator totals are logged to an Access database every 15 minutes and this data is used to generate hourly, daily and monthly reports. Accounting personnel routinely check data from the archived Access databases over the office network, and use it to resolve billing questions. Data can be transferred into other business systems and improves the core operation.
- Reduction of paperwork. Most paper records and logs have been eliminated.
- Multiple displays. Huffman engineers used the Citect window resizing feature and Cicode programming to devise a toolbar icon so that an operator can fit up to four windows on a single display in any arrangement they like. In addition, the water distribution operator has two 21-inch monitors connected to his PC, which sees the two as one single display. It lets him drag and drop a window from one monitor to the other-seamlessly. This makes monitoring of alarms during other operations much more efficient.
- Flexibility. The district can replace PCs with commercial off-the-shelf Windows-based units in order to upgrade their SCADA incrementally and inexpensively. They also were able to use the existing 19-inch rack enclosure for the I/O servers with no modification. Standardization on Windows-based software ensures an economical path for future upgrades.
- In house modifications and documentation. The system now allows technical personnel to note status of sensors or control elements on the popups used by the operators. Operators can immediately see the reason for any discrepant input from the field, or notify maintenance when a unit appears to be out of service.
An extensive Help system was created in Microsoft Word, complete with illustrations, published as an HTML document and can be accessed in the Citect SCADA system as an ActiveX window using Internet Explorer. - Compatibility with existing control hardware. No existing control hardware had to replaced or modified.
