Virtualising critical solutions with confidence
Pinellas County (FL) Utilities delivers 71 million gallons of clean water and treats 30 million gallons of wastewater daily for one million year-round residents and 4.2 million visitors to St. Petersburg and the county’s 23 other communities. To do that, it manages 2,000 miles of pipe, 13 well fields, five surface water sources, three wastewater treatment facilities and 350 lift stations.
Business Situation
In order to maintain high service levels in the face of a shrinking tax base, funding cuts and staffing challenges, a new automation system was required. To help keep everything running smoothly, Pinellas County Utilities (PCU) relies on a supervisory control and data acquisition (SCADA) system from Wonderware that monitors some 20,000 individual “objects” such as valves, pumps, temperature gauges, etc. When the county installed the system in 1999 for Y2K compliance, it took that opportunity to free itself from inflexible, expensive proprietary technology. In addition to improving system-wide monitoring and management, the new system enabled problems to get addressed faster, field technicians to accomplish more and tighter overhead expense control.
Technicians remotely access and adjust data at any object data point from anywhere any time, via the Internet, handheld and mobile devices, thin clients and computer nodes. PCU management can also securely access and analyse real-time and historical information via an Internet portal. The resulting trend and numerical data analyses and comprehensive data reporting help them to optimise performance and decision-making.
Business Objectives
The utility’s value-added reseller partner, InSource Solutions, implemented a thin-client server system based on the Microsoft .NET framework using three pairs of Stratus ftServer systems at three geographically dispersed controller sites. Each server in a pair supports three critical applications. One runs ACP ThinServer, Microsoft Terminal Services and Wonderware InTouch software. Wonderware’s Object Server, I/O Device Integration Server and Historian Server software run on the other.
The three sites are connected to a central control room, which hosts the Galaxy Repository (the SCADA system’s Microsoft SQL Server® database for all Wonderware software) and the Wonderware Information Server, each on its own general-purpose x86 server. In early 2009, these two servers were five years old and due for replacement. As in previous cases, SCADA supervisor Ken Osborne saw an opportunity to invest in new technology.
Instead of new servers, Osborne believed that virtualisation software could leverage past technology investments, reduce costs, and protect services today, and yield future savings and operational improvements. Firsthand experience with Stratus availability solutions and support gave him the confidence to move ahead without worry, knowing it would be safe to run critical utility applications on a virtual environment.
“Keeping the water on is a public health and safety issue. We can’t tolerate any downtime,” said Osborne. “Our operation has relied on Stratus systems with no unscheduled downtime caused by a server failure. Parts occasionally need replacement, but the server and applications never stop running. Replacing clustered failover servers with ftServers saved us a lot of money and simplified the entire operation. I’ve never looked back.”
The Continuous Availability Solution
The three primary control centers are core to PCU’s ability to service the county, so each has ftServer systems installed. The Wonderware Galaxy Repository Server and the Information Server in the central control room are also important. However, the 1999 infrastructure design enabled the three controller sites to continue operating if the central repository went offline. Similarly, an Information Server outage would temporarily halt access to system-wide data, but field operations would not be affected. It was decided ordinary servers could handle the job.
That was then, VMware virtualisation is now. Osborne knew the Galaxy application alone was not sufficiently critical to justify the investment. However, adding Terminal Server for mobile HMIs together with the I/O Device Integration Server application to the mix, several benefits emerged:
- Centralising remote access to the entire SCADA system was more efficient than dividing it among the three controller sites
- Running three high-value services on three virtual machines protected by a single ftServer improved overall system integrity and flexibility and made economic sense
- With a virtualisation platform in place, consolidating other applications and introducing new ones would be a matter of simply creating a new virtual machine and putting applications to work immediately
“I can foresee a time when the SCADA operation runs entirely on virtual machines and three fault-tolerant servers,” said Osborne about his current eight-server infrastructure. “That option didn’t exist a decade ago. Our decisions then proved to be the right ones in every regard and today we’re smarter about our virtualisation strategy because of it.”
The Information Server running on the second x86 server eventually would get its own ftServer system. The county was implementing a high-speed wireless network to which many employees would have access. Running this application on a continuously available platform was essential, given that it would be servicing the county at large. But more importantly, placing a firewall between the SCADA system and the county-wide network would be more secure.
Business Impact
Had he used server clusters instead of ftServer systems, Osborne figures the virtualisation project cost would have doubled and taken weeks longer to implement. “When you compare the total cost of a fault-tolerant server solution to building a cluster, the investment is really a wash,” he said. “The fault-tolerant server is also more flexible and much simpler than buying many individual x86 servers. For me, the big benefit is support. Having Stratus engineers monitor and protect my server 24/7/365 costs less and is better than hiring a full-time technician. The server always runs and we never lose a thing. That’s peace of mind.”
