A new approach to SCADA helps a Florida utility to protect historical data and improve performance.
By Christopher Little
By 2003, the City of Ocala had converted their two monitoring & control systems to VTScada software from Trihedral. In 2005, they merged the two into a single application.
In that system, servers at six remote sites logged sensor data to a central historical database at the main office. When operators at each site opened a trend window or ran a report, the data came from this database.
This common SCADA architecture poses two potentially serious issues; especially for utilities that rely on historical data to control costs. If the network goes offline, users at each site cannot access the historian. Also, data collected during the interruption is lost.
In April of 2011, Trihedral President, Glenn Wadden, and programmer, Will Dumaresq, traveled to Ocala to implement a new distributed approach to historical data that Trihedral developed specifically for large data-rich SCADA systems. In this article Wadden describes the challenges of transforming the database architecture without interrupting the process or losing data.
The City of Ocala
Located in central Florida, the City of Ocala serves the water and wastewater needs of over fifty-five thousand people. Their SCADA system currently encompasses a spray field, three water reclamation facilities, a water treatment plant and a group of 125 lift stations.
In the past, a SCADA server at each site allowed workers to perform local monitoring & control based on local sensor data. A primary SCADA server at the main office polled each site and logged its data to a central historical database. A synchronized server a few feet away provided automatic failover as well as a backup historical database (Diagram 1).
Glenn Wadden is the president of Trihedral and head of the research & development team for VTScada, the monitoring & control software at the heart of the system. “If they had a network problem, they would still be able to control the site and see live information. However, for that period, the I/O data would be lost forever because it never made it to the historical database on the other side of the city. The problem was that there was no local data logging.”
A New Way to use Historians
Wadden noted that over the last decade, utilities have been moving toward larger, more data-rich SCADA systems. By mining historical data using trends and reports, utilities hope to reduce costs by finding hidden problems, extending the life of equipment and reducing waste. He and the VTScada development team set out to create a better way to protect this precious information. One of their solutions was to provide support for multiple synchronized historians.
“What we have done is added an historian at each facility to collect local data,” says Wadden. “The central server at the office runs six backup historians. Each one synchronizes with an historian in the field over the wide area network. Since each site has its own historian, it continues to log its I/O data even when isolated from the network. It is self contained. They can see their own history, but they can’t see any other site until they get back online.”
“Once they do, all their history that was collected while they were offline will automatically upload to the main server where everybody can see it.” Similarly, if a local server is replaced; the new server’s database is backfilled with all the history from the backup database which also supplies historical data to other locations while the server is offline. (Diagram 2)
All of this happens without the involvement of operators. “When a user opens a trend window for one of the sites in the system, it is transparent where the information comes from. So, if you are at WRF 1 and you want to see what is happening at WRF 2, VTScada pulls that information from WRF 2 because it is the primary server for that site. If WRF 2 was not available, it would automatically go to the back-up historian at the central office.”
Upgrade Challenges
Over two days, Wadden and Dumaresq upgraded Ocala’s servers to VTScada version 10 which included the new historian functionality. They then changed thousands of input tags spread across servers all over the city to point to a specific backup historian based on their location. “The existing historical data was left untouched in the legacy format on the primary server,” says Dumaresq. “The remote sites can still access it over the network.” In the future, this data could be split up and distributed across the new historians.
According to Wadden, the biggest challenge was completely changing the database architecture with no system downtime and without losing any historical data. “We had 10,000 tags that we had to change over. To get that right, we needed to do some careful planning.”
The pair converted the system in stages. Since the sites were spread out across the city, they used a VNC program to remotely upgrade each server from the main office. “We had to shut down VTScada, upgrade VTScada, make some other modifications and restart.” To keep each site running during this process, they switched monitoring and control over to a VTS Internet Client.
In total, the process took 20 hours but might have taken longer if not for another new VTScada feature. “Version 10 is much faster,” says Wadden. “An application that once took ten minutes to restart now only takes three. When doing this kind of upgrade you may need to restart each server at least twice. The faster you can get it back online the better.” This speed also improves synchronization of the historians across the network.
More Benefits
Now that historical data for each site resides on the local server, local users can access it without being constrained by the speed of the network resulting in better performance. According to Wadden, the upgraded system comes with other benefits. “In addition to logging to multiple historical databases, the utility can also choose to log to other database formats like MySQL, SQL Server or Oracle. This allows commonly used reporting products to access SCADA information from computers that do not have VTScada installed. This is a new spin on what a SCADA historian can do.”
