Trihedral’s Bryan Sinkler is responsible for VTScada software customers in the southeast US. Bryan will discuss how his SCADA system customers fared during Hurricane Irma and the important role that system redundancy and remote access play during dangerous weather events.
Todd: Good Afternoon and welcome back to Water Talk. I am your host, Todd Schnick, joined by my friend and colleague, Bill King. Bill, I have been fortunate to work with you all and broadcast this great show for six years at WEFTEC, but I have also covered the show for seven years straight. In all seven years, I have had the pleasure of having a conversation with someone from Trihedral; amazing organisation doing amazing work in the industry.
Bill: Yeah, a very interesting organisation. We look at what we have been talking about on the show over the last couple of days; a lot of topics and trends, but I am excited about this next conversation because this is really current events that we are going to tap on to.
Todd: Yeah, this is very important. This gentleman was at ground-zero in Hurricane Irma in Florida. That will be a great topic for our conversation. Let’s welcome Bryan Sinkler. He is a Technical Sales in Florida with Trihedral.
Bryan: Thanks Bill and Todd. I appreciate you having us.
Todd: We appreciate you Bryan for making time to join us, and you are awfully busy; and at an event like WEFTEC, we are grateful for you swinging by. As I mentioned, we have always featured Trihedral on this show, so our audience is very familiar with your work, but just in case, take a quick minute and tell us a bit about the important work that you are doing to serve the marketplace.
Bryan: Trihedral has been around for 30 years. We are a SCADA HMI Software platform; we work with lots of customers all over the world; water / wastewater is one of our biggest vertical markets and we have a variety of different feature-sets that make us a very powerful HMI option. We are very glad to be working in this industry.
Bill: Like you say Bryan, water / wastewater market is one of your larger markets. We are interested to learn first-hand from you like when you talk to your customers that we are affected by the recent Hurricane Irma, how did they fare?
Bryan: Most of them did ok, as we were just talking to Todd that his parents did very well, I did pretty well, others not so good, better than in Puerto Rico obviously, but most of the utilities did well; lot of power outages.
Most of the issues we saw were on the SCADA side of things, some servers going offline, redundant fail-over coming up; but for the most part, I have talked to many of my customers since the storm has passed and most of them fared pretty well, the exception may be being in the Keys; they took a beating down there.
Todd: Yeah they did. So, in the utilities, when there is an event approaching, particularly one that you can’t prepare for and you don’t get an advanced warning, like the hurricane for instance, what kind of precautions can you take? What kind of precautions did your customers take and how did you help them do that?
Bryan: Generally, I am very well-immersed with my customers. I spend a lot of time personally, sitting on emails, making phone calls, letting them know what our 24/7 support line was just to make sure they had those numbers and contact those folks in case they needed us.
On their side, lots of debris pick up to get ready, generators that sort of thing, but from the SCADA side of things, which is the world we live in, lots of UPS checks, generator back-up checks, making sure that EOC was up and running; a lot of customers will have their servers placed in a VM environment in their EOCs or in a hardened facility, this is pretty common in Florida specially.
We do two of the water management districts in Florida, which were very important during the storms; so definitely checked with those guys. They were very busy, obviously.
We are involved with Southwest Florida Water Management District as well as St. Johns River Water Management District, which is basically the north east and gulf coast of Florida. They were quite busy. But, all in all, there were some issues that came up, but for the most part they fared well.
Bill: Bryan, all through these events that we have had recently, we have had this slew of events, we have had Houston and then Florida, more recently in Puerto Rico, I am just feeling fortunate that I live in Pennsylvania; but for those areas that are prone to storm conditions, what would you is the reasonable amount of server network redundancy for the critical scale assistance that you talk about?
Bryan: That varies from application to application; a small utility, say city of Wildwood in Florida, which is very small utility, which you may have heard of; they may just have a primary and a backup, which are fairly common servers. But then you get into the Miami Dade’s, we deal with larger systems; even the city of Ocala has a fairly complex redundant system.
At each facility, they have a server, so it gives them a redundancy of five to six servers at any given time, which did pay some big dividends during the storm for them. It is this wide-area network redundancy that is one of our great feature-sets. So, it really does depend on the application; typically you see a primary and a backup. We do have some folks, surprisingly still, that don’t have any redundancy. Much of this is budget-dependent, but it did bite some of them in the behind.
Todd: Power failure is a big concern; when you were forming this, how everyone fared from Irma, power failure was a big deal. What happens to your SCADA system during a power failure? How do you deal with that?
Bryan: What typically happens, assuming that they have some sort of redundancy at another facility, a hardened facility, like an EOC typically, the way the software behaves is that it is seamless to the user and it just fails over to the other server, without any intervention from the user. Once that server comes back online, the primary, it synchronises all the historical data that it missed. That’s how that is dealt with typically.
Todd: Are you battery-operated for backups or is it generator? I am just concerned about if there is a power outage, does that affect your SCADA systems?
Bryan: It affects it in the regard that the data has not been collected. Most users will run a UPS, that’s the last line of defence after the generator fails, but you have only so much battery power at that point. In a storm like this, like I mentioned to you, I was out of power for nine days, so sometimes you can only do so much, but typically a UPS is on every server and typically, a high-end UPS.
Bill: I am really interested to hear your perspective on how your customers actually dealt with getting to work, or not getting to work? We hear about, when these events come through, everybody is locked-down or moved out of the area of the storm, how did that affect your customers in terms of getting back to work? I know a lot of these roads were impossible.
Bryan: That’s true. What typically happens is, if you live through these storms as a utility worker, which I was one in my former life, you get the luxury of staying the night in some EOC, living on cots and eating Subway for the night, or whatever is available.
It does become a problem, getting out to things; safety is the biggest concern, there are power lines down, there is flooding, things like that; so you do it in a safe manner, you get to where you can get to and at the end of the day, you would prefer things work but you also don’t want people to die and to sustain injuries.
Todd: Based on your reports, sounds like your customers fared very well through Irma and that you did good work in doing preparations leading up to the weather events, so kudos to you on that; however, I suspect there were still some lessons learnt from this. What can you share on that?
Bryan: I had one customer, specifically, I suppose I should not mention who that customer is, but they did not have a back-up, as I had mentioned, and they did indeed take a nasty power failure and it took out their server. Luckily, we do have some really nice features in VTScada; our Version Control and our ChageSet feature, which allow the users to, if they do suffer such an event, to quickly bring up a new server and in no matter and no time.
Once again, we do offer 24/7 emergency support so that we can assist customers in those times of need. So, I would venture to say that that’s one of the biggest lessons that I would tell a customer, when we look back at these storms, is always have a redundant server – at a minimum one.
We had another utility in North Florida, near Jacksonville, fairly one of our larger customers that ran out of mobile connections. They were just so busy; they are looking at that now. These are the kinds of things that when you don’t need them, you don’t think about it but when you need them, you really want them.
Todd: Take advantage of what just happened and learn from other utilities as well, that’s a part of why I am asking this. Others listening can think ooh I need to think about this.
Bill: Last few years, when we have had Trihedral on the show, there has been a real movement in controls and SCADA systems to more remote access and the ability to access data over a variety of different mobiles, tablets and that type of thing; how did that show up in relation to these major weather events?
Bryan: Several utilities, like the utility in near Jacksonville, they ran out of the mobile clients; they use them quite a bit for going out to the sites and checking on their phones, their tablets, their laptops; so that they don’t have to rely on the dispatchers to give them information.
The City of Ocala, where I came from, they use that quite a bit extensively. During the storm that was one of the key benefits of VTScada for them. They called me and told me that it was a life saver for them.
Todd: That’s going to look good on the brochure. That’s very powerful. You are exhibiting here at WEFTEC; what are you principally showcasing? What kind of feedback are you getting? What is generating some buzz?
Bryan: We are doing some giveaways at the booth every half hour, where we are giving an IIoT device away. IIoT is a big buzz word this year at the show, I am sure with some of the other SCADA groups that you have been around; so every half hour we are giving it away.
It’s an arduino device, it measures temperature and humidity and it comes packaged up with a little VTScada application. So, that’s one of the big things we have been promoting at the show. Version 11.3 is what has been showcased at the booth with all the great benefits and features of VTScada.
Todd: Outstanding. Can radio hosts enter that competition?
Bryan: Sure.
Todd: Bryan, running low on time, before I let you go, should anyone have questions on any of this or to learn more about how to prepare for an extreme weather event, how do they find you?
Bryan: They can certainly find us on our website at www.trihedral.com or they can find us on Facebook, on Twitter, on other social sites as well.
Todd: Outstanding. Bryan Sinkler, Technical Sales for Florida with Trihedral. Bryan, great to have you; thanks again for stopping by and joining us.
Bryan: Thank you guys, it was good to talk to you.
Todd: All right, all the time we have for now, on behalf of all of us at Water Talk and Water Online, thank you for listening in. To hear more of our conversations and to learn more about us, go to wateronline.com. Todd Schnick and Bill King signing off for now, but don’t worry; Water Talk will be right back.
*The above is a transcript of Bryan’s interview from WEFTEC, we have tried to keep it as close as possible to the actual questions and answers without too much editing. Please excuse grammar and sentence structure!