News: 1599058211

Ah, a new and bigger attack surface then....

It also makes, possibly even local, control and monitoring vulnerable to internet down or cloud down events, which could be caused deliberately by DDoS attacks, even for encrypted communications.

Water provision should be recognised as part of critical survival infrastructure, so should have military grade security and local control, and only be linked via a private WAN; with secured internet access/messaging as a non-essential option, via a separate WAN node. I suggest that Water companies should be regarded as military contractors.

I work for a company that develops cloud-based SCADA for utilities (no involvement with this project though). There are ways to address these issues.

1. In our system, equipment control (PLC) and alarm processing run at the site, not in the cloud. The cloud HMI sends control requests and receives status responses, alarms, performance data etc. If the cloud HMI goes down, the site will continue to do the last thing it was told, and record alarms and historical data values.

2. Communication from the HMI to the site is completely separate from communication between the HMI and the users. Also, site comms automatically fail over between multiple connections. Typically, at least one of these does not entail tunneling over the public internet.

3. If the HMI is down hard due to hack, crash, DOS, or whatever, there is a local HMI running at the site that a customer can either lay hands on (if at the site) or remote into (if not), which can also issue control requests to the PLC.

4. Since point data is recorded in an onsite SQL DB, when HMI comms come back up the operator has a complete picture of everything that took place during the comm outage.

We have other protections and redundancies in place as well. The point is, we're not idiots and we're not making $15 IOT light bulbs. We know that this is critical infrastructure, and we don't want it to be fragile or vulnerable.

No security is perfect, obviously. Natanz and Russia's attacks on Ukraine's grid are just two examples of how even air-gapped, non-cloudy SCADA can be vulnerable to a well-resourced attacker.

That a highly trained human being has to sit in front of the controls of the treatment plants, and only the human is allowed to Control the treatment plant.

A well-trained but highly-insured human being has the passwords that let you change all the settings in the control system, and then the computer does all the busywork of actually running things. This frees up the human being to take laboratory samples, go look at things that have stopped responding, receive chemical deliveries, and all the other exciting jobs that water treatment entails. The plants don't quite run themselves, but can quite often be left to do their own things for weeks at a time without raising an alarm that needs a human being to go look at it *right now*.

I’m a Chartered Chemist, and also have a strong background in IT. In for 25 of the last 50 years have specialized in water and environmental analysis, including for a national authority, and am a volunteer technical assessor for ISO17025. My comment still stands.

So you are taking a mission-critical system and hosting it in the cloud? Why? Something like SCADA, which doesn't have any real increases in resource demand during its function (It's not like something like a ticketing system or an ecommerce platform, where transactions and resource demands fluctuate wildly. Nor is it a temporary or seasonal need, SCADA is a year-round requirement.) should be a perfect candidate for being on-prem, where it can be secured more easily.

So you are taking a mission-critical system and hosting it in the cloud? Why?

Several reasons.

You have distributed resources that cover a large geographic area, but which need to be centrally managed and monitored.

You need both local and op center users to be able to monitor and manage your plant(s).

You are in the middle of a pandemic, and you need at least some of your operators and managers to be able to work remotely.

You are rapidly expanding, and you need your control infrastructure to be highly scalable on demand.

Just a few scenarios.

See my comment below. *Analytics* computation requirements can indeed fluctuate wildly, and that seems to be what they're talking about here. Plus lots of historical data, which means cheap, reliable storage is highly desirable.

Hopefully the description here has been over-simplified for IT people? Severn Trent's water treatment plants have a hierarchy of control:

- generally, each process area is controlled by a Programmable Logic Controller (PLC) - these are sat in a cabinet in a Motor Control Centre (MCC) and are responsible for monitoring instrument readings, starting and stopping drives, opening closing valves, that kind of thing. They're generally hardwired to each device; slightly more modern plants would use a token-ring networking system called Profibus to connect everything up, but Severn Trent are old-school. PLCs are hard-real time controllers about the size of a fag packet that use specialised programming techniques, usually ladder logic. The PLCs will have a local Human Machine Interface (HMI) screen, which shows the status of all the kit and all the set-points in effect, in pretty picture form, called a mimic. They're super simple and reliable; we've still got PLCs from the 60s that have just been ticking away every day.

- the PLCs are monitored over the network using a Supervisory Control And Data Acquisition (SCADA) system - generally over ethernet, sometimes ethernet over fibre optics if they're a bit further apart. SCADAs are generally server blades running Windows. They'll have a copy of the mimics for every HMI, so you can supervise the status of the whole plant; will let you change any set-point anywhere, which saves having to wander around the site to do it, and they'll have a lot of trending information available - they record all of the PLC instrument readings, so you can check how deep a tank was a year ago, for instance. You wouldn't want to outsource them; running a plant without one changes it from a one-or-two man job to an all-hands-on-deck, 24/7 cover disaster, and so an outage would be very very bad. Many of these sites are in the arse end of nowhere, and have unreliable internet connections: local, network-isolated Windows running on redundant servers can actually have a very good uptime.

- the SCADAs will report some telemetry info to STW's central database, so that the bods in head office can monitor the info. I think that's what's being proposed for changing over here? At the moment, STW only monitor a few key pieces of information (total plant flow, etc). United Utilities have been attempting to change over to their own system for the last several years, which monitors literally every piece of info for every asset they own, which is costing them millions per year in huge Oracle racks and which is very very slow; it's a prime candidate for moving to the cloud. STW are generally a bit more cautious and conservative, but can possibly leapfrog the 'do it on the premises' step - I think it would make a lot of sense. The SCADAs themselves generally keep years' worth of trending info stored, so they can buffer for a while if the networks go down and just update whenever otherwise - there's hardwired emergency callouts for things like power failures otherwise.

The three suppliers are Siemens, Allen Bradley, and Mitsubishi, I think. STW have been generally changing over from AB to Mitsi for all their control needs, but Siemens have been basically giving away the hardware and then stinging you for licenses and spares lately, and have been making inroads.

This is the level of insight I come here for. Great comment!

Hi Addie, see my self-aggrandising comment above. I liked and upvoted your post, but I suspect that I am more cynical than most, and I believe that my cynicism has been earned - Including being the senior chemist who did the analysis when ST was prosecuted by the relevant authority when THEY had polluted a major system (Admittedly that was before privatisation in 1989, I suspect that things have not necessarily improved).

Makes a great deal of sense. Particularly if there is a very variable query workload you could stream the information into Azure Data Lake Storage and run queries using Azure Data Lake Analytics. That would provide cost effective storage as well as usage-priced analytics compute instead of relying on provisioning loads of expensive traditional data warehouse nodes (and their associated licenses) that are probably lying fallow most of the time, and insufficient when you do get busy.

This kind of analytical workload is normally a slam dunk for cloud over on-prem, and doesn't usually pose a direct threat to integrity or availability of operational systems - obviously confidentiality may obviously still be very important, depending on the nature of the data. The data flow is from the sensitive operational network to the less sensitive cloud analytics one, and you can make going the reverse way very difficult (even data diodes etc. for very high assurance).

The exception is possibly the monitoring side of things, where a DoS/compromise might slow some types of response. But it sounds like the biggest problem would be plain old non-malicious unreliable plant network reliability issues - any response would have to be resilient to that, and thus to more malicious attacks.