The Oldsmar security breach: What your utility needs to know

On February 5, an unauthorized person twice gained access to the supervisory control and data acquisition (SCADA) system of a water treatment plant in Oldsmar, Florida. During the second breach, the amount of sodium hydroxide (lye), a caustic and potentially hazardous chemical used as part of the treatment process, was increased to a dangerous level. Fortunately, an operator noticed the actions and was able to quickly correct the situation before water quality or safety were impacted. 

Since the incident, several issues have come to light on how someone could have gained access to the system. First, the computers in use were running the 32-bit version of Microsoft Windows 7, an operating system which had reached end of support more than a year prior to the incident. 

Second, the compromised computer was running TeamViewer, which allows remote access. TeamViewer had remained installed and was set to allow remote access through an internet connection with a shared password for all users after the organization moved to another remote access solution roughly six months earlier.

Securing your systems means putting up roadblocks for potential threats 

Incidents like this are always a good reminder to revisit cybersecurity best practices and limit potential gaps in your systems. A few steps utilities can take to secure their systems include:

• Keeping all software versions current and patched 
• Removing or updating software that is no longer in use or supported by vendors 
• Regularly changing passwords; leverage user-level authentication for additional security  
• Enabling multi-factor authentication and maintaining secure password and authorization controls when allowing users to access systems remotely 
• Using firewalls and other approaches to isolate operational technology (OT) systems such as SCADA from the IT network and the internet 
• Performing an architectural solution review for how interactive users access real-time network environments (i.e., assess the use cases, methods, and security controls in place for interactive users to access systems within the real time network) 
• Auditing logs for all remote connection protocols
• Conducting active threat hunting 

These are good initial steps to take, but combating cyber threats fully requires a culture of security awareness and sensitivity throughout the utility. The best cybersecurity programs are founded on the understanding that everyone in the organization must work together to address threats.

While you’re in it, address other concerns outside of cybersecurity 

System operators often work long shifts in what can become a stressful environment if an incident or malfunction were to occur. Because of this, a great deal of thought typically goes into developing a well-designed, intuitive, and reasonably failsafe user interface. For example, limit checking is typically implemented so that an operator cannot accidentally put the system into an unsafe or unstable condition. In the Oldsmar incident, the amount of lye was able to be changed from 100 parts per million (ppm) to a dangerously high 11,100 ppm. 

Beyond the SCADA system, there are often interlocks and monitoring devices that require manual intervention when something strays outside normal operating parameters. For example, if a pump could be damaged with either the upstream or downstream valve closing, relays might inhibit operation of the pump under those conditions. While a sensor could fail or there may be a need to operate a system in an unusual configuration, entering these configurations by accident or without a system of checks and balances is something well-designed systems should avoid (e.g., requiring manual operation of a local override switch that is separate from SCADA control).

Utilities can add layers of protection by reviewing old processes and implementing new ones 

What happened in Oldsmar was a cybersecurity event, but utilities can protect customer health and safety by refreshing and reviewing their layers of protection and implementing lessons learned from this incident. Processes and systems for daily operations should have protections to prevent a single act (intentionally or not) from cascading into a much more serious situation. 

Addressing the challenges that come with an incident like this requires a different, flexible, and holistic approach to ensure operational resilience. The ability to prevent, respond to, recover, and learn from operational disruptions will safeguard key business services against severe, but plausible threats.