Every major infrastructure attack eventually produces the same headline: attackers got in, systems were disrupted, and the details of how are still coming into focus. What’s more interesting to me than the details of any single incident is a pattern that shows up almost every time: the assumption, baked into most response plans, that the tools and networks you’ll use to respond are still the ones working.
They usually aren’t.
Water and wastewater utilities have been in the news again for exactly this reason. Industrial control systems get exposed, credentials get changed, operations get disrupted. The technical entry point varies. What doesn’t vary is what happens next inside the organization.
Most water systems in this country run lean. A small IT team, outsourced or shared security resources, some external vendors, and everyone wearing multiple hats. When a major incident occurs, the operational systems and the communication systems often live on the same compromised network. The incident response pulls in various internal stakeholders from the business like legal, customer service, executives and the board as well as third parties like outside counsel, forensics firms and state and local government agencies. The organization needs to keep all the various threads compartmentalized yet progressing together, while managing who gets to see what and when.
That’s the real problem. It’s not just whether you can detect the intrusion. It’s whether you can keep talking to each other, with a record of what happened, while you’re not sure what else is compromised.
This is where most incident response plans fall apart in practice. They assume the tools you’ll use to respond are the tools still working. In control system attacks, that assumption runs backwards.
The organizations that come through these events successfully tend to share one thing. They didn’t attempt to figure out how to work together for the first time in the middle of the incident. They had already worked out who talks to whom, what gets escalated, and how decisions get made and recorded, before anything happened. And they had a way to keep doing all of that even when their own systems were compromised or unavailable.
That’s the principle we built CYGNVS around: prepare, practice, respond, report.
- Preparation ahead of time with adaptive playbooks by incident type, geography, role and function.
- Practice through structured tabletop exercises to refine and hone playbooks.
- Response runs in an Out-of-Band environment independent of the organization’s own systems, so a compromised network doesn’t take the effort down with it.
- Report is the clear account of what happened, from evidence capture and chain of custody to reporting to the various regulators and jurisdictions including the public.
In practice, organizations that build this rhythm tend to save meaningfully on the cost of an event as well as the time to get back to business. Not because the incident disappears, but because the chaos does. Every CISO who has dealt with a major incident will tell you that misalignment and chaos cause more problems than the original threat actor.
Water utilities aren’t the only ones exposed here. Any operator running OT alongside limited IT staff, in energy, transportation, manufacturing, is one exposed control system away from the same problem. Worth checking now, not after the fact, whether your response plan survives losing the very systems it depends on.
About the Author
Arvind Parthasarathi is the Founder and CEO of CYGNVS, the out-of-band command center that over 3,000 organizations rely on for cyber incident response management. Arvind also serves on the technical advisory council of the Allen Institute for Brain Sciences and on the board of trustees of the Center for Excellence in Education.
Follow Arvind on LinkedIn for more on cyber resilience and incident response.
