Spire: Intrusion-Tolerant SCADA for the Power Grid

Overview

Spire is an open-source intrusion-tolerant SCADA system for the power grid. Spire is designed to withstand attacks and compromises at both the system level and the network level, while meeting the timeliness requirements of power grid monitoring and control systems at the control center level (on the order of 100-200ms update latency).

The Spire system includes a SCADA Master and PLC/RTU proxy designed from scratch to support intrusion tolerance, as well as several example HMIs based on pvbrowser. The SCADA Master is replicated using the Prime intrusion-tolerant replication engine. Communication between Spire components is protected using the Spines intrusion-tolerant network. The Spire PLC/RTU proxy can interact with any devices that use the Modbus or DNP3 communication protocols over IP. We use OpenPLC to emulate PLCs that can be monitored and controlled by the system.

As the research progressed, Spire evolved into a toolkit with components that can support different deployment options, and that can provide intrusion-tolerance capabilities for power grid control systems at both control center and substation levels. Specifically, Spire 2.0 release includes: Spire for the Substation and Confidential Spire in addition to the original base Spire.

Confidential Spire is an intrusion-tolerant SCADA system that provides the same resilience guarantees as the base Spire. However, Confidential Spire enables system operators to maintain strong confidentiality guarantees for potentially sensitive or proprietary system data, while still leveraging commodity data centers to support cost-effective network-attack resilience. In Confidential Spire, only replicas hosted in the control centers execute SCADA logic and process system updates. Data center replicas participate in the replication protocol, but only process and store encrypted state and updates. No application logic or unencrypted application data is exposed to the data center replicas

Confidential Spire consists of the same modules as Spire (modified to support confidentiality). The main change is that Spire’s standard SCADA master is replaced by the Confidential SCADA Master, which additionally performs the needed encryption/decryption of requests and state, along with generating threshold signatures on encrypted contents to prove their validity to data center replicas.

Confidential Spire is described in the paper “Toward Intrusion Tolerance as a Service: Confidentiality in Partially Cloud-Based BFT Systems“ published at [IEEE DSN 2021]

Spire for the Substation is built to support the real-time Byzantine resilience required for power grid substations. The system is designed to withstand both system-level protective relay intrusions and network attacks on substation LAN, while meeting the stringent quarter of a power-cycle latency requirement (4.167ms).

The Spire for the Substation includes a Trip Master, Relay Proxy and Breaker Proxy. Additionally, we provide emulated relays to simulate real substation fault-free and faulty operating conditions. We support substation communication protocol of IEC61850 using open-source libiec61850.

Spire for the Substation is described in the paper “Real-Time Byzantine Resilience for Power Grid Substations” published at [SRDS 2022]


Spire 2.1b is the latest release. It extends Spire o support reconfiguration.

Spire 2.0 extends the Spire 1.3 to support real-time Byzantine resilience of power grid substations. This release includes Spire for the Substation code that successfully withstood a red-team attack conducted by Sandia National Laboratories in an exercise at Pacific Northwest National Laboratory (PNNL) in 2022. Furthermore, it includes Confidential Spire, a system that provides the same resilience guarantees as the base Spire. However, Confidential Spire enables system operators to maintain strong confidentiality guarantees for potentially sensitive or proprietary system data, while still leveraging commodity data centers to support cost-effective network-attack resilience.

Spire 1.3 updates Spire 1.2 to use Spines 5.5, OpenSSL 1.1.1 and includes a new ML-based NIDS (Network Intrusion Detection System) framework. This version of Spire uses Prime 3.3.

Spire 1.2 updates Spire 1.1 to use Spines 5.4, fixing a bug in Spines that could affect Spire in certain configurations. The Spire 1.1 release consists of the version of the Spire code that was used in a test deployment with the Hawaiian Electric Company (HECO) from January 22 to February 1, 2018. This test deployment was conducted by the DSN lab and Spread Concepts LLC as part of a DoD ESTCP project led by Resurgo LLC. This version of the code was deployed using Prime 3.1 and Spines 5.3.

Spire 1.1 builds on the Spire 1.0 release, which consisted of the version of the Spire code that successfully withstood a red-team attack conducted by Sandia National Laboratories in an exercise at Pacific Northwest National Laboratory (PNNL) from March 27 to April 7, 2017, as part of the same DoD ESTCP project. This version of the code was deployed using Prime 3.0 and Spines 5.2.

Spire 1.1 supports six different example SCADA systems, with their associated HMIs:

  • jhu: an example system we created to represent a power distribution system with 10 substations, each monitored and controlled by a different PLC or RTU
  • pnnl: the exact system that was used in the red-team exercise at PNNL, where it monitored and controlled a real PLC provided by PNNL
  • heco_3breaker: the system that was deployed at the Hawaiian Electric Company, monitoring and controlling to a real PLC that controlled three physical breakers
  • heco_5breaker: a system similar to heco_3breaker but including two additional breakers
  • heco_timing: the system used at the Hawaiian Electric Company to measure the end-to-end response time of the system by flipping a breaker and measuring the time for the HMI to reflect the change
  • ems: a system modeling an Energy Management System (EMS) that controls several different types of generators with different ramp-up rates and renewable energy sources that can be connected to the grid or deactivated

The SCADA Master of Spire 1.1 can support all of these systems; we provide a separate HMI for each system. Note that because the pnnl and heco systems use the same underlying infrastructure, only one of the pnnl, heco_3breaker, heco5_breaker, and heco_timing systems can be run at once. However, any one of these systems can be simultaneously run with both the jhu and ems systems. We also provide emulated PLCs for both systems that were created using OpenPLC.

Spire was created by Yair Amir, Trevor Aron, Amy Babay, Thomas Tantillo, Sahiti Bommareddy, and Maher Khan. It is currently developed by the Distributed Systems and Networks Lab at Johns Hopkins University and by the Resilient Systems and Societies Lab at the University of Pittsburgh.

Available materials describing Spire include:

Spire has also been featured in the following articles:

Rack with Spire at HECO
The Spire system installed at HECO.
Spire in action
The Spire system in action. Each of the six computers in the center runs a SCADA Master replica, a Prime daemon, and two Spines daemons. The monitors show three HMIs. One switch connects the replicas to the HMI and a PLC proxy (not shown), while the other is exclusively for communication among the six replicas.
Spire in Rack at HECO
The Spire system installed at HECO.

Funding

Partial funding for Spire is provided by the Department of Energy as part of a project on Byzantine Resilience in collaboration with Pacific Northwest National Laboratory (PNNL).

Partial funding for Spire was provided in the past by the Defense Advanced Research Projects Agency (DARPA), as part of our project Toward Intrusion Tolerant Clouds under the Mission-Oriented Resilient Clouds (MRC) program, and by the Department of Defense (DoD) as part of the Environmental Security Technology Certification Program (ESTCP) in the Energy and Water project led by Resurgo LLC. Spire is not necessarily endorsed by DARPA or the DoD.

Software

Spire is available for download. Please contact us if you are interested in learning more about Spire.

Releases

  • Version 2.1b - November 8, 2023
  • Version 2.0 - January 27, 2023
  • Version 1.3 - December 23, 2020
  • Version 1.2 - November 26, 2018
  • Version 1.1 - March 14, 2018
  • Version 1.0 - May 17, 2017

See the Changelog for release details

License

Spire may be freely used and distributed under some conditions. Please review the license agreement for more details.