Spire: Intrusion-Tolerant SCADA for the Power Grid
Overview
Spire is an open-source intrusion-tolerant SCADA system for the power grid. Spire is designed to withstand attacks and compromises at both the system level and the network level, while meeting the timeliness requirements of power grid monitoring and control systems at the control center level (on the order of 100-200ms update latency).
The Spire system includes a SCADA Master and PLC/RTU proxy designed from scratch to support intrusion tolerance, as well as several example HMIs based on pvbrowser. The SCADA Master is replicated using the Prime intrusion-tolerant replication engine. Communication between Spire components is protected using the Spines intrusion-tolerant network. The Spire PLC/RTU proxy can interact with any devices that use the Modbus or DNP3 communication protocols over IP. We use OpenPLC to emulate PLCs that can be monitored and controlled by the system.
As the research progressed, Spire evolved into a toolkit with components that can support different deployment options, and that can provide intrusion-tolerance capabilities for power grid control systems at both control center and substation levels. Specifically, Spire 2.0 release includes: Spire for the Substation and Confidential Spire in addition to the original base Spire.
Confidential Spire is an intrusion-tolerant SCADA system that provides the same resilience guarantees as the base Spire. However, Confidential Spire enables system operators to maintain strong confidentiality guarantees for potentially sensitive or proprietary system data, while still leveraging commodity data centers to support cost-effective network-attack resilience. In Confidential Spire, only replicas hosted in the control centers execute SCADA logic and process system updates. Data center replicas participate in the replication protocol, but only process and store encrypted state and updates. No application logic or unencrypted application data is exposed to the data center replicas
Confidential Spire consists of the same modules as Spire (modified to support confidentiality). The main change is that Spire’s standard SCADA master is replaced by the Confidential SCADA Master, which additionally performs the needed encryption/decryption of requests and state, along with generating threshold signatures on encrypted contents to prove their validity to data center replicas.
Confidential Spire is described in the paper “Toward Intrusion Tolerance as a Service: Confidentiality in Partially Cloud-Based BFT Systems“ published at [IEEE DSN 2021]
Spire for the Substation is built to support the real-time Byzantine resilience required for power grid substations. The system is designed to withstand both system-level protective relay intrusions and network attacks on substation LAN, while meeting the stringent quarter of a power-cycle latency requirement (4.167ms).
The Spire for the Substation includes a Trip Master, Relay Proxy and Breaker Proxy. Additionally, we provide emulated relays to simulate real substation fault-free and faulty operating conditions. We support substation communication protocol of IEC61850 using open-source libiec61850.
Spire for the Substation is described in the paper “Real-Time Byzantine Resilience for Power Grid Substations” published at [SRDS 2022]
Spire 2.1 is the latest release. It extends Spire to support reconfiguration.
Spire 2.0 extends the Spire 1.3 to support real-time Byzantine resilience of power grid substations. This release includes Spire for the Substation code that successfully withstood a red-team attack conducted by Sandia National Laboratories in an exercise at Pacific Northwest National Laboratory (PNNL) in 2022. Furthermore, it includes Confidential Spire, a system that provides the same resilience guarantees as the base Spire. However, Confidential Spire enables system operators to maintain strong confidentiality guarantees for potentially sensitive or proprietary system data, while still leveraging commodity data centers to support cost-effective network-attack resilience.
Spire 1.3 updates Spire 1.2 to use Spines 5.5, OpenSSL 1.1.1 and includes a new ML-based NIDS (Network Intrusion Detection System) framework. This version of Spire uses Prime 3.3.
Spire 1.2 updates Spire 1.1 to use Spines 5.4, fixing a bug in Spines that could affect Spire in certain configurations. The Spire 1.1 release consists of the version of the Spire code that was used in a test deployment with the Hawaiian Electric Company (HECO) from January 22 to February 1, 2018. This test deployment was conducted by the DSN lab and Spread Concepts LLC as part of a DoD ESTCP project led by Resurgo LLC. This version of the code was deployed using Prime 3.1 and Spines 5.3.
Spire 1.1 builds on the Spire 1.0 release, which consisted of the version of the Spire code that successfully withstood a red-team attack conducted by Sandia National Laboratories in an exercise at Pacific Northwest National Laboratory (PNNL) from March 27 to April 7, 2017, as part of the same DoD ESTCP project. This version of the code was deployed using Prime 3.0 and Spines 5.2.
Spire 1.1 supports six different example SCADA systems, with their associated HMIs:
- jhu: an example system we created to represent a power distribution system with 10 substations, each monitored and controlled by a different PLC or RTU
- pnnl: the exact system that was used in the red-team exercise at PNNL, where it monitored and controlled a real PLC provided by PNNL
- heco_3breaker: the system that was deployed at the Hawaiian Electric Company, monitoring and controlling to a real PLC that controlled three physical breakers
- heco_5breaker: a system similar to heco_3breaker but including two additional breakers
- heco_timing: the system used at the Hawaiian Electric Company to measure the end-to-end response time of the system by flipping a breaker and measuring the time for the HMI to reflect the change
- ems: a system modeling an Energy Management System (EMS) that controls several different types of generators with different ramp-up rates and renewable energy sources that can be connected to the grid or deactivated
The SCADA Master of Spire 1.1 can support all of these systems; we provide a separate HMI for each system. Note that because the pnnl and heco systems use the same underlying infrastructure, only one of the pnnl, heco_3breaker, heco5_breaker, and heco_timing systems can be run at once. However, any one of these systems can be simultaneously run with both the jhu and ems systems. We also provide emulated PLCs for both systems that were created using OpenPLC.
Spire was created by Yair Amir, Trevor Aron, Amy Babay, Thomas Tantillo, Sahiti Bommareddy, and Maher Khan. It is currently developed by the Distributed Systems and Networks Lab at Johns Hopkins University and by the Resilient Systems and Societies Lab at the University of Pittsburgh.
Available materials describing Spire include:
A presentation titled "Cyber-Resilient Power Grid Control Systems: Tales from the Bleeding Edge" that was given at the DRI 2024 Conference (March 2024). - A presentation about Spire that was given at CERAWeek (March 2023).
- A paper about Evaluation and Tradeoffs in Real-Time Byzantine Resilient Power Grid Infrastructure that was presented at the I`nternational Workshop on Explainability of Real-time Systems and their Analysis at the IEEE Real-Time Systems Symposium (RTSS) (December 2022).
- A paper about Real-Time Byzantine Resilience for Power Grid Substations that was presented at the International Symposium on Reliable Distributed Systems (SRDS) (September 2022).
- A paper about Confidential Spire titled "Towards Intrusion Tolerance as a Service: Confidentiality in Partially Cloud-Based BFT Systems" that was presented at the IEEE/IFIP Dependable Systems and Networks (DSN) (June 2021).
- Detailed instructions on configuring and running Spire
- A presentation about Spire that was given at the SCADA Tech Summit (August 2019).
- A paper about our experience red teaming and test deploying Spire that was presented IEEE/IFIP DSN 2019 and the presentation that was given (June 2019).
- A presentation about Spire that was given at the Army Engineer Association seminar (August 2018).
- A vision paper about challenges facing the power grid that was presented at the IEEE ICDCS 2018 conference and the presentation that was given (July 2018).
- A paper about Spire that was presented at the IEEE/IFIP DSN 2018 conference and the presentation that was given (June 2018).
- A presentation about Spire that was given at the Hawaiian Electric Company (January 2018)
- A presentation about Spire that was given at the IFIP 10.4 working group (June 2017)
- A poster describing Spire that was presented by Trevor Aron at the Johns Hopkins Day of Undergraduate Research in Engineering, the Arts and Humanities, Medicine, and the Sciences (DREAMS) (April 2017)
Spire has also been featured in the following articles:
- Johns Hopkins Magazine, December 2018
- POWER Magazine, February 2018
- Futurity, February 2018
- JHU Hub, February 2018
Funding
Partial funding for Spire is provided by the Department of Energy as part of a project on Byzantine Resilience in collaboration with Pacific Northwest National Laboratory (PNNL).
Partial funding for Spire was provided in the past by the Defense Advanced Research Projects Agency (DARPA), as part of our project Toward Intrusion Tolerant Clouds under the Mission-Oriented Resilient Clouds (MRC) program, and by the Department of Defense (DoD) as part of the Environmental Security Technology Certification Program (ESTCP) in the Energy and Water project led by Resurgo LLC. Spire is not necessarily endorsed by DARPA or the DoD.
Software
Spire is available for download. Please contact us if you are interested in learning more about Spire.
Releases
- Version 2.1 - February 29, 2024
- Version 2.0 - January 27, 2023
- Version 1.3 - December 23, 2020
- Version 1.2 - November 26, 2018
- Version 1.1 - March 14, 2018
- Version 1.0 - May 17, 2017
See the Changelog for release details
License
Spire may be freely used and distributed under some conditions. Please review the license agreement for more details.