Software-Defined Substation: Why vPAC Is Only the First Layer

A software-defined substation is one in which the whole secondary system is described in software, deployed from a version-controlled definition, and managed across its service life by an automation toolchain instead of by hand. vPAC is the workload. Software-defined is the operating model around it.

Most discussions of virtualization in substations stop at the hypervisor. A hardened server, a real-time kernel, a handful of virtual machines, and the observation that the protective relay is now software. All of that is correct, and it is the smaller part of the change. The larger part is everything that happens to those virtual machines afterward, across a fleet of substations, for the next fifteen years.

This article is about that larger part. It is written for the people who will have to live with the result, which is to say protection and substation automation engineers, and it takes their objections seriously rather than treating them as resistance to be managed.

The Diagram Has One Axis Too Few

The standard picture of the digital substation runs vertically. Process level at the bottom, bay level in the middle, station level on top, with a gradient from hardware-defined at the bottom to software-defined at the top. It is a good picture. It answers the question "which functions can be virtualized" and it shows correctly that the answer keeps moving downward, from gateway and HMI, to bay control, and eventually to protection itself.

Figure 1: The conventional digital substation architecture
The conventional digital substation architecture and the question it answers: which functions can run as software. Process level, bay level, and station level, with the boundary between hardware-defined and software-defined functions descending over time, from telecontrol gateway and station HMI to bay control and eventually to protection itself.

What it does not show is time. A substation is not commissioned and then finished. Over three decades it will absorb settings changes, security patches, firmware updates, added functions, replaced hardware, and at least two complete secondary system refreshes. The vertical axis tells you what runs where. It says nothing about how any of it changes after the commissioning report is signed.

That missing axis is where the difference between a virtualized substation and a software-defined one actually lives. You can virtualize every function in the diagram and still operate the result exactly the way relay panels have been operated since 1985: one engineer, one laptop, one site, one undocumented change at a time. That is virtualization without a software-defined operating model, and it captures very little of the available benefit while adding a hypervisor to the list of things that can go wrong.

Figure 2: Two axes, not one
Two independent axes define the software-defined substation: how much of the secondary system runs as software, and how the substation is operated across its life. Virtualization moves a substation up; only the operating model moves it right.

Reading the two axes as independent has a consequence worth stating plainly. You can move to the right without moving up at all. Putting IED configuration under version control, automating inventory collection, and generating compliance evidence by scanner are all available today, on the equipment already installed, with no virtualization anywhere in the picture. That is the quickest available return in this whole subject and it requires no protection engineer to accept anything new.

The direction of travel on the vertical axis also deserves precision, because it runs downward rather than upward. Virtualization arrives at station level first, where the gateway and the HMI have no timing obligation worth mentioning. It reaches bay level next. It reaches the trip path last, and that step is the difficult one: the process bus carries thousands of Sampled Value frames per second from the merging units that digitize the current and voltage transformer signals, with no retransmission mechanism, and the timing budget is fixed by the power system rather than by anyone's roadmap.

The difference between the two operating models is easiest to see task by task.

Table 1 — The same eight lifecycle tasks in a conventional and a software-defined operating model. Note that none of these tasks concern the trip path.

IEC 61850 Already Made the Substation Declarative

Here is the part of this argument that usually surprises people on both sides of it. Protection engineers have been writing declarative infrastructure descriptions since IEC 61850 Edition 1.

An SCD file is a machine-readable statement of what the substation is supposed to be. Which logical nodes exist, which datasets are published, which subscriber consumes which stream, which control block publishes on which interval. Nobody in the industry calls this infrastructure as code. That is precisely what it is, and the industry adopted it for the reason that always drives such a change: hand-maintained wiring lists and per-vendor configuration files did not scale, could not be verified mechanically, and could not be handed to the next engineer with any confidence.

The software-defined substation applies the same idea one layer down.

The SCD file describes the application. It says nothing about which kernel version runs underneath, which CPU cores are isolated for the protection instance, which NIC is passed through and which is bridged, which firewall rules are in place, which packages are installed, which accounts exist, or when any of that last changed. Today that layer usually lives in a commissioning report, a folder of screenshots, and the memory of whoever was on site. It is the least documented and least reproducible part of the most safety-relevant system in the substation.

So the honest framing of software-defined is not "a new IT idea arriving in the substation." It is "finish the job IEC 61850 started, one layer further down."

Table 2 — What is declared where in a software-defined substation. Only the top row exists as a declared artifact in most substations today, and the bottom row has no description format at all.

The bottom row is the weak one, and it is worth being explicit about why. The chain of declared artifacts runs from the application layer down to the operating system image and then stops. There is no equivalent of SCL for the compute platform itself, no agreed file in which a substation server's class, interface layout, timing capability, and thermal envelope are written down in a form another tool can read. The vPAC Alliance Server Profile Levels give the industry a shared vocabulary for sizing, which is genuinely useful in a tender and is not the same thing as a description format. Inventory can be read from a running machine through Redfish or IPMI where the platform supports them, which is how fleet inventory gets built in practice, but that tells you what a server is rather than what it was specified to be.

We have not found a published IEC standard that closes this gap. IEC 61850-3 and IEEE 1613 set environmental and EMC requirements rather than a description format, and if something does emerge it will most likely come through the liaison between IEC TC 57 and CIGRE SC B5, whose scope now covers the virtualization of protection and substation automation systems explicitly. Until then the hardware layer is described in prose and pinned down in a tender, which is the condition the rest of this article argues against.

Figure 3: What is declared, and where
The declarative stack of a software-defined substation: IEC 61850 SCL files already declare the application layer, and workload placement, platform configuration, and the operating system image become declared artifacts one layer at a time. Only the hardware profile still lacks a machine-readable description format.

What the Management Plane Actually Contains

It helps to separate three planes and be clear about who owns each.

Table 3 — The three planes of a software-defined substation. Virtualization alone delivers the middle one. The value case rests mostly on the third.

Figure 4: The three planes of a software-defined substationThe three planes of a software-defined substation. The IEC 61850 data plane with Sampled Values, GOOSE, and PTP is unchanged by virtualization; the control plane adds workload placement and failover; the management plane, which handles builds, deployment, patching, and evidence, is the part that is new.

The data plane is not up for discussion, and nothing in this article proposes changing it. Sampled Values still arrive every 250 microseconds at 50 Hz under the IEC 61850-9-2LE protection profile (80 samples per cycle, with IEC 61869-9 allowing other rates), the trip still has to be issued inside the same budget it always did, and the platform requirements for achieving that are covered in our technical explainer on vPAC.

The management plane is the part that barely exists today, and it is not a theoretical construct. It has a reference implementation.

SEAPATH, the LF Energy platform now running in production at a European transmission system operator, is an Ansible project as much as it is a virtualization platform. The cluster network configuration, the high-availability setup, the firewall rules, the system hardening, and the deployment of individual virtual machines are all done through Ansible playbooks and roles held in a public repository. A machine joins the cluster because an inventory file says it should, not because someone configured it. Version 2.0, released in June 2026, added automated CI/CD pipelines for official and weekly releases along with initial high-availability support for containerized workloads under Podman. That reference platform is itself built and shipped by a pipeline. What that relationship means for the wider ecosystem is covered in our article on vPAC, the vPAC Alliance, LF Energy, and SEAPATH.

Red Hat's published vPAC demonstrator uses the same pattern with a different stack. Deployment and security compliance are handled by Ansible Core, the host is scanned against CIS benchmarks with OpenSCAP, and the operator-facing view is a web console rather than a serial cable. Their description of Ansible as a workforce multiplier is worth reading in full, because it names the actual constraint most utilities are under.

Proprietary stacks are equally real in production. Commercial virtualized protection and control products exist and run as virtual machines under a hypervisor rather than on hardware of their own, and utility programs built on VMware have been in service for years. A utility that already operates vSphere in its control center and wants the same tooling in the substation is making a defensible choice, subject to the licensing caveat discussed further down.

That is the point worth holding on to. Nothing in this article requires open-source software. Two implementations happen to be documented publicly in enough detail to serve as evidence, which is why they are cited here rather than because they are the only options. The management plane is a list of functions, and every stack answers that list somehow. What matters is whether the functions are present and whether the definitions can be moved when the commercial terms change.

Table 4 — The twelve functions a substation management plane has to provide, with examples of how each is met. The functions are the requirement. The products in either column are substitutable, which is the whole point.

One row in that table deserves more than a cell. Real-time capability is what separates a hypervisor that can carry a protection instance from one that cannot, and the difference is architectural rather than a matter of tuning. KVM with a real-time kernel and isolated cores can hold a trip path, which is why both published vPAC reference stacks are built on it. Microsoft Hyper-V has no real-time scheduling class and is not designed for bounded latency, so a virtual IED does not belong on it.

That is not a criticism of the product. Hyper-V is deployed in substations today in significant numbers for station-level workloads such as the station HMI and the telecontrol gateway, and for those it is a perfectly reasonable choice. The mistake to avoid is assuming that a hypervisor standard set by the IT organization for station-level virtualization extends downward to the trip path. It does not, and that boundary belongs in the architecture document rather than being discovered during a factory acceptance test.

The operating system itself can be delivered in more than one way, and the choice has consequences that show up years later rather than during commissioning.

Red Hat now supports two deployment models. Package mode is the familiar one: install RHEL 9 or RHEL 10, then apply packages and configuration on top of it. Image mode, generally available and supported in RHEL 9.6 and 10 and newer, treats the operating system as an OCI container image built from a Containerfile using a tool called bootc. The image carries the components normally left out of an application container, including the kernel, the initrd, and the bootloader, so the running system is one versioned artifact rather than the outcome of a long sequence of package transactions. Updates are transactional and require a reboot, which RHEL 10 can soften to a userspace-only soft reboot when the kernel is unchanged, the previous image stays on disk, and Red Hat documents a path for applying updates in offline and air-gapped environments.

SEAPATH reached a similar destination from a different direction. Its Debian model gives package-based lifecycle management and its Yocto model gives fully reproducible builds from source. The two ecosystems are also less separate than the branding suggests, since Red Hat published a SEAPATH-on-RHEL reference architecture in November 2025 and contributes to the project.

Table 5 — Three ways to deliver the operating system to a substation node. Only the two image-based rows make rollback a boot selection rather than a repair job.

For a protection host, one question decides whether the image-based models are usable at all, and it should be settled before a platform is selected rather than after. The base bootc image ships with a kernel included, so moving to the real-time variant is a build-time decision inside the image rather than a package installed on a running system. Whether the vendor supports and validates that combination is a fair question to put in an RFQ, and the answer belongs in the technical specification rather than in a conversation.

Figure 5: From repository to substation
How a change reaches a substation in a software-defined operating model: from a reviewed repository difference through an automated build, a signed artifact store, and hardware-in-the-loop laboratory validation to a staged rollout, with release approval by protection engineering gating the trip path.

The Conservative Position Was Right for Forty Years

Protection engineers are careful for good reasons, and the reasons deserve to be stated rather than dismissed.

A protective relay is the last line of defense between a fault and a fire, and the discipline has two words for what can go wrong with it. A loss of security is an unwanted operation: the scheme trips when it should not, and the result is an outage with a customer number attached to it. A loss of dependability is a failure to trip when required, and the result is equipment damage or an arc flash. The vPAC Alliance specification adopts the same definitions and treats reliability as the combination of the two. It is also explicit that security in this sense is a property of relay operation rather than of access control, which is worth keeping in mind in an article that otherwise uses the word in its cyber sense. Against those two consequences, "do not touch a system that is working" is not stubbornness. It is a correct risk heuristic, learned expensively, and it has kept European grids at availability figures that most industries cannot approach.

The heuristic rests on two assumptions. The first is that a system left alone stays as safe as it was. The second is that the hardware will be available for as long as the primary plant needs it. Both assumptions have quietly failed.

A system left alone no longer stays as safe as it was. The threat environment moves whether or not the substation does. A device that cannot practically be patched is not unchanged. It is accumulating unmitigated vulnerabilities at the rate the world discovers them, and the operational reason it goes unpatched is rarely technical. It is that patching costs a site visit, a permit, an outage window, and a person who is already scheduled elsewhere.

Regulation has caught up with this. In Germany the NIS2 implementation act entered into force on 6 December 2025 with no transition period, which means risk management and incident reporting duties applied from day one. The EU Cyber Resilience Act entered into force on 10 December 2024, its reporting obligations for actively exploited vulnerabilities apply from 11 September 2026, and its full requirements apply from 11 December 2027. The reporting obligations reach products already on the market, not only new ones. A utility running equipment it cannot update is now carrying a documented obligation it has no mechanism to discharge.

North America arrived at the same place from a different direction and rather earlier. The NERC CIP standards have been prescriptive about patching and configuration for a decade, and the prescription is expressed in calendar days. What the European instruments have added is a reporting clock; what the North American standards have always had is an evidence burden. Both end in the same place, which is that a manual process cannot produce the record.

Table 6 — Regulatory and normative drivers behind the operating model change in Europe and North America, with the dates that determine planning.

The 35-day figure in two of those rows is worth sitting with. A fleet of four hundred substations under CIP-007 means a patch evaluation cycle that closes every 35 days, forever, with dated evidence each time. Under CIP-010 it means every applicable system carries a documented baseline and gets checked for unauthorized change on the same cadence. Neither requirement mentions automation. Both are close to unreachable without it, which is why North American utilities generally arrived at configuration management tooling before their European counterparts did, and why the vocabulary in this article will already be familiar to anyone who has sat a CIP audit.

CIP-015 is the interesting one for a software-defined substation, because it asks for something the conventional architecture cannot easily supply. Monitoring inside the security perimeter requires knowing what normal internal traffic looks like, and a substation whose configuration was assembled by hand over fifteen years has no authoritative answer to that question. A substation whose network, workload placement, and communication paths are declared in files has the answer already, as a byproduct.

The hardware will not last as long as the plant. Primary plant runs 30 to 40 years. Secondary system hardware runs perhaps 12 to 15. Nothing about that arithmetic is new, and it is the reason the vPAC economic case exists at all.

And there are fewer people every year. The engineers who can walk into a 1990s panel and read it from memory are retiring. At the same time the number of substations needing modification is going up, driven by distributed generation connections, storage, and load that did not exist when the schemes were designed. The per-site manual model does not scale to that workload at any staffing level anyone is actually going to be granted.

Standing still is therefore not the zero-risk option any more. It is a risk position that simply has no signature on it.

Automation Removes Variance, Which Is What Reliability Means

This is the argument that matters most, so it is worth being concrete.

Consider what happens today when a platform-level change is needed on a station computer. Platform configuration in this sense is a different object from a protection setting or a setting group, and nothing here proposes moving those out of the protection engineer's hands. An engineer travels to the site. A laptop is connected. A vendor tool is opened, a value is changed, a service is restarted, and afterward a report is written describing what was done. The change and its documentation are two separate artifacts, related to each other only by the diligence of the person doing the work. Nothing verifies that they match. Nothing verifies that the same change was applied identically at the other eleven sites in the program. Nothing records what the system looked like beforehand.

Now count the failure modes. A step performed out of order. A setting applied to bay 3 instead of bay 4. A temporary firewall rule opened for the tool and never closed. A restart that did not happen because it was going to be done at the end and the end came at 17:40 on a Friday. An undocumented deviation made for a good local reason that nobody will remember in four years, and that quietly makes the next change behave differently at this one site.

None of this describes careless people. It describes the highest-variance process in the substation being applied to its most safety-relevant system.

Automation is not attractive because it is fast. It is attractive because it is invariant. An Ansible playbook is idempotent, which means running it twice produces the same system state as running it once, and running it on twelve sites produces twelve identical systems. Configuration drift between bays and between substations stops being something you discover during the next audit and becomes something the tooling cannot produce in the first place. The change exists as a reviewable text difference before it exists as a running system. It can be inspected by a second engineer, approved, applied, and reverted, and every one of those steps leaves a record that was generated by the machine rather than typed up afterward.

Borrowing from DevOps does not mean borrowing the release cadence. Nobody serious is proposing continuous deployment to a distance protection function, and a proposal along those lines has misunderstood which part of the practice transfers and which part does not. What is worth borrowing is narrower and older than the buzzword: describe the desired state in a file, keep the file under version control, test the change on a system identical to the target, let a machine apply it the same way every time, and keep the previous state available to return to.

The objections that come back at this point are usually good ones, and each has a technical answer that does not require anyone to change their risk appetite.

Table 7 — The six objections that come up most often, and what can honestly be said in response to each.

Updating a Protection System Without Having to Trust Anybody

The reasonable follow-up question from a protection engineer is: fine, but how do you validate a change to a system that is protecting live plant?

Most of the answer already exists in the substation. It just was not designed for this purpose.

Main 1 and main 2 are the update mechanism. Redundant protection instances run on physically separate nodes because cluster failover takes seconds while a trip decision takes milliseconds. That separation was designed for hardware failure, and it works exactly as well for planned change. Main 2 is patched while main 1 protects. It is brought up against the live process bus and observed under real load before it is given responsibility back. Then the roles swap and the other side is done. No outage window, no fleet-wide exposure, and at every moment one instance is running the version that was working this morning. The scheme does not need to be invented. It needs to be used for something it was not originally intended for. Put in the discipline's own terms, the standby-side update is what protects dependability, because one instance is live throughout, and the verification step before the roles swap is what protects security.

The lab is the same system as the site. This is the part that makes automation credible rather than alarming. If the site is provisioned by a playbook, the lab can be provisioned by the same playbook, and it will not be approximately the same. It will be the same. Add a hardware-in-the-loop simulator and you can inject faults into a digital twin of the actual feeder and measure the trip loop end to end. Red Hat's demonstrator does exactly this on a ten-bay medium-voltage model, including automated stress runs over 25-hour cycles with a fault injected every minute, which is a validation depth that no site visit has ever produced.

Validation becomes regression testing rather than sampling. Once the laboratory is provisioned from the same definition as the site, testing attaches to the pipeline instead of being scheduled as an event. A change to an SCL file, to an operating system image, or to a protection workload triggers a build, and the build is deployed into the digital twin without anyone asking for it. A real-time simulator or an automated COMTRADE playback device then runs the accumulated fault library against it: evolving faults, cross-country faults, CT saturation, frequency excursions, and whatever else the utility has collected from real events. The protection engineer still signs the release. What changes is that the signature now rests on several hundred replayed scenarios rather than on the handful anyone has time to run by hand.

Figure 6: Inside the laboratory
Inside the validation stage of a software-defined substation: because the laboratory is provisioned from the same declaration as the site, every build triggers automated hardware-in-the-loop regression testing against a digital twin before a protection engineer signs the release.

Rollback is a boot, not a rebuild. With image-based delivery, the previous known-good state is an artifact you still have. Under Yocto or under RHEL image mode the earlier image remains on disk, so reverting is a boot selection rather than a reconstruction from a change log at two in the morning.

Rolling maintenance covers the functions that tolerate a restart. Bay control, the telecontrol gateway, the station HMI, and the engineering workstation can be migrated between nodes in a cluster and updated one node at a time. A SEAPATH cluster is defined as three machines, and the third can be a lighter observer node rather than a full hypervisor; the architecture exists for rolling maintenance as much as for failure tolerance.

Table 8 — How each class of function is updated. The trip path is the only row where the answer depends on redundancy design rather than on the cluster.

Figure 7: Patching without an outage window
Patching virtualized protection without an outage window: the standby instance is updated and verified against live Sampled Values with trip outputs blocked while the active instance holds the trip path, then the roles swap. The live instance protects dependability; the verification step protects security.

The fastest network path is also the least movable. A virtual IED reaches the process bus either through PCIe passthrough, which assigns a whole physical port to one virtual machine, or through SR-IOV, which divides a port into virtual functions so several instances can share it while each still bypasses the hypervisor's software switch. Which route a platform uses, and how it handles hardware timestamping and multicast on a virtual function, is a workload question covered in our technical explainer. The lifecycle consequence is the part that belongs here. A workload bound to a physical port or to a virtual function is pinned to that node, and live migration is not available to it in the general case. The functions with the tightest timing are therefore precisely the ones the cluster cannot move, which is a second and independent reason why protection continuity has to come from main 1 and main 2 rather than from the hypervisor.

Network redundancy is becoming a declarable artifact. The process bus rides on PRP or HSR because a lost Sampled Value frame is simply gone, and until recently the link redundancy entity that duplicates and discards those frames lived in dedicated hardware. The Linux kernel has carried a software implementation since 2020, and Red Hat moved it from technology preview to full support in RHEL 9.8 and 10.2. That is a management-plane milestone rather than a performance one: a redundancy function that used to be a hardware purchase can now be declared, patched, and supported on the same commercial terms as everything else on the node, with a CVE lifecycle behind it.

It is not, on its own, a process bus answer. The kernel's own configuration text describes the implementation as a best effort toward IEC 62439-3 with no conformance testing behind it, which is not a sentence a protection acceptance test wants to meet, and a guest reached through SR-IOV or passthrough bypasses the host kernel entirely, so host-side PRP is never in its path. Software PRP earns its keep on the station bus and the management network. The process bus still argues for redundancy implemented in hardware.

Be clear about what this does not solve. Cluster failover remains far too slow to preserve a trip decision, so protection continuity still comes from main 1 and main 2 on separate physical nodes rather than from the hypervisor. The process bus still needs PRP or HSR under IEC 62439-3, because a lost Sampled Value frame has no retransmission mechanism and is simply gone. Orchestration improves the lifecycle. It does not repeal the real-time requirements.

What This Buys You Technically

Cybersecurity

Patch latency stops being a story and becomes a number. In the conventional model, the time between a published CVE and a remediated substation is measured in quarters and is bounded by travel and outage windows. In an automated model it is measured in days and is bounded by validation. That single quantity is the most honest security metric a utility can put on a slide, and it is the one that regulators are increasingly going to ask for.

Underneath it, several things become possible that are impractical by hand. Network separation stops depending on someone remembering which cable goes where: management traffic, station bus, and process bus are separated by design and reasserted every time the playbook runs. Red Hat's demonstrator uses a minimum of six separate interfaces on a single host for exactly this. Local accounts and shared passwords give way to centrally managed identity. Compliance evidence is produced by a scanner against a published benchmark rather than assembled by hand before an audit, which turns NIS2 and NERC CIP evidence from a project into a report. Software bills of materials stop being a document somebody assembles and become an output of the build. Each artifact emits a machine-readable inventory in CycloneDX or SPDX, the management plane retains those inventories, and a newly published CVE becomes a query rather than a project: which images contain the affected component, and which nodes in which substations are running them. That is what the Cyber Resilience Act will require in any case, and it is the difference between answering a regulator in an afternoon and answering in a quarter.

Figure 8: From build to CVE answer
From build to CVE answer in minutes: every image build emits a software bill of materials in CycloneDX or SPDX, the management plane ingests it into a fleet inventory, and a newly published vulnerability becomes a query instead of a site-by-site audit. This is the workflow the EU Cyber Resilience Act assumes.

The three planes also map onto the zone and conduit model of IEC 62443-3-2 without strain, which matters when the argument has to be made to people who have no interest in hypervisors. The data plane is the most restricted zone, isolated by dedicated physical ports or by virtual functions that carry no management traffic. The control plane sits in the substation automation zone. The management plane sits outside the substation altogether. What joins them is a conduit rather than a flat network, and the direction of that conduit is the entire security argument: the substation opens an outbound TLS session and pulls signed artifacts, and nothing in the center is able to open a session into the substation. Zone numbering is assigned by the asset owner's risk assessment rather than by the standard, so the labels will differ between utilities. The structure does not.

There is a real counterweight here, and it is discussed below.

Reliability

The reliability gain from a software-defined operating model is not that software fails less often. It is that the failures become the ones you have already seen.

Configuration drift is a quiet, recurring cause of commissioning surprises and of the class of fault where one bay behaves differently from its eleven identical siblings for reasons nobody can reconstruct. Declarative configuration removes the mechanism that produces drift. Self-supervision is not new to protective relays, and a virtualized instance extends it to its own processing latency, exported continuously rather than read out during a visit. Degradation becomes visible as a trend rather than discovered during an event. Fleet-wide, you gain something the conventional architecture never offered, which is the ability to ask a question like "which of my 400 substations is running a kernel older than the current baseline" and get an answer in seconds rather than a work package.

Flexibility

The flexibility argument is usually made with panel space, and panel space is the least interesting version of it.

The interesting version is adaptive protection. As distributed generation changes fault levels and direction of flow through the day, fixed settings calculated for a worst-case study become conservative most of the time and occasionally wrong. Protection settings that follow the actual network state require a mechanism to compute, validate, distribute, and record settings across many sites under control. That mechanism is an orchestration layer. There is no version of adaptive protection that is deployed by hand.

Published utility programs have been built on exactly this reasoning, virtualizing protection alongside active network management and adaptive protection, and using routable IEC 61850 between vPAC instances so that neighboring stations can cooperate in locating a disturbance and isolating only the affected section. The business cases behind them were framed in grid terms rather than IT terms, in avoided reinforcement, hosted generation capacity, and outage minutes rather than in server consolidation ratios. Whatever one thinks of any individual set of projected figures, the architectural point stands: the value came from coordination between sites, and coordination between sites is a management-plane capability.

Every claim in this section should survive contact with a business case, which means each one needs an indicator attached to it.

Table 9 — Benefits, the mechanism behind each one, and the indicator that tells you whether it materialized.

Where the Real Risks Are

An article that only listed benefits would not be worth much to the audience it is written for.

Table 10 — Risk register for a software-defined substation program. The second row is the honest counterweight to the variance argument made earlier.

The second row deserves saying out loud, because it is the price of the argument this article makes. Automation applies a correct change identically everywhere, and it applies an incorrect one identically everywhere too. A manual process contains errors by containing everything. The mitigations are staged rollout and a pilot site rather than confidence, and any program that cannot roll back within minutes has not finished building its management plane.

The management plane is a new high-value target. A system that can reach every substation is worth attacking more than any single relay. This is the strongest objection to the whole approach and it deserves a real answer rather than reassurance. The answer is that the orchestration layer has to be engineered as an OT asset rather than adopted as an IT convenience: signed artifacts, deployment that the substation pulls rather than accepts, separated management networks, multi-party approval for anything touching a trip path, and an assumption in the design that the central system will eventually be compromised. Utilities that treat automation as a productivity tool and skip this will build a fragile grid with a very tidy audit trail.

Air-gapped is a design constraint, not an objection. Orchestration does not require permanent connectivity to a substation. What it requires is that artifacts be signed and that the substation be able to verify them without asking anyone. Those are two different problems, and only the second one needs a network.

The mechanism is worth describing concretely, because the phrase "carried to site" hides the part that does the work. The center assembles a release exactly as it would for a connected substation: an operating system image, workload definitions, configuration roles, a software bill of materials, and a manifest listing every artifact with its hash. The whole set is signed once and written to removable media prepared on a dedicated staging host rather than on whichever laptop is nearest. At the substation, the cluster verifies that signature against a trust anchor installed at commissioning. Verification happens locally, offline, before anything is applied, which is the entire reason for signing artifacts instead of relying on a secure channel. A bundle altered in a vehicle, in a hotel room, or on somebody's desk fails verification and never reaches the node.

The return path matters as much as the outbound one, and it is the part most descriptions leave out. After the roles run, the node writes an evidence file recording what was applied, when, the resulting inventory, and the scan results. That file goes back on the same medium and is ingested at the center. A disconnected substation therefore does not drop out of the fleet record. It reports in batches, with the latency of a site visit rather than of a network, which is a very different thing from reporting nothing.

Table 11 — The offline path for a substation with no permanent connection. Every step leaves evidence behind, which is what makes the disconnected case auditable rather than merely workable.

Three things have to be engineered rather than assumed. The medium is itself an attack path, and the industry has known this since Stuxnet. NERC CIP treats it explicitly as Removable Media under CIP-010 Attachment 1 and requires authorized use and detection of malicious code before connection, so the procedure is regulated rather than improvised, and the signature check at the node is the control that actually stops a tampered bundle. Trust anchor rotation needs a plan that works without connectivity, and the usual plan is to carry the new anchor on the same medium, signed by the old one, well before the old one expires. Revocation is the genuinely hard case, because a disconnected node cannot consult a revocation list, so a compromised signing key is contained only at the next visit. That is an argument for short-lived signing keys and a documented emergency visit procedure. It is not an argument against the model, since a manually maintained substation has no revocation story at all.

One detail works in your favor here. Signature and certificate checking depends on the node knowing the correct time, and a substation running PTP already keeps better time than most data centers.

Disconnected deployment is under active development in the SEAPATH project, and Red Hat documents an offline path for image mode updates. It is a legitimate target architecture rather than a fallback.

Neither mode is the correct one. They trade different things, and the trade is worth setting out plainly before a program commits to either.

Table 12 — What connected and disconnected substations actually trade against each other. The disconnected column is a different risk profile rather than a worse one.

The choice is made per site rather than per fleet, and most utilities will end up running both. A dense urban ring with reliable fiber and a remote site reachable by a two-hour drive do not need the same answer. What matters for this article is that both modes run from the same declaration. A site can move from carried media to a live conduit, or back the other way after a security review, without anyone rewriting what the substation is supposed to be. That portability is a property of the declaration, and it is one of the strongest practical arguments for having one.

One rule follows from this, and it belongs in the specification before anything is procured. The substation has to keep running when the management plane is unreachable. A cluster that has lost contact with the center holds its current state and keeps executing protection and control. Logs are cached locally until the link returns. Loss of contact is a reporting outage and never a protection outage, and a platform that cannot demonstrate that behavior does not belong in a substation. It is worth asking in the same breath as the real-time performance question, because the answer is architectural rather than a configuration setting.

Lock-in migrates to the orchestration layer. Utilities that adopted virtualization early were, reasonably, buying a proprietary stack for it. Several of them then discovered that the commercial terms of that stack can change on the vendor's schedule rather than theirs, which is one lesson many utilities drew from the licensing changes that followed a change of ownership in the hypervisor market. The exposure is worse at the management layer than at the hypervisor, because a hypervisor can be swapped in a refresh cycle while an operating model built around one vendor's tooling touches every process the organization runs. The defense is portability of the definition. Playbooks, images, and manifests that describe the desired state in open formats can be moved. The hardware side of that argument is the reason the vPAC Alliance specification defines Server Profile Levels rather than a single server, so that platforms of different sizes and vendors can coexist in one architecture, as discussed in our comparison of the RSAPC Mk2 and the RSAVP.

Type testing and certification are genuinely unresolved. A conformance certificate applies to a defined combination of hardware and software. If the platform underneath a certified protection function is patched monthly, the industry does not yet have a settled answer to what the certificate covers, what constitutes a substantial change, and what has to be retested. This is one of the open items in the standardization work rather than a solved problem, and anyone claiming otherwise is selling something. Our overview of who is working on vPAC standardization sets out which body is responsible for which part of the question.

Accountability has to be written down first. If a playbook misconfigures core isolation and a trip is issued late, the question of who is responsible needs an answer that predates the event. The workable split is that the protection engineer keeps authority over protection settings and over the release of any change into the trip path, while platform engineering owns the layer beneath and is accountable for delivering it exactly as specified. That boundary is easier to draw with automation than without it, because for the first time there is an artifact that says what the layer beneath is supposed to be.

Figure 9: Fleet orchestration and the security boundary
Fleet orchestration under IEC 62443 zones and conduits: substations open outbound TLS sessions and pull signed artifacts, no inbound session reaches into the substation, and a disconnected site exchanges the same signed bundle and its evidence record on carried media.

A Migration Path That Does Not Start With Protection

The fastest way to discredit this approach in an organization is to begin with the trip path. There is a ladder that produces value earlier and costs far less credibility.

1. Virtualize the workloads with no protection function. The telecontrol gateway, station HMI, disturbance record collection, asset management, and the engineering workstation all run on the station computer with no impact on any trip decision. The organization learns the platform on workloads where a mistake is an inconvenience.
2. Put the configuration under version control before automating anything. Even applied by hand, a configuration that lives in a repository can be reviewed, compared between sites, and audited. This step alone finds drift that nobody knew about.
3. Automate reading before writing. Inventory collection, compliance scanning, and health reporting across the fleet are read-only, carry almost no operational risk, and produce the evidence base that the security and compliance argument depends on.
4. Build the lab as an identical target. The same playbooks, the same image, the same hardware profile. Add a hardware-in-the-loop simulator once the platform work is stable. Nothing after this point should reach a substation without having run here first.
5. Automate deployment for the non-critical workloads. By now the tooling has been exercised, the review process exists, and the rollback path has been used at least once in anger.
6. Move protection functions last, on separate physical nodes, keeping main 1 and main 2 independent. By this point the update mechanism, the validation environment, and the accountability model all exist. The protection function is the last thing to move, not the first.

Table 13 — The migration ladder, with the protection risk of each step made explicit.

Most utilities that are doing this well are somewhere between step two and step four, and the ones that are struggling generally tried to start at step six.

Terms Used in This Article

Table 14 — Terms used in this article, with the distinctions that are most often collapsed in vendor material. The middle column is where the categories in this article are actually staked.