Research
Firmware Faults Overtook Change Management as an Outage Cause
Configuration and change management is still the single largest cause of major network outages, at 41% of incidents in the Uptime Institute Annual Outage Analysis 2026. That share fell nine percentage points in one year. In IT systems and software the ranking has already flipped: firmware and software faults account for 53% of major outages, ahead of configuration and change management at 51%. Change discipline is working. The failure moved to a place your change process does not reach.
That is an uncomfortable result for anyone who has spent a career arguing that better change control prevents outages. The argument was right. The data now shows the win, and it shows what took the win's place. For a firewall team the practical question is no longer whether your change advisory board is disciplined enough. It is whether your change record covers the changes you did not author.
What the 2026 outage data actually says
The Uptime Institute Annual Outage Analysis 2026, published in May, is the eighth edition of the series. It draws on the Uptime Institute Global Data Center Survey 2025, the Data Center Resiliency Survey 2026, publicly reported incidents, and an anonymised incident database shared under NDA. Network-related issues remain the largest share of IT service outages in the 2026 resiliency survey.
Inside that network category, respondents who had suffered a significant, serious or severe network outage in the past three years ranked the root causes as follows.
| Root cause of major network outages | Share (n=122) | Direction vs 2025 |
|---|---|---|
| Configuration or change management failure | 41% | Down 9 points |
| Third-party network provider failure | 34% | Stable |
| Hardware failure | 34% | Rising |
| Firmware or software error | 29% | Rising |
| Line breakages | 25% | Up 8 points |
| Network overload or congestion | 14% | Stable |
| Malicious cyberattack | 9% | Stable |
| Corrupted firewall rules or routing tables | 8% | Stable |
| Weather-related incident | 5% | Stable |
Two lines in that table deserve more attention than they usually get. The nine-point fall in configuration and change failures is the largest single movement in the network category, and Uptime attributes the relative shift to other causes rising rather than to change failures disappearing. And corrupted firewall rules or routing tables, the outcome most firewall teams are explicitly organised to prevent, accounts for 8%.
The overtake happened in the software layer first
The clearer signal sits one category over. For outages caused by IT systems and software, Uptime writes that in previous years these were mostly driven by configuration and change management problems, and that the 2026 survey now shows firmware and software faults accounting for the largest share.
| Root cause of major IT system and software outages | Share (n=74) |
|---|---|
| Firmware or software fault | 53% |
| Configuration or change management issue | 51% |
| Hardware failure | 38% |
| Capacity or congestion issue | 19% |
| Data synchronisation or corruption | 15% |
| Cyberattack or security issue | 15% |
Uptime draws the distinction plainly: a configuration issue involves an incorrect setting or change, while a software or firmware fault stems from a bug or defect in the system itself. The report's conclusion is that outage risk is moving deeper into the technology stack, away from operator-driven errors and toward faults tied to software behaviour, updates, and interactions between components.
Two percentage points is a narrow margin on a sample of 74, and it would be wrong to treat the crossover as settled. The direction is what matters. One curve has been falling for years because organisations invested in change control. The other has been rising because there is more vendor code in the path than there used to be, and because none of it passes through your approval process.
A firmware update is a change with no change record
Every mature firewall change process has the same shape. A request is raised, reviewed for policy impact, scheduled into a window, implemented, verified, and recorded. The record is what makes the change auditable afterwards, and it is what a NIS2 or ISO 27001 assessor asks to see. Our firewall change management guide sets out that lifecycle in full.
A vendor firmware release enters the same production estate and skips every one of those steps. Nobody in your organisation wrote the code. Nobody reviewed its policy impact, because the impact is not expressed as a policy. It arrives on the vendor's schedule, often bundled with a security fix you have little practical choice about taking, and it can change packet handling, session table behaviour, NAT ordering, or default-deny semantics without a single line changing in your rule base.
That is the gap the 2026 data is pointing at. The rule base is under control. The platform underneath it is not, and it changes more often than the rule base does.
Why the human-error numbers do not contradict this
Uptime treats human error as a contributing factor rather than a root cause, and 92% of respondents said human error contributed at least minimally to their most recent impactful outage. Among human-error causes, failure to follow established procedures remains the leading driver at 59%, ahead of incorrect processes or procedures at 36%, installation issues at 25%, and in-service issues at 25%.
Read alongside the firmware finding, that is not a contradiction. It is the mechanism. Uptime notes the attribution problem directly: an outage from a misconfigured system update may stem from a software defect, an incorrect change, or failed testing, and untangling which is genuinely difficult. A firmware fault that a proper staged rollout would have caught in pre-production gets recorded as a firmware fault. It is also a procedure that was skipped.
The most useful figure in the report is from the 2025 survey: 87% of respondents who had suffered an impactful outage in the previous three years said it could have been avoided with better management, processes or configuration, up seven points on 2024. Nearly nine in ten outages were preventable by the organisation that suffered them, whoever wrote the defective code.
What this changes in a firewall change process
Treating vendor firmware as a change you did not author, rather than as maintenance, is the practical response. Five adjustments follow from the data.
- Put firmware versions in the change record. If your CMDB records rule changes but not the platform version those rules ran on, you cannot correlate an outage with the update that caused it. Version state belongs in the same record as policy state.
- Stage vendor code the way you stage rule changes. A lab pair, then a low-criticality site, then the estate. The 59% procedure-failure figure is mostly this control being skipped under time pressure.
- Test the rollback you actually have. Firmware rollback on a high-availability pair is not the same operation as reverting a rule commit, and on several platforms it is not reversible at all once a configuration schema migration has run. Our note on emergency firewall change rollback covers the failure modes; high-availability failover testing covers the pair behaviour.
- Diff the policy after the upgrade, not just before. A firmware release can alter effective policy without altering the rule base. Comparing intended state against running state afterwards is the only way to see it, which is the same discipline as policy drift detection.
- Give the vendor's schedule an owner. Someone has to hold the release calendar, the end-of-support dates, and the known-issue lists per platform. In a multi-vendor estate that is a real job, and it is usually nobody's.
The tension with faster patching
None of this argues for slower updates. Regulators have moved the other way, and the compression of remediation deadlines we covered in the three-day patch window is not going to reverse. The point is that speed and staging are being traded against each other without anyone writing the trade down.
An organisation that patches within three days and stages nothing has chosen firmware risk over exploitation risk. That may well be the correct choice. It should be a decision with a name on it and a compensating control behind it, not an accident of a deadline. The same logic applies to rule hygiene: a recertification cycle that runs annually while firmware turns over quarterly is measuring the stable half of the system.
Why it matters
The comfortable reading of this data is that change management is a solved problem and attention should move elsewhere. The accurate reading is narrower. Change management is working on the changes it can see, and the share of production change that it can see is falling. Uptime's own framing, that risk is moving deeper into the stack toward faults tied to software behaviour and updates, describes a boundary problem rather than a discipline problem.
Every control described here already exists in most firewall change processes. They are applied to rule changes and not to platform changes, for the historical reason that platform changes used to be rare. They are not rare now. Extending the same record, the same staging, and the same verification to vendor code is a scope change, not a new methodology, and 87% of the organisations in the survey believe better process would have prevented the outage they actually had.
Want to know whether your change records would survive an assessor asking which firmware version a rule ran on? The free NIS2 Readiness Check walks through the evidence a firewall change process has to produce.
