Firewall High Availability: Testing the Failover You Assume Works

Firewall high availability fails more often by failing open than by failing over. An HA pair that has never been tested under real load is an assumption, not a control. The failover you trust is usually the one you have only ever seen succeed during a quiet maintenance window with no traffic on it. Across hundreds of firewall migrations, the single most common resilience gap I find is a healthy-looking HA pair that has never once been forced to fail over with production load and state on the line.

This guide covers why HA pairs silently lose the ability to fail over, the failover test that actually proves resilience, the real trade-off between active/passive and active/active, and the change-management discipline that keeps a pair genuinely redundant rather than redundant on paper.

Why HA Pairs Stop Being Highly Available

An HA pair degrades quietly. Both units stay green, the dashboard says synchronised, and yet a real failover would drop traffic. The failure is almost never the hardware. It is drift and untested assumptions accumulating between the two units over months of changes.

Five causes account for most silent HA failures. Configuration drift between the active and standby unit, where a change was applied to one and not the other. Session-table sync gaps, where the standby never received the state needed to carry existing connections. Asymmetric routing that the standby cannot reproduce. Mismatched licences, software builds, or SFP modules that only matter at the moment of cutover. And preempt settings nobody validated, so the pair flaps or refuses to hand back. Each is invisible until failover, which is exactly when you cannot afford to discover it.

The Failover Test That Actually Proves Resilience

A real failover test forces the active unit out of service under production-like load and measures whether sessions survive, not just whether the standby turns active. A ping during a change window proves almost nothing. The test that matters has four properties: it runs under representative traffic, it is observed end to end, it measures session survival, and it confirms the pair fails back cleanly.

• Under load. Schedule it, but do not drain traffic first. The point is to see what real connections do when the active unit disappears. Long-lived sessions (database, VoIP, file transfer) are where state-sync gaps show.
• Observed end to end. Watch from the application side, not the firewall console. The firewall will report a clean failover even when the application saw a 30-second outage.
• Session survival, not just role change. Measure how many established connections carried over. A standby that goes active but resets every session has failed the test while reporting success.
• Clean failback. Force the original unit back and confirm the pair returns to a known state without a second outage. Untested failback is its own incident.

Active/Passive vs Active/Active

The HA mode you choose changes what you are testing and what you are exposed to. Active/passive is simpler and the right default for most estates. Active/active doubles throughput and complexity, and it hides asymmetric-path bugs until load tips past one unit's capacity.

The active/active trap is capacity planning. Two units each running at 60 percent look healthy until one fails and the survivor is asked to carry 120 percent. It cannot, so failover becomes a brownout. Active/active is only honest when each unit can carry the entire load alone.

Change Management Is What Keeps a Pair Redundant

Redundancy is not a one-time design, it is a property you maintain on every change. The fastest way to break an HA pair is to apply a rule change, a software update, or a certificate rotation to the active unit and forget the standby, or to apply it to both but never confirm they converged. Configuration drift between HA peers is a specific, recurring class of unauthorized-divergence that a healthy change process catches and an ad-hoc one does not.

Every firewall change to an HA pair has three steps, not one: apply, replicate, and verify in sync. The verification is the step teams skip. The same drift-detection discipline that catches unauthorized single-box changes, covered in the policy drift detection guide, applies between the two members of a pair. Emergency changes are the highest-risk moment: under pressure, the standby is the first thing forgotten, which is why the emergency change process must include a post-incident sync check. The broader rhythm sits inside the firewall change management process.

Compliance Expectations

Resilience and continuity are explicit obligations under the frameworks that govern essential and financial entities. The evidence an auditor expects is a documented HA design plus a record of a real, recent failover test, not a vendor datasheet claiming five-nines.

For financial entities, the failover-test record is part of the DORA resilience-testing evidence covered in the DORA firewall compliance guide, and for NIS2 essential entities it sits alongside the documentation set in the NIS2 Article 21 evidence mapping.

Recurring Failure Modes

Four HA failure patterns recur across the migrations reviewed, and all four are caught by a real test before they become an outage.

The never-tested standby. The pair was built correctly at install and never failed over since. Years of drift sit unvalidated. The first real failover is also the first test, in production, during an incident.

The split-brain pair. A heartbeat-link failure leaves both units believing they are active. Dedicated, redundant heartbeat paths and tested preempt logic prevent it. A shared data-and-heartbeat link is the usual cause.

The overcommitted active/active. Both units run comfortably until one drops and the survivor cannot carry the combined load. Capacity must be sized for single-unit operation, not steady-state.

The asymmetric-routing surprise. Traffic enters via one unit and tries to leave via the other, and the stateful inspection drops it. Common after a routing change that was never tested against a failover scenario. This overlaps with the consistency problems of running mixed platforms, covered in the multi-vendor firewall management guide.

Frequently Asked Questions

How often should a firewall failover be tested?

At least annually, and after any significant change to the pair: a software upgrade, a routing change, or a hardware swap. High-consequence estates test semi-annually. The trigger that matters more than the calendar is change: a pair that has been modified since its last test is effectively untested.

Does session synchronisation guarantee zero-downtime failover?

No. Session sync improves survival but does not guarantee it. UDP flows, certain application-layer states, and very new connections may not be in the synchronised table at the moment of failover. The only way to know your real survival rate is to measure it under load.

Is active/active always faster than active/passive?

Only until something fails. Active/active delivers more throughput in steady state, but if each unit is not sized to carry the full load alone, a failure turns the speed advantage into a brownout. For most estates, active/passive with a correctly sized single unit is the more honest design.

What is split-brain and how do I prevent it?

Split-brain is when both HA members believe they are active, usually after a heartbeat-link failure, causing duplicate IPs and traffic disruption. Prevent it with dedicated, redundant heartbeat links separate from the data path, and validated preempt and quorum settings.

Can HA replace backups and configuration version control?

No. HA protects against hardware and link failure. It does not protect against a bad change, which replicates instantly to both units. You still need configuration version control and the ability to roll back, the same way a healthy change process treats every rule edit.

Further Reading

Authoritative external sources:

• NIST SP 800-34 Rev 1, Contingency Planning Guide for Federal Information Systems
• ISO 22301, Business Continuity Management Systems
• ENISA, Cybersecurity Resilience Resources