Fail Over or Wait It Out: Deciding During a Vendor Outage

A dependency is failing, you built a secondary path for exactly this, and somebody has to decide whether to use it. The decision arrives with no information about how long the outage will last, which is the one input that would settle it.

Both options are expensive when wrong. Failing over into an outage that resolves four minutes later means you paid the full switching cost, including whatever reconciliation the cutover implies, for nothing. Waiting out an outage that runs six hours means you had a working alternative and chose not to use it while customers were failing.

There is no formula that resolves this in the moment, because the missing input is unknowable while the incident is live. What works is deciding most of it in advance, when nobody is under pressure, and leaving on-call with a short judgement instead of an open-ended one. This article covers the four inputs, the cases where failing over reliably makes things worse, and how to write criteria that hold up at three in the morning.

Every version of this decision reduces to four quantities. Three are knowable before the incident and should already be written down. Only the fourth is genuinely unknown at decision time, which is useful to notice, because it means most of the analysis can be done calmly in advance.

You cannot know the remaining duration, and you can bound it usefully. Two signals help more than anything the vendor publishes.

The first is the vendor's incident phase. A provider still investigating has not identified the cause, and an incident that has not been diagnosed is unlikely to be resolved within minutes. A provider that has moved to identified or monitoring has a much shorter expected remaining time. This is the single most informative field on a status page and it is routinely ignored in favour of the severity badge.

The second is that vendor's own history. A provider whose incidents typically run forty minutes is a different proposition from one whose median is four hours, and you only have that comparison if somebody has been keeping the record. This is the most practical argument for retaining vendor incident history rather than reading a status page live: during an incident it converts an unbounded unknown into a distribution.

The useful rule is to set a threshold before the incident and act on it. If the outage is still unresolved at N minutes, fail over. Choosing N in advance turns an open-ended judgement into a timer, and a timer is something a tired on-call engineer can execute correctly. Cost per minute of downtime, which sets N, is covered in calculating third-party downtime cost.

Four situations turn failover from a mitigation into a second incident, and all four are recognisable in advance.

Untested paths. A secondary that has never carried production traffic is a hypothesis, and an incident is a poor place to test one. The failure mode is specific and common: the secondary works for the first few requests, then hits a quota, a missing credential, or a cold cache under real load, and now two paths are broken instead of one.

Partial vendor degradation. Failover logic usually assumes the primary is down. A primary that is slow, or failing 30% of requests, can leave both paths live simultaneously, which is how you get duplicate writes and split state. Degradation is more common than hard failure and is the case failover designs handle worst.

Stateful cutover with no reconciliation plan. Switching a stateless read path is cheap. Switching anything that accumulates state means the two sides diverge from the moment you cut, and the return trip has to merge them. If nobody has written down how that merge works, failing over is a decision to create a data problem later in exchange for an availability improvement now, which is sometimes correct and should be made knowingly.

Correlated secondaries. A backup that shares a cloud region, a DNS provider, or an identity provider with the primary may already be affected by the same event. Verify the secondary is actually healthy before cutting, not merely that it is configured. Why this happens more often than expected is covered in cloud concentration risk.

Write failover criteria per dependency, and keep each one short enough to read during an incident. A page per critical vendor is the right size; anything longer will not be read when it matters.

Each should state the trigger condition including a duration threshold, who can authorise the switch, what verification proves the secondary is healthy, what capability is degraded while failed over, and what the return trip requires. The authorisation line matters more than it looks: an on-call engineer who is not sure they are allowed to make the call will wait, and that hesitation is frequently longer than the outage.

Then exercise it. An untested failover path is worth roughly what an untested backup is worth, which is nothing until proven otherwise. Quarterly is a reasonable cadence, and a game day that deliberately triggers the path under load will teach you more about the cost input than any amount of design review. The general practice is covered in circuit breakers for third-party APIs and the wider response process in the third-party outage runbook.

After the incident, record what you chose and what it cost, whichever way you went. The value of that record is that it corrects the threshold. Teams that never review these decisions keep the same N for years, usually one that was guessed, and a threshold guessed once and never tested is not a policy.

How long should I wait before failing over? Set the threshold in advance from your cost per minute of downtime and your measured cost of switching. Most teams land between 10 and 30 minutes for a critical path. Choosing the number during the incident is what produces inconsistent calls.

How can I estimate how long a vendor outage will last? Read the incident phase, since investigating implies a longer remaining time than identified or monitoring, and compare against that vendor's own incident history. Neither gives certainty, and together they turn an unbounded unknown into a usable range.

Is failing over always safer than waiting? No. An untested path, a partially degraded primary, a stateful cutover with no reconciliation plan, or a secondary that shares infrastructure with the primary can all make the situation worse. Failover has its own failure rate and it belongs in the calculation.

Should failover be automatic? For stateless read paths with a well-tested secondary, often yes. For anything stateful, or where the secondary is expensive or partially capable, a human decision with a written threshold is safer, because automation cannot weigh the reconciliation cost.

What if we fail over and the vendor recovers immediately? That is a normal outcome and not a mistake, provided the threshold was reasonable. Record the cost and use it to tune the number. Judging a decision by an outcome that was unknowable at the time teaches on-call to hesitate, which is more expensive than the occasional early cutover.