Going multi-region: latency, failover and the data problem

Running in several cloud regions sounds like the obvious next step in reliability: if one region goes down, serve from another. In practice it is one of the most expensive things an engineering organisation can take on. It doubles the infrastructure, puts the speed of light into every design decision, and introduces failure modes, such as split brain and lost writes, that a single region never had.

This article covers when multi-region is worth it, what physics costs you, how to reason about data loss and recovery time, the main topologies from active-passive to global consensus, and what it takes for a failover to work on the day you need it.

Why go multi-region, and why not

There are three good reasons to run in more than one region:

• Surviving a region-wide outage. They are rare but real. AWS’s us-east-1 has had several, including the S3 outage of February 2017 and the network event of December 2021, each of which took down services that ran only there.
• Latency for users far away. A user in Sydney talking to a server in Virginia pays roughly 200 ms per round trip before any work happens, and a page load needs many round trips.
• Data residency. Regulation or customers may require that some data stay in a particular jurisdiction.

Before taking it on, look at what has actually caused your outages. For most teams it is bad deploys, configuration changes, expired certificates, overloaded dependencies and database mistakes. Multi-region helps with none of these: a bad deploy goes to every region. Running across several availability zones within one region, which are separate data centres a few milliseconds apart, protects against most infrastructure failures at a fraction of the cost and complexity. Get multi-zone, progressive deploys and fast rollback right first.

The speed of light

Light in optical fibre travels at about 200,000 km/s, two-thirds of its speed in a vacuum. That is 1 ms of round trip for every 100 km of fibre, and real cable routes are longer than the great-circle distance. Typical round-trip times between cloud regions:

No amount of engineering removes these numbers. They mean that any design which waits for another region on every write adds tens of milliseconds to every write, forever. Abadi’s PACELC formulation (2012) captures the trade-off: even when there is no network partition, a replicated system chooses between latency and consistency. Across regions, that choice is expensive both ways.

RPO and RTO

Two numbers describe what a region failure costs:

• Recovery point objective (RPO): how much recent data you can lose. With asynchronous replication it is the replication lag at the moment of failure; with synchronous replication it is zero.
• Recovery time objective (RTO): how long until service is restored. It is the sum of detecting the failure, deciding to fail over, promoting the other region, and moving traffic to it.

Set both per kind of data, not for the whole system. Losing the last few seconds of analytics events is fine. Losing the last few seconds of payments is not. Many systems end up with a mix: a small set of critical tables replicated synchronously or through consensus, and everything else asynchronously.

Active-passive and failover

The simplest topology runs the primary database in one region and streams changes asynchronously to a replica in another. The application may run in both regions, but writes go to the primary. If the primary region fails, you promote the replica and move traffic to it.

The figure shows the two costs. Two subtler problems sit behind them:

• Who decides? From region B, a dead region A and a broken network link between them look the same. If B promotes itself while A is still alive and accepting writes, you have two primaries: split brain. The safe patterns are a third region acting as a witness so the decision has a quorum (two of three regions must agree), or a human who makes the call. In either case, the old primary must be fenced, stopped from accepting writes, before the new one starts.
• What happens when A comes back? Its log contains writes that B never received. They can’t simply be replayed on top of B’s history, because B has accepted conflicting writes since. They must be found, reconciled by hand or by business rules, and the old primary rebuilt from B.

GitHub’s October 2018 incident is the standard case study. A 43-second loss of connectivity between its US East network hub and its primary data centre led its failover tooling to promote MySQL primaries on the West Coast. The East Coast databases held a few seconds of writes that hadn’t replicated, the West Coast had then taken new writes, and application latency to the cross-country databases was too high. Reconciling the two histories and restoring service took just over 24 hours.

Active-active

Running stateless services in every region is easy: put them behind a global load balancer or latency-based DNS. The hard part is always the data. There are three broad ways for more than one region to accept writes:

• Partition by home region. Each user, tenant or account is mastered in one region, usually the one nearest to them, and writes for it go there. Most writes are local, and there are no conflicts, because each record has exactly one writer region. The costs are operations that span homes, such as a transfer between users in different regions, and users who move or travel. This is the most common design for large consumer products.
• Multi-leader with conflict resolution. Every region accepts writes to any record and replicates asynchronously to the others. Concurrent writes to the same record conflict and must be merged. Last-writer-wins, the default in many systems including DynamoDB global tables, silently discards one of the writes. Data types designed to merge, such as CRDTs, avoid losing writes but constrain what the data can be.
• Consensus across regions. Each write commits to a majority of replicas spread over three or more regions, using Paxos or Raft (see the consensus article). Spanner and CockroachDB work this way. Losing a region loses no data and needs no manual failover, but every write waits for a cross-region round trip. Placing a majority of replicas near most writers, as CockroachDB’s regional tables do, keeps the common case fast.

Reading your own writes

A common hybrid serves reads from a local replica in every region and sends writes to the home region. Reads are fast, but a user who writes and then immediately reads, perhaps routed to a different region, may not see their own change. That bug looks to the user like the save didn’t work.

The fix is to make the session carry what it has seen. After a write, record the commit’s position in the replication log. A later read can use a replica only if the replica has replayed at least that far; otherwise it goes to the primary. Most reads stay local, and the guarantee holds for each user, which is the guarantee users actually notice.

Capacity and static stability

Failing over a region only works if the remaining regions can absorb its traffic. With two active regions, each must be able to carry the full load, so in normal operation each runs at no more than 50% of its capacity. With three, each must be able to take an extra half, so each runs at no more than about 67%. Teams that run both regions at 70% to save money discover during the failover that they have built a way to overload the surviving region.

AWS’s term for the principle is static stability: a system should keep working through a failure without needing to make changes. Capacity is already provisioned rather than scaled up during the incident, when every other customer is trying to do the same. Failover shouldn’t depend on a control plane that may itself be affected. For example, AWS documents that Route 53’s control plane runs in us-east-1, while its health checks and DNS answers are served globally, so failover should rely on health checks, not on making DNS changes during the outage.

Failover you can trust

A failover path that runs once a year, during a real incident, will fail in a way nobody predicted. Practices that make it reliable:

• Exercise it on a schedule. Fail over in production, on purpose, regularly. Game days find the missing permission and the stale runbook before an outage does.
• Make the failover path the normal path. If every region always serves some traffic, the “backup” region is never cold and its bugs are found the day they are written.
• Audit hidden single-region dependencies. Identity provider, secrets manager, container registry, feature-flag service, CI, payment provider, the monitoring system itself. A second region that can’t start because its images live in the first one is not a second region.
• Decide in advance who presses the button, and with what information. Automatic failover is fast but can act on a false signal, as in GitHub’s case; manual failover is safe but slow. Many teams automate detection and preparation and keep the final decision with a person.

Cross-region read-your-writes

A sketch of session-based read-your-writes for Postgres with asynchronous replicas in each region. Writes go to the session’s home region and record a WAL position; reads anywhere use the local replica once it has replayed past that position, and fall back to the home primary until then.

// Read-your-writes across regions with asynchronous Postgres replicas. After a write,
// the session remembers a log position (LSN) at or after its commit. A read in any
// region uses the local replica only if that replica has replayed at least that far.
import type { Pool } from "pg";

declare const pools: { primary(region: string): Pool; replica(region: string): Pool };
type Session = { home: string; minLsn?: string }; // an LSN such as "16/B374D848"

export async function write(s: Session, sql: string, params: unknown[]) {
  const client = await pools.primary(s.home).connect();
  try {
    await client.query("BEGIN");
    await client.query(sql, params);
    await client.query("COMMIT");
    // Read after COMMIT on the same connection, so the position covers our commit.
    const { rows } = await client.query("SELECT pg_current_wal_lsn()::text AS lsn");
    s.minLsn = rows[0].lsn;
  } finally {
    client.release();
  }
}

export async function read(s: Session, region: string, sql: string, params: unknown[]) {
  const replica = pools.replica(region);
  if (s.minLsn) {
    // Replay only moves forward, so once this is true it stays true for the query below.
    const { rows } = await replica.query("SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS ok", [s.minLsn]);
    if (!rows[0].ok) return (await pools.primary(s.home).query(sql, params)).rows; // pay the cross-region trip
  }
  return (await replica.query(sql, params)).rows;
}

References

1. GitHub (2018). October 21 post-incident analysis. The GitHub Blog. github.blog
2. Corbett, J. C. et al. (2012). Spanner: Google’s Globally-Distributed Database. OSDI. usenix.org
3. Taft, R. et al. (2020). CockroachDB: The Resilient Geo-Distributed SQL Database. SIGMOD. doi:10.1145/3318464.3386134
4. Abadi, D. (2012). Consistency Tradeoffs in Modern Distributed Database System Design. IEEE Computer. doi:10.1109/MC.2012.33
5. Weiss, B. & Furr, M. Static stability using Availability Zones. Amazon Builders’ Library. aws.amazon.com/builders-library
6. AWS. Disaster Recovery of Workloads on AWS: Recovery in the Cloud (whitepaper). docs.aws.amazon.com
7. AWS. Creating Disaster Recovery Mechanisms Using Amazon Route 53. AWS Networking & Content Delivery Blog. aws.amazon.com/blogs
8. Kleppmann, M. (2017). Designing Data-Intensive Applications, ch. 5: Replication. O’Reilly. dataintensive.net