One RPC endpoint is a single point of failure, and launch week is when it fails. Traffic spikes, a client upgrade goes wrong, a disk fills, or a provider has an outage, and every wallet, app, and indexer that depends on that endpoint stops at once. The fix is not a bigger server. The fix is a second endpoint, a health check that can tell a healthy endpoint from a stale one, and a switching rule that acts before users notice. Failover is therefore not a feature you add after an outage. It is the design that prevents the next one from being an outage at all.
This post covers the simplest reliable pattern: a primary endpoint and a fallback endpoint, both fully synced, with health checks on chain-tip lag and a client or proxy that switches between them. It does not cover multi-region active-active clusters or load balancing across dozens of nodes. Those are later steps, and they build on exactly the health checks described here. If your health check cannot detect a stale endpoint, no amount of extra capacity will help, because traffic will keep flowing to the stale one.
Why one RPC is a single point of failure
RPC failures come in two kinds, and only one is obvious. The obvious kind is a dead endpoint: connection refused, timeout, or a 500 error. Every client notices that, and retry logic usually handles it. The dangerous kind is a stale endpoint: the server answers promptly, returns valid-looking data, and is quietly behind the chain. A wallet shows an old balance, an app misses a new event, and a transaction appears to fail because it was built against old state. Stale is worse than dead, because nothing errors loudly enough to page anyone, and users discover the problem before operators do.
Stale endpoints happen for ordinary reasons. A node falls behind during a client upgrade, loses peers, or hits a disk bottleneck, while its RPC server keeps answering. A node that restarted and is resyncing can also look healthy to a naive check that only asks whether the port is open. That is why a health check must ask about chain position, not just reachability. A port that answers is not evidence. A block number that matches the network head, within an acceptable lag, is evidence.
Primary plus fallback architecture
The architecture is simple on purpose. Run two independent nodes, on separate machines if possible, each fully synced and each exposing RPC only to your proxy, load balancer, or application layer, never to the public internet by default. One is designated primary, the other fallback. Both receive the same monitoring and the same upgrade discipline, because a fallback that is unmaintained is not a fallback. It is a second outage waiting for the first one to happen.
Independence matters more than identical hardware. If both nodes share a host, a disk, or a network path, a single failure takes both down, and the failover diagram is fiction. If both nodes run the same client version and the same configuration, a client bug can take both down at once, which is why larger operators diversify clients across primary and fallback. For a two-node setup, separate machines and separate upgrade timing are the minimum: never upgrade primary and fallback at the same moment, or a bad release removes both endpoints in one command.
Health checks that actually detect staleness
A useful RPC health check asks three questions on a short interval, for example every 10 to 30 seconds. First, does the endpoint answer at all, within a timeout? Second, what block number does it report, using a standard method such as eth_blockNumber on Ethereum-style chains, or the equivalent head query on other chains? Third, is that block number close to the network head, as reported by the other endpoint or by a trusted independent source? The third question is the one that catches staleness. An endpoint that answers, but whose block number stops advancing, is unhealthy, however fast its responses look.
Set the lag threshold as an operator decision, and write it down. An example rule is: fail over if the primary lags the fallback by more than a small number of blocks, or if its block number has not advanced for several minutes, whichever comes first. The exact numbers depend on block time and on how stale your application can tolerate. What matters is that the rule exists before an incident, that it is tested, and that it pages a human when it fires. A failover that switches silently and never alerts is a failover that hides a dying primary until the fallback dies too.
Also check sync status, not just block height, where the client exposes it. A node can report a block number while still syncing, and a consensus client can lag while its execution client looks current, as covered in the earlier sync-check post. The health check should therefore include the syncing endpoint for each client where available, and treat a syncing node as a fallback candidate, not a primary, until it reports fully synced and stays synced. Logs and peer counts belong in the same dashboard, so a health-check failure can be diagnosed in minutes, not by guessing.
Switching, and testing the switch
Switching can happen in the client, in a proxy, or in a load balancer, and each place has tradeoffs. Client-side switching, where the app holds a list of endpoints and tries the next one on failure or staleness, is simple and needs no extra infrastructure, but every client must implement the same logic. A proxy or load balancer centralises the logic, so clients point at one address and the proxy chooses the healthy backend, but the proxy itself becomes a component that needs its own redundancy. Either way, the switching rule must be the same rule the health check uses, or the two will disagree during an incident.
Test the failover, do not just trust it. The drill is short: stop the primary RPC service, confirm traffic moves to the fallback, confirm the alert fires, then restore the primary and confirm traffic returns or stays on the fallback according to your policy. Then test the harder case: make the primary stale, not dead, by pausing its sync or pointing it at old data on a test setup, and confirm the health check detects staleness and switches. A failover that has only been tested for dead endpoints has only been tested for the easy half of failures.
Failover also has a cost, and it should be stated honestly. Two nodes mean twice the server, disk, and monitoring cost of one, plus the time to maintain both. That cost buys availability during the moments when availability matters most: launches, unlocks, and traffic spikes, when a single endpoint failure becomes a public incident. If your application can tolerate downtime, one well-monitored node may be enough. If it cannot, two nodes with a tested health check are the minimum, not a luxury. If you are designing that second endpoint, or if your current failover has never been drilled, blockchain node setup and DevOps node operations cover endpoint architecture, health checks, and failover drills, starting with a free infra audit. Bring your current endpoint list and your health-check rule, if you have one: those two items show immediately whether failover is real or only drawn on a diagram.
Disclaimer: This post is educational only, not financial advice. Figures are listing or project data as of 2026-10-31 and may change - always confirm version numbers, requirements, and dates in the official docs before acting on them.