Most validator downtime is not bad luck. It is a short list of repeatable mistakes, made by new operators and experienced ones alike, usually under time pressure, usually right before a launch or an upgrade. The good news is that the list is short, the fixes are known, and every one of them is cheaper to fix before it happens than after. This post walks through the nine mistakes we see most often, what each one costs, and the fix that prevents it.
If the validator role itself is new to you, start with our validator node explainer and our slashing explainer. This post assumes you already run a validator or are about to, and it is about the operational mistakes, not the theory.
1. Keeping the withdrawer key on the server
This is the mistake with the worst downside, so it goes first. A Solana validator has three separate keys, and they do not belong in the same place. The identity key lives on the host, because it has to: it signs gossip and votes. The vote account key is kept separate from the identity key, so one compromise does not take both. And the authorized withdrawer, the key that can change commission and move rewards, is kept 100 percent offline, on a hardware wallet or HSM, and never loaded onto validator hardware. If the withdrawer key sits on the server, a single host compromise lets an attacker raise commission to 100 percent or redirect rewards, and no monitoring alert will stop it in time. The same principle holds on every chain: the keys that can move funds or change validator settings stay cold. The keys that must be hot are the smallest set possible, on a dedicated unprivileged user, with file permissions locked down. Our validator key security guide covers the separation in detail, and our Cosmos setup guide shows where the same separation applies on Cosmos chains.
2. Upgrading without a snapshot
Upgrades cause most avoidable validator downtime, and the pattern is always the same: a new client version lands, the operator upgrades on announcement day, the new version breaks sync or changes a flag, and there is no way back because nobody took a snapshot first. The fix is a sequence, not a feeling: snapshot before you touch anything, test the new version on testnet or a staging node, upgrade, verify sync and voting, and keep the rollback plan written down before you start. Never upgrade on announcement day if you can avoid it, and never upgrade without a tested restore. Our upgrade guide walks through the full sequence, and our backup guide covers what to back up and how to test the restore.
3. Going to mainnet without a testnet run
Mainnet is the wrong place to learn how your setup behaves. A testnet run costs a few hours and finds the mistakes that mainnet punishes: the firewall rule that blocks peers, the disk that fills in three days, the key file with the wrong permissions, the monitoring alert that never fires. Run the full setup on testnet first, including the upgrade path and the restore path, and only move to mainnet when the testnet node has run clean for days. Our testnet setup guide covers the setup, and our testnet to mainnet migration guide covers the move itself.
4. Ignoring disk growth until the node stops
Chain data grows every day, and a full disk stops sync silently. The node does not crash loudly, it just stops making progress, and the first sign is usually missed votes or a stale RPC. The fix is boring and it works: alert on disk usage at 80 percent, not 99 percent, track the daily growth rate so you can forecast the next disk purchase, and plan pruning or a larger disk before you need it. Check free space before anything else when a node falls behind, because it is the cheapest thing to rule out. Our Grafana monitoring guide shows how to turn disk growth into an alert that pages a human, and our snapshot sync guide covers the faster recovery when a node is far behind.
5. One node and no plan B
A single node is a single point of failure, and every part of it fails eventually: the disk, the network, the host, the operator attention at 3am. The fix is redundancy at the level that matters for your setup: a second node that can take over, a fallback RPC endpoint that is actually tested, and a failover drill that has been run, not just written. An untested fallback is not a fallback, it is a hope. Our redundancy guide covers adding a second node, and our RPC failover guide covers two synced endpoints with health checks and automatic switching.
6. Copying config values from a blog post
Entrypoints, bootnodes, known-validator lists, chain IDs, and peer addresses change, and a value copied from a months-old guide can point your node at a dead peer or the wrong network. This applies to this blog too: treat any config value here as a starting point, then verify it against the chain official docs on the day you use it. The same goes for commands and flags, which change between client releases. A node configured from stale values can look healthy while syncing the wrong chain or talking to nobody, and the sync check is the only thing that catches it. Our sync check guide shows how to prove the node is at the tip, not just running.
7. Treating monitoring as optional
A node without monitoring is a node you find out about from your delegators, and by then the damage is done. The minimum is four alerts: sync gap against the chain tip, disk usage, peer count, and process health. Alert on the sync gap, not on a single RPC call, because a single call can report healthy while the node is stuck. Record the history too, so when a node falls behind you can see whether it has been drifting for an hour or just had one slow minute. Monitoring is not a nice extra for later, it is part of the setup, and a node is not finished until the alerts are tested. Our Grafana guide covers the dashboards and the alerts, and our Telegram alerts guide covers getting the alert to a human fast.
8. Setting commission caps carelessly at creation
On Cosmos chains, the commission max rate and the max change rate are fixed at validator creation and can never be changed afterwards. Set the max rate too low and you are trapped when costs rise. Set the max change rate too high and a mistake, or a compromised key, can jump the rate in one move. New validators rush this step because it looks like paperwork, and then live with it forever. Read the creation step carefully, choose caps you can live with for years, and remember the pace limit: at most one commission change per 24 hours. Our commission change guide covers the rules on Cosmos and Solana in full.
9. No runbook for the 3am page
The incident is the wrong time to decide what to do. A runbook is a short, written page for each failure you expect: node out of sync, disk full, peers gone, client upgrade failed, key compromise suspected. Each page says what to check first, what not to do, and who to call. Write it when you are calm, test it when you are calm, and keep it where the on-call person can find it at 3am without asking anyone. The runbook is also the fastest way to hand the node to someone else: if only one person knows how the node works, the node has a single point of failure with a name on it.
None of these mistakes is a sign of a bad operator. They are the normal mistakes of a busy one, and the fix for each is a habit, not a purchase: cold keys, snapshots before upgrades, testnet first, disk alerts, a second node, verified configs, real monitoring, careful caps, and a written runbook. If you want those habits built into your node, RPC, or validator setup instead of bolted on after the first incident, book a free infra audit with DevOpsBY. We run nodes, RPC, and validator infrastructure for Web3 teams, and the first thing we check is this list.
Disclaimer: This post is educational only, not financial advice. Figures are listing or project data as of 2026-11-12 and may change - always confirm version numbers, requirements, and dates in the official docs before acting on them.
