A validator node talks constantly, and almost everything it says goes unread until something breaks. Then, in the middle of an incident, an operator opens a log file for the first time and faces thousands of lines of unfamiliar output. That is the wrong moment to learn how to read logs. Log reading is a skill you build on a quiet day, so that on a bad day you can find the one line that matters in under a minute. This post teaches that skill: where logs live, the five kinds of lines worth recognising, and a short routine that turns a wall of text into a diagnosis.
Start with a simple mental model. A node has two main jobs, and each produces its own log stream. The execution client follows the chain state and serves data, and the consensus client handles attestations, proposals, and finality. On many setups both run as separate containers or services, so there are at least two logs to check, not one. When a problem appears, the first question is always which client is unhappy, because the fix for a consensus client that stopped attesting is different from the fix for an execution client that stopped syncing. Checking the wrong log first is the most common beginner mistake, and it costs more time than any other.
Where the logs actually live
The location depends on how you run the node, and you should know all three patterns before you need them. If you run with Docker Compose, the command is docker compose logs, with -f to follow live output and --tail to limit how far back you look. A typical incident command looks like this:
docker compose logs -f --tail 200 consensus
# or: docker compose logs --tail 500 executionIf the node runs as a systemd service on Ubuntu, journalctl is the tool: journalctl -u your-node-service -f follows the log live, and adding --since "30 minutes ago" narrows it to the incident window. If the client writes its own log files, find the path in the client flags or config file and use tail -f on that file. In all three cases the discipline is the same: follow live output during an incident, and always narrow the window before you start reading, or you will drown in old lines.
One more setup step pays for itself many times over: make sure logs are kept somewhere they survive a container restart. Docker logs kept only in the default driver disappear when a container is removed, and a client that restarts on crash can erase the very evidence you need. Either configure a log driver that writes to disk, mount a log volume, or ship logs to a central place. The Grafana monitoring post on this blog covers the monitoring side; logs are the detail behind the dashboard, and you need both.
The five lines you must recognise
You do not need to understand every line. You need to recognise five patterns, because nearly every incident shows up as one of them.
1. Sync progress
A healthy node reports its sync position regularly: current slot or block height, distance from the chain head, and peers. The line to watch is the gap between your node and the head. A small, shrinking gap is normal after a restart. A gap that grows, or a height that stops moving entirely, is the first sign of a sync problem. Note the exact height where it stopped, because that tells you whether the node is stuck at a specific block or simply slow.
2. Peer count and peer churn
Peer count is the node's social health. A sudden drop to zero peers usually means a network problem: firewall, port, or connectivity, not a chain problem. Rapid peer churn, where peers connect and drop constantly, often points to a version mismatch or a clock problem. Clock drift is a classic silent killer for validators, because consensus depends on accurate time; if your system clock is off by more than a second or two, peers and duties both start failing in confusing ways.
3. Attestations and proposals
For a validator, the lines that matter most are the ones that say a duty was performed or missed. A missed attestation once in a while is normal noise. Missed attestations in a run, or a missed block proposal, is an incident. Read these lines together with the sync lines: if the node is behind the head, duties will fail as a consequence, and fixing sync fixes the duties. If the node is synced and duties still fail, the problem is usually keys, the validator client, or the connection between the consensus and execution clients.
4. Errors and warnings, ranked by action
Not all errors are equal. Connection refused between your own clients is urgent and usually local: a client is down, a port changed, or a config broke. Disk full or database errors are urgent and get worse fast: stop writing, free space, and do not restart blindly, because a corrupt database after a forced restart is worse than a paused node. TLS or API errors on an RPC endpoint are usually less urgent for a validator, but they matter immediately if that endpoint serves users. The rule is simple: rank errors by what stops working, not by how scary they look.
5. The absence of lines
The most underrated log signal is silence. A node that normally logs every few seconds and suddenly logs nothing has usually crashed, frozen, or lost its log path. Silence is why you pair logs with the alerting setup from the Telegram alerts post: an alert on "no log output for five minutes" catches the failure that produces no error at all.
A 10-minute log-reading routine
When something looks wrong, run this routine in order, and resist the urge to jump straight to restarting:
- Confirm which client is affected: check both logs, and note the last healthy line in each.
- Narrow the window: look at the last 30 minutes, not the last 30,000 lines.
- Find the first bad line, not the last: the first error in a cascade is usually the cause, and everything after it is a symptom.
- Check the three usual suspects: sync gap, peer count, and clock drift, before touching configuration.
- Search, do not scroll: grep for "error", "warn", "missed", and "peer" to jump to signal instead of reading noise.
A useful habit is to keep a small notebook of lines you have seen before, with the fix that worked. Node logs repeat themselves: the same warning before an upgrade, the same peer error after a firewall change. Operators who recognise a line in seconds are not smarter, they have just seen it before and wrote it down.
When logs say stuck versus slow
The distinction matters because the fixes are opposite. A slow node is advancing, just behind the head: give it time, check disk speed and peer quality, and do not restart, because a restart resets sync progress. A stuck node is not advancing at all, or is repeating the same error: that is when you check disk space, check whether the client version matches the network, and check the client project's status page for a known issue. Restarting a stuck node is reasonable; restarting a slow node usually makes things worse. The testnet setup guide on this blog is a safe place to practise this distinction, because a testnet node can be broken and rebuilt without real funds at risk.
From logs to a runbook
The final step is to turn what you learn into a runbook: one page that says, for each of the five patterns above, what it looks like, what it usually means, and what to do first. That page is what turns a 2 a.m. incident into a 10-minute fix instead of a 2-hour investigation. If your team runs nodes for clients, that runbook is also part of your service quality: it is the difference between "we noticed and fixed it" and "a user told us". DevOpsBy builds monitoring and runbooks for node operators as part of its node operations service - and if you want a second pair of eyes on your setup, the free infra audit is the place to start. Bring your last incident's logs: they tell us more about your setup than any architecture diagram.
Disclaimer: This post is educational only, not financial advice. Figures are listing or project data as of 2026-11-03 and may change - always confirm version numbers, requirements, and dates in the official docs before acting on them.