A validator without monitoring is a pager that never rings until the damage is done. Most validator incidents do not start with an explosion - they start with a slow disk filling up, a peer count quietly dropping to zero, or attestations slipping for an hour while nobody watches. Grafana plus Prometheus turns those quiet failures into visible ones: a dashboard you can read in ten seconds, and alerts that reach a human before users, delegators, or the chain notice. This guide shows you the setup that matters, not a dashboard full of charts nobody reads.
The answer in one paragraph: run Prometheus to scrape metrics from your node and its host, point Grafana at Prometheus as a data source, and build one dashboard around four numbers - chain-tip lag, peer count, disk free, and attestation or block performance - with alerts on each. Prometheus listens on port 9090 and Grafana on port 3000 by default, and most chain clients expose a metrics endpoint you enable in their config. Keep the dashboard small: if you cannot tell in ten seconds whether the node is healthy, the dashboard has failed, no matter how pretty it is.
The 4 numbers that matter
First, chain-tip lag: how far your node's latest block is behind the network's latest block. A lag that grows is a sync problem; a lag stuck at zero while the network moves is a peer problem. Second, peer count: a healthy node holds steady peers, and a count collapsing toward zero usually precedes a stall. Third, disk free and disk growth rate: chain data grows every day, and a full disk is the most embarrassing validator outage because it is entirely predictable. Fourth, validator performance: missed attestations or missed blocks, which on most chains map directly to lost rewards and, if sustained, penalties. Everything else - CPU graphs, memory curves, network throughput - is supporting detail for diagnosing one of these four, not a replacement for them.
Prometheus + Grafana with Docker
The simplest reliable stack is Docker Compose: one container for Prometheus, one for Grafana, one for node_exporter on each host you monitor. Enable the metrics endpoint on your chain client first - the flag and port are chain-specific, so take them from the chain's official docs, not from a generic tutorial. Then point Prometheus at three scrape targets: the client's metrics endpoint, node_exporter (which exposes host CPU, memory, and disk on port 9100), and Prometheus itself. In Grafana, add Prometheus as a data source with the URL of your Prometheus server (http://localhost:9090 if they share a host or network), test the connection, and import or build a dashboard. Many chains publish a community dashboard you can import as a starting point - use it, but delete the panels you do not understand, because an unread panel is noise that hides the four numbers above.
# prometheus.yml - scrape targets (adapt ports to your chain's docs)
scrape_configs:
- job_name: "node"
static_configs:
- targets: ["localhost:9100"]
- job_name: "chain-client"
static_configs:
- targets: ["localhost:YOUR_CLIENT_METRICS_PORT"]Dashboards and alerts
Build the dashboard around the four numbers, one panel each, plus a small row of host metrics (CPU, memory, disk) underneath. Then add alerts, because a dashboard nobody watches is decoration. Alert on: chain-tip lag above your threshold for more than a few minutes, peer count at zero, disk free below a threshold with enough days of growth left to act, and missed attestations over a window. Route alerts somewhere a human actually reads - Telegram, Slack, email, or a paging service - and test the route by triggering a test alert. An alert channel you have never seen fire is not an alert channel, it is a hope. Keep alert thresholds tight enough to catch real faults and loose enough that you do not train yourself to ignore them; alert fatigue is how a real outage gets missed.
What to do when an alert fires
Every alert should map to a first action you can take without thinking. Chain-tip lag growing: check peers and logs, restart the client if it is stuck, and check disk before anything else. Peer count zero: check network, firewall, and whether the client process is actually running. Disk full: prune or expand following your chain's docs - never delete chain data blindly, and never delete a slashing-protection database. Missed attestations: check sync, check the validator process separately from the node process, and check the clock - clock drift breaks attestations silently on many chains. Write these first actions into a runbook next to the dashboard, and rehearse them on testnet. The goal is not a perfect dashboard; it is a ten-second read and a known first move at 3am.
When to hire help
Monitoring is the part of node operations most teams postpone and most regret postponing. DevOpsBY runs validators and nodes with Prometheus and Grafana monitoring, alerting, and runbooks as standard, so an incident pages a human who already knows the first move. If your validator is running but unwatched, start with a free infra audit at /contact - bring your chain and your current monitoring setup, and we will check the gaps before an outage finds them.
CTA: Hire DevOpsBY for validator monitoring and node operations, or start with a free infra audit at /contact. Tell us your chain and what you monitor today, and we will scope the right setup.
Disclaimer: This post is educational only, not financial advice. Port numbers and metric names vary by chain client - always confirm the metrics endpoint and flags in your chain's official docs.