Visualizing Oxidized Backup Status in Grafana and Prometheus

Quick answer: Oxidized does not ship a native Prometheus exporter, so the practical approach is to read node status from the Oxidized REST API (the /nodes.json endpoint provided by oxidized-web), convert it into Prometheus metrics with a small exporter, scrape that exporter with Prometheus, and build Grafana panels and alert rules on top. The result is a live view of which devices backed up successfully, which failed, and which have gone stale.

Configuration backups only protect you if they are actually running. Many teams discover during an outage that a device has not been backed up for months because a password changed or a model definition broke. This guide shows how to make backup health visible and alertable so that silent failure stops being possible. As with any tool that evolves, verify API fields against your installed Oxidized version before relying on them. I flag the places where behavior varies.

Why Backup Status Needs Its Own Monitoring

Oxidized logs failures, and it exposes node status through its web interface. That is enough when you manage ten devices. It stops being enough when you manage hundreds, for three reasons:

1. Failures are quiet by default. A node that cannot be reached produces a log line and a red entry in a list that nobody is looking at.
2. There is no trend view. You cannot tell whether last week’s failure rate was normal or whether it is climbing.
3. There is no escalation path. Without alerting, the information depends on someone remembering to check.

Prometheus and Grafana address all three. Prometheus stores the status as time series data and evaluates alert rules. Grafana turns the same data into dashboards that operations staff and managers can read at a glance. Together they move backup health from something you look up to something that tells you when it is wrong.

What “Healthy” Means for a Backup System

Before building anything, decide what you want to measure. A backup system is healthy when:

1. Every device in the inventory has been backed up successfully within an agreed window, for example two polling intervals.
2. The proportion of failed nodes stays below an agreed threshold.
3. Oxidized itself is running and its API is answering.
4. Devices that should be in the inventory actually are.

Notice that the first item is about freshness, not just the result of the last attempt. A node that failed once and then succeeded is healthy. A node that has not been attempted in two days is not, even if its last recorded status was a success.

Key Concepts

Term	Meaning in this context
Oxidized	Open source network configuration backup tool written in Ruby
oxidized-web	Web interface and REST API for Oxidized
Exporter	A service that exposes metrics in Prometheus format on an HTTP endpoint
Scrape	Prometheus pulling metrics from an endpoint at a fixed interval
Gauge	A metric type whose value can go up and down
Label	A key and value pair that identifies a time series dimension
PromQL	The Prometheus query language
Alertmanager	The component that routes, groups, and silences Prometheus alerts
Cardinality	The number of unique time series, driven by label combinations

Pull Versus Push Models

Prometheus is a pull-based system. It contacts targets on a schedule and reads their metrics. This matters because many people reach for the Pushgateway when they think about “sending Oxidized events to Prometheus”. The Prometheus documentation positions the Pushgateway for service-level batch jobs and warns that it has drawbacks, such as becoming a single point of failure and keeping stale metrics until they are deleted. For node-level backup status, a pull-based exporter is the better fit, because the data already lives in a queryable API.

Architecture Overview

The design has five parts:

1. Oxidized with oxidized-web polls devices and exposes node status over REST.
2. A custom exporter reads the REST API on each scrape and converts the response into Prometheus metrics.
3. Prometheus scrapes the exporter, stores the series, and evaluates alert and recording rules.
4. Alertmanager deduplicates and routes notifications to chat, email, or paging systems.
5. Grafana queries Prometheus and renders dashboards.

Component Roles

Component	Role	Where it runs
Oxidized and oxidized-web	Source of truth for node status	Backup server
Exporter	Translates API output to metrics	Same host as Oxidized, or a nearby container
Prometheus	Time series storage and alert evaluation	Monitoring server
Alertmanager	Notification routing	Monitoring server
Grafana	Visualization	Monitoring server

Design Decision: Textfile Collector or Custom Exporter?

You have two reasonable ways to expose the metrics, and the choice affects how you maintain the solution.

Aspect	Textfile collector (node_exporter)	Custom HTTP exporter
Extra service to run	No, uses existing node_exporter	Yes
Freshness	Depends on cron or timer interval	Fresh on every scrape
Stale series handling	Manual cleanup of files	Series disappear naturally when nodes are removed
Complexity	Low	Low to medium
Best for	Small setups already running node_exporter	Most production setups

I prefer the custom exporter for this job. It evaluates the API at scrape time, so Prometheus always sees the current state, and when you remove a device from Oxidized its series disappear from the next scrape instead of lingering. The textfile approach is covered later as an alternative, because it is a good option if you already depend on node_exporter and do not want another service.

Prerequisites

You need the following in place:

1. A working Oxidized installation that backs up at least a few devices.
2. oxidized-web enabled, so the REST API answers.
3. Prometheus and Grafana installed, or access to run them in containers.
4. Python 3 on the host that will run the exporter.
5. Network access from Prometheus to the exporter, and from the exporter to the Oxidized API.

Enable the Oxidized REST API

The REST API is enabled with the rest setting in the Oxidized configuration. Bind it to localhost unless you have a reason to do otherwise.

This snippet enables the web interface and API on the loopback address, which is the safest default when the exporter runs on the same host.

yaml

rest: 127.0.0.1:8888

After restarting Oxidized, confirm the API is answering. The jq tool makes the output readable.

bash

curl -s http://127.0.0.1:8888/nodes.json | jq '.[0]'

Expect a JSON object describing the first node, including its name, IP, model, group, and a last object with the most recent job result. Field names and formats can vary between releases, so read the actual output on your system before you write any parsing logic. If the command returns nothing, check that oxidized-web is installed and that the service is bound to the address you expect.

Understanding the Oxidized Node Data

The /nodes.json endpoint returns an array with one object per node. The fields that matter for monitoring are the node name, group, model, and the details of the last job, including its start and end time and its status.

Field	Why it is useful
name	Primary identifier for each device
group	Lets you break down status by site or device class
model	Helps spot model-specific problems, such as a changed prompt
last status	Success or the type of failure
last end time	Basis for freshness calculations

Status Values

A successful job is reported with a status of success. Failures carry other status values that describe what went wrong, such as a connection problem or a timeout. The precise set depends on the version, so list what your installation actually returns before deciding how to map it.

This command lists the distinct statuses currently reported, which is a quick way to see what your own system produces.

bash

curl -s http://127.0.0.1:8888/nodes.json | jq -r '.[].last.status' | sort | uniq -c

The output shows each status with a count. Anything that is not success is a failure for the purposes of this article. Treating every non-success value as a failure keeps the exporter simple and robust against new status strings.

Designing the Metrics

Good metric design pays off later when you write queries and alerts. Follow Prometheus naming conventions: lowercase names with underscores, a unit suffix such as _seconds, and timestamps expressed as Unix time in seconds.

Metric Set

Metric	Type	Labels	Meaning
oxidized_up	Gauge	none	1 if the exporter could read the API, otherwise 0
oxidized_node_last_run_success	Gauge	node, group, model	1 if the last job succeeded, 0 otherwise
oxidized_node_last_run_timestamp_seconds	Gauge	node, group, model	Unix time when the last job ended
oxidized_nodes	Gauge	none	Number of nodes returned by the API

Why Not Put the Status in a Label?

A common beginner mistake is to expose a metric whose value is always 1 and carry the status as a label, such as status="timeout". That creates a new series every time the status changes, which inflates cardinality and makes queries awkward. A 0 or 1 value for success, plus a timestamp for freshness, answers the important questions with far fewer series. If you want the failure reason, expose it through a separate info-style metric and keep it out of your alert rules.

Cardinality Check

Each node produces two series with three labels each. A thousand devices therefore produce about two thousand series from this exporter, which is small by Prometheus standards. Avoid adding the device IP address or the full configuration text as labels. The IP adds little over the node name, and anything unbounded or frequently changing hurts performance.

Building the Exporter

The exporter below is a custom collector. Each time Prometheus scrapes it, the collector calls the Oxidized API and yields fresh metrics. If the API is unavailable, it reports oxidized_up as 0 and emits nothing else, which prevents stale node data from looking healthy.

Install the Dependencies

This installs the Python libraries the exporter needs inside a virtual environment, which keeps them separate from system packages.

bash

sudo mkdir -p /opt/oxidized-exporter sudo python3 -m venv /opt/oxidized-exporter/venv sudo /opt/oxidized-exporter/venv/bin/pip install prometheus_client requests

The Exporter Code

Save the following as /opt/oxidized-exporter/exporter.py. It reads its settings from environment variables, so you can change the API address or port without editing the code.

python

import logging import os import time from datetime import datetime, timezone import requests from prometheus_client import start_http_server from prometheus_client.core import GaugeMetricFamily, REGISTRY API_URL = os.environ.get("OXIDIZED_URL", "http://127.0.0.1:8888/nodes.json") LISTEN_PORT = int(os.environ.get("EXPORTER_PORT", "9899")) TIMEOUT = float(os.environ.get("OXIDIZED_TIMEOUT", "10")) logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") def parse_time(value): """Convert an Oxidized timestamp string to Unix seconds, or None if unparseable.""" if not value: return None text = str(value).strip() for fmt in ("%Y-%m-%d %H:%M:%S %Z", "%Y-%m-%d %H:%M:%S"): try: parsed = datetime.strptime(text, fmt) return parsed.replace(tzinfo=timezone.utc).timestamp() except ValueError: continue return None class OxidizedCollector: def collect(self): up = GaugeMetricFamily("oxidized_up", "1 if the Oxidized API was readable") try: response = requests.get(API_URL, timeout=TIMEOUT) response.raise_for_status() nodes = response.json() except Exception as exc: logging.error("Could not read Oxidized API: %s", exc) up.add_metric([], 0) yield up return up.add_metric([], 1) yield up total = GaugeMetricFamily("oxidized_nodes", "Number of nodes known to Oxidized") success = GaugeMetricFamily( "oxidized_node_last_run_success", "1 if the last backup job for the node succeeded", labels=["node", "group", "model"], ) last_run = GaugeMetricFamily( "oxidized_node_last_run_timestamp_seconds", "Unix time when the last backup job for the node ended", labels=["node", "group", "model"], ) total.add_metric([], len(nodes)) for node in nodes: labels = [ str(node.get("name", "unknown")), str(node.get("group") or "none"), str(node.get("model") or "unknown"), ] last = node.get("last") or {} success.add_metric(labels, 1 if last.get("status") == "success" else 0) ended = parse_time(last.get("end")) if ended is not None: last_run.add_metric(labels, ended) yield total yield success yield last_run if __name__ == "__main__": REGISTRY.register(OxidizedCollector()) start_http_server(LISTEN_PORT) logging.info("Oxidized exporter listening on port %s", LISTEN_PORT) while True: time.sleep(3600)

Three points about this code are worth explaining. First, the parse_time function assumes timestamps are in UTC and tries two common formats. Confirm the format on your system with the earlier curl command, and adjust the format strings if your output differs. Second, a node with no recorded end time simply has no timestamp series, which your freshness alert will treat as missing data rather than as a false value. Third, the exporter has no state of its own, so you can restart it at any time without losing history. The history lives in Prometheus.

Run It as a systemd Service

This unit file runs the exporter under its own unprivileged user and restarts it if it fails.

ini

[Unit] Description=Oxidized Prometheus exporter After=network-online.target oxidized.service Wants=network-online.target [Service] User=oxidized-exporter Group=oxidized-exporter Environment=OXIDIZED_URL=http://127.0.0.1:8888/nodes.json Environment=EXPORTER_PORT=9899 ExecStart=/opt/oxidized-exporter/venv/bin/python /opt/oxidized-exporter/exporter.py Restart=on-failure RestartSec=5 NoNewPrivileges=true ProtectSystem=strict ProtectHome=true PrivateTmp=true [Install] WantedBy=multi-user.target

Create the service account, enable the unit, and check the output.

bash

sudo useradd --system --no-create-home --shell /usr/sbin/nologin oxidized-exporter sudo systemctl daemon-reload sudo systemctl enable --now oxidized-exporter curl -s http://127.0.0.1:9899/metrics | grep '^oxidized_'

You should see oxidized_up 1.0 and one pair of node metrics per device. If oxidized_up is 0, the exporter cannot reach the API, and its log (available through journalctl -u oxidized-exporter) tells you why. A common pitfall is an Oxidized service bound to a different address or port than the one in the unit file.

Configure Prometheus

Prometheus needs a scrape job for the exporter, a place to find rules, and a link to Alertmanager.

Scrape Configuration

This block adds the exporter as a scrape target with a one-minute interval. Backup jobs run on an hourly cadence by default, so a minute is more than frequent enough and keeps load negligible.

yaml

global: scrape_interval: 60s evaluation_interval: 60s rule_files: - /etc/prometheus/rules/oxidized.yml alerting: alertmanagers: - static_configs: - targets: ["127.0.0.1:9093"] scrape_configs: - job_name: oxidized static_configs: - targets: ["127.0.0.1:9899"] labels: service: network-backup

Reload Prometheus after editing, and confirm the target is healthy in the Status, Targets page of the Prometheus web interface. A target that shows as down usually indicates a firewall, a wrong port, or an exporter that is not running.

Validate Before You Reload

Prometheus ships with promtool, which checks configuration and rule files for syntax errors. Running it before every reload prevents a typo from taking down your monitoring.

bash

promtool check config /etc/prometheus/prometheus.yml promtool check rules /etc/prometheus/rules/oxidized.yml

Each command reports success or points to the line with the problem.

Writing Alert Rules

Alerts are where this project pays for itself. Design them around the definition of “healthy” you agreed earlier.

Alert Design Principles

1. Alert on symptoms that need human action, not on every blip.
2. Use a for duration so that transient failures do not page anyone.
3. Align thresholds with your polling interval. With an hourly interval, a single failed run is normal noise, and two or three in a row is a signal.
4. Make alerts specific. Include the node name and group so that the responder knows where to look.

The Rule File

Save the following as /etc/prometheus/rules/oxidized.yml. It contains five rules covering the exporter, the API, individual nodes, stale backups, and fleet-wide failure rate.

yaml

groups: - name: oxidized rules: - alert: OxidizedExporterDown expr: up{job="oxidized"} == 0 for: 10m labels: severity: warning annotations: summary: "Oxidized exporter is not being scraped" description: "Prometheus cannot scrape the Oxidized exporter. Backup visibility is lost." - alert: OxidizedApiUnavailable expr: oxidized_up == 0 for: 10m labels: severity: critical annotations: summary: "Oxidized API is not answering" description: "The exporter cannot read the Oxidized API. Backups may not be running." - alert: OxidizedNodeBackupFailing expr: oxidized_node_last_run_success == 0 for: 2h labels: severity: warning annotations: summary: "Backup failing for {{ $labels.node }}" description: "Node {{ $labels.node }} in group {{ $labels.group }} (model {{ $labels.model }}) has failed its backup for over two hours." - alert: OxidizedNodeBackupStale expr: time() - oxidized_node_last_run_timestamp_seconds > 21600 for: 15m labels: severity: warning annotations: summary: "No recent backup attempt for {{ $labels.node }}" description: "The last backup job for {{ $labels.node }} ended more than six hours ago." - alert: OxidizedHighFailureRate expr: count(oxidized_node_last_run_success == 0) / count(oxidized_node_last_run_success) > 0.2 for: 30m labels: severity: critical annotations: summary: "More than 20 percent of nodes are failing backups" description: "A large share of devices cannot be backed up, which often points to a shared cause such as a changed credential or an AAA outage."

The thresholds here are starting points. With an hourly polling interval, a two hour for clause on the failing node rule lets one retry cycle pass before notifying anyone, and the six hour stale threshold allows several missed cycles. Tune both to match your own interval and tolerance.

Why the Fleet-Wide Rule Matters

When one device fails, the cause is usually local, such as a reboot or an ACL change. When many fail at once, the cause is almost always shared: a rotated password, a TACACS+ or RADIUS outage, a firewall rule blocking the Oxidized server, or a change to the account used for backups. The high failure rate alert catches this pattern early and gives it higher severity, since it means a broad loss of coverage.

Routing Alerts

Alertmanager decides who is notified. A minimal route groups alerts by group and alert name, so a storm of node failures produces one message instead of fifty.

yaml

route: receiver: netops-chat group_by: ["alertname", "group"] group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: netops-chat webhook_configs: - url: http://127.0.0.1:8080/alert send_resolved: true

The receiver here is a generic webhook, which you can point at the same style of verified receiver described in the earlier article on GitLab and GitHub webhooks, or replace with a native Slack, Microsoft Teams, email, or PagerDuty integration. Check amtool check-config before reloading Alertmanager.

Deploying Prometheus and Grafana with Docker Compose

If you do not already have a monitoring stack, containers are the quickest way to stand one up. The following Compose file runs Prometheus and Grafana, mounts your configuration, and keeps data in named volumes so it survives restarts.

yaml

services: prometheus: image: prom/prometheus:latest container_name: prometheus network_mode: host volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - ./prometheus/rules:/etc/prometheus/rules:ro - prometheus-data:/prometheus restart: unless-stopped grafana: image: grafana/grafana:latest container_name: grafana network_mode: host environment: - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password volumes: - grafana-data:/var/lib/grafana - ./grafana/provisioning:/etc/grafana/provisioning:ro secrets: - grafana_admin_password restart: unless-stopped secrets: grafana_admin_password: file: ./secrets/grafana_admin_password.txt volumes: prometheus-data: grafana-data:

Replace latest with specific version tags in production, so that an upgrade is a decision you make and not something that happens when a container restarts. Host networking is used here so that Prometheus can reach the exporter on localhost, which keeps the example short. In a larger deployment, use a dedicated Docker network and scrape the exporter by address. The __FILE suffix is a Grafana feature that reads a setting from a file, which keeps the admin password out of the Compose file.

Provision the Grafana Data Source

Provisioning lets Grafana configure itself from files, so your dashboard setup is repeatable and can live in version control.

Save this as grafana/provisioning/datasources/prometheus.yml. It registers Prometheus as the default data source.

yaml

apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://127.0.0.1:9090 isDefault: true

After starting Grafana, open Connections, Data sources and use the Save and test button to confirm that Grafana can query Prometheus. A common problem is a wrong URL when Grafana runs in a container that cannot see the host’s localhost address.

Building the Dashboard

A dashboard for backup status should answer three questions quickly: is everything fine, what is broken, and is it getting worse. Arrange panels in that order, from the top left.

Panel Plan

Panel	Visualization	Answers
Total nodes	Stat	How many devices are in scope
Failing nodes	Stat with thresholds	How many need attention now
Success rate	Gauge	Overall fleet health
Failures by group	Bar gauge	Where the problems cluster
Failing node list	Table	Exactly which devices to look at
Backup age	Table or bar gauge	Which nodes are stale
Success rate over time	Time series	Whether things are improving or degrading
Node state history	State timeline	Per-device behavior across days

Queries for Each Panel

These PromQL expressions power the panels. Enter each one in the query editor of a new panel.

For the Total nodes stat panel, use the value the exporter already provides.

promql

oxidized_nodes

For the Failing nodes stat panel, count series where the last run failed. The or vector(0) part makes the panel show zero instead of “No data” when nothing is failing.

promql

count(oxidized_node_last_run_success == 0) or vector(0)

For the Success rate gauge, calculate the fraction of nodes that succeeded.

promql

sum(oxidized_node_last_run_success) / count(oxidized_node_last_run_success)

For Failures by group, sum the failures per group so that site or device class problems stand out.

promql

sum by (group) (1 - oxidized_node_last_run_success)

For the Failing node list, return only failing series. Set the query type to Instant and the format to Table.

promql

oxidized_node_last_run_success == 0

For Backup age, calculate seconds since each node’s last run, then display it with the seconds (s) or duration unit.

promql

time() - oxidized_node_last_run_timestamp_seconds

For the Success rate over time graph, use the same ratio as the gauge in a time series panel.

promql

sum(oxidized_node_last_run_success) / count(oxidized_node_last_run_success)

Each query is intentionally small and readable. If a panel shows nothing, run the same expression in the Prometheus expression browser first, which separates a data problem from a Grafana configuration problem.

Panel Configuration Details

Visualization choices do as much work as the queries. A few settings make a big difference:

1. Thresholds on the failing nodes stat. Set green at 0, orange at 1, and red at a number that represents a serious outage for your estate. The color shift tells viewers whether to worry before they read the number.
2. Value mappings on the state timeline. Map 1 to “OK” in green and 0 to “Failed” in red. Without mappings, the panel shows bare numbers that mean nothing to a reader outside the team.
3. Table transformations. For the failing node list, add the Labels to fields or Organize fields transformation to hide Prometheus internals such as job and instance, and keep node, group, and model.
4. Sort order. Sort the backup age table by age descending, so the stalest devices appear first.
5. Units. Set the success rate panel to Percent (0.0 to 1.0), so the ratio shows as a percentage.

Add Variables for Filtering

Dashboard variables make one dashboard useful to several teams. Create a Query variable named group with this definition.

promql

label_values(oxidized_node_last_run_success, group)

Enable Multi-value and Include All option, then reference it in panel queries with a regular expression matcher. For example, the failing node list becomes the following.

promql

oxidized_node_last_run_success{group=~"$group"} == 0

Now the firewall team can filter to their group, while the core network team sees everything. Repeat the pattern with a model variable if you want to isolate vendor-specific issues.

Provision the Dashboard from Files

Once your dashboard looks right, export its JSON and store it in version control, then have Grafana load it automatically. This provider file tells Grafana where to find dashboard JSON files.

yaml

apiVersion: 1 providers: - name: oxidized orgId: 1 folder: Network Backups type: file disableDeletion: true allowUiUpdates: false options: path: /etc/grafana/provisioning/dashboards/json

Save this as grafana/provisioning/dashboards/oxidized.yml and place the exported dashboard JSON in the json subdirectory. Setting allowUiUpdates to false means changes must go through version control, which prevents configuration drift in the monitoring system itself. That is the same discipline you apply to device configurations.

The Textfile Collector Alternative

If you already run node_exporter on the Oxidized host and prefer not to add a service, you can generate a metrics file on a schedule instead. node_exporter’s textfile collector reads files with a .prom extension from a directory you configure with the --collector.textfile.directory flag.

How It Works

1. A script or timer queries /nodes.json every few minutes.
2. The script writes metrics in Prometheus text format to a temporary file.
3. The script renames the temporary file to its final .prom name, so node_exporter never reads a half-written file.
4. Prometheus scrapes node_exporter as usual and picks up the metrics.

The atomic rename in step 3 is the detail most people forget. Without it, a scrape that happens mid-write can produce corrupted or partial metrics. The trade-off compared with the custom exporter is freshness and cleanup: metrics reflect the last time the script ran, and you must handle stale files yourself if the script stops. If the script fails silently, node_exporter keeps serving the old file, so pair it with a freshness alert based on the file’s modification time, which the textfile collector exposes as a metric.

Real-World Use Cases

Morning Health Check on a Wall Display

A network operations team puts the top row of the dashboard on a shared screen. At the start of each shift, a quick look shows zero failing nodes or a short list to investigate. This replaces a manual review of the Oxidized web page and takes seconds.

Catching a Credential Rotation Gone Wrong

A security team rotates the backup account password and forgets to update Oxidized. Within a few polling cycles, the failure rate climbs above the fleet-wide threshold. The high failure rate alert fires, and the responder sees dozens of devices failing at the same time, pointing at credentials rather than at the network. Diagnosis takes minutes instead of hours.

Finding Chronic Problem Devices

A device that fails intermittently never triggers the failing node alert for long enough to notice. A time series of per-node state shows the pattern across weeks. Engineers find devices with flaky management plane connectivity, slow login prompts that trip timeouts, or banners that confuse the prompt regex in the model.

Audit and Compliance Reporting

Auditors want evidence that configuration backups run regularly and failures are handled. A dashboard with historical success rates, together with alert history from Alertmanager, shows that backup monitoring exists and that someone responds. Framework controls such as NIST SP 800-53 CP-9 (system backup) and CM-2 (baseline configuration) are relevant here, though a dashboard on its own is supporting evidence and not proof of compliance.

Validating Inventory Sync

If your inventory comes from LibreNMS or another source of truth, the count of nodes in Oxidized should track the count of devices in the source. A sudden drop in oxidized_nodes suggests a source problem, such as an API token that expired or a filter that excluded devices. Add a panel or an alert on large changes in the node count to catch this.

Security Considerations

Monitoring systems are often treated as harmless, but this one exposes your network inventory.

What This Data Reveals

Node names, groups, and models describe your network’s structure and vendor mix. Treat the dashboard and the metrics endpoint as sensitive information, even though no configuration content is exposed.

Exposure	Risk	Mitigation
Exporter endpoint	Inventory leak to anyone who can reach it	Bind to localhost or a management network, firewall the port
Oxidized API	Provides access to node data and, in some setups, configuration	Keep it on localhost, do not expose it to the network
Grafana	Inventory and status visible to all users	Use authentication, roles, and folder permissions
Alert messages	Device names sent to chat tools	Consider what third-party services receive

Practical Hardening Steps

1. Run the exporter on the same host as Oxidized and bind the API to 127.0.0.1, so that the API never crosses the network.
2. If Prometheus is on a different host, put the exporter behind a reverse proxy with TLS and authentication, or use a private management network. Prometheus supports basic authentication and TLS in scrape configuration.
3. Do not put the device IP address into metric labels unless you need it, to reduce the sensitivity of the data.
4. Enable authentication in Grafana and use organization roles and folder permissions to limit who sees which dashboards. Anonymous access should stay off.
5. Run the exporter as an unprivileged user with systemd sandboxing options, as shown earlier.
6. Keep Prometheus, Alertmanager, and Grafana patched, since monitoring stacks are a favored target because they hold broad visibility.

Never Expose the Oxidized API Directly

The Oxidized REST API can return configurations and trigger actions, depending on your setup. Placing it on a public interface or exposing it to a wide network to make scraping easier is a serious mistake. The exporter exists specifically so that only a narrow, read-only set of status metrics leaves the host.

Performance and Scalability

This workload is light, but it is worth understanding where it could grow.

Scrape Cost

Each scrape triggers one API request that serializes all nodes. For a few hundred devices, this is negligible. For many thousands, the response grows, so scrape less often (for example every two to five minutes) and consider caching the API response in the exporter for a short period. Because backup jobs run on the order of an hour, scrape frequency beyond a minute adds no value.

Series Growth

Each node adds a small, fixed number of series. Growth is linear in the number of devices, and device churn is low. The main risk is adding high-cardinality labels. Keep labels to node, group, and model, and avoid values that change often.

Recording Rules

If a dashboard computes expensive aggregations on every refresh across a large fleet, precompute them with recording rules. This rule file precomputes the fleet success ratio so that dashboards and alerts read a single cheap series.

yaml

groups: - name: oxidized_recording rules: - record: oxidized:success_ratio expr: sum(oxidized_node_last_run_success) / count(oxidized_node_last_run_success) - record: oxidized:failures_by_group expr: sum by (group) (1 - oxidized_node_last_run_success)

Use the recorded series in panels and alerts afterward. The colon in the name follows the Prometheus convention that marks a series as the output of a recording rule.

Retention

Prometheus keeps data for fifteen days by default. For trend analysis over months, increase retention with the --storage.tsdb.retention.time flag, or forward data to long-term storage if you already operate such a system. Backup status data is small, so longer retention costs little.

Automation Opportunities

Once metrics exist, they can drive further automation.

1. Auto-created tickets. Route the failing node alert to your ticketing system through Alertmanager, with the node and group as fields.
2. Self-healing attempts. For known transient problems, trigger a re-poll through the Oxidized API’s node refresh feature from an alert webhook. Keep this narrow and rate-limited, so that you never create a retry storm against a struggling device.
3. Change correlation. Combine this data with the Git push pipeline from the earlier article. A burst of configuration changes followed by backup failures can indicate that a change broke management access.
4. Dashboards as code. Keep dashboard JSON, rule files, and provisioning YAML in a Git repository and deploy them through CI, so that the monitoring configuration has the same review process as your network configuration.
5. Blackbox probing. Use the Prometheus blackbox exporter to check that the Oxidized web endpoint returns an HTTP success response, which gives you a second, independent signal alongside the exporter’s own oxidized_up metric.

Monitoring the Monitoring

A monitoring system that fails silently is worse than none, because it creates false confidence. Build in a few safeguards.

Check	Method	Purpose
Exporter scrape health	up{job="oxidized"} alert	Detects a dead exporter
Prometheus alive	External or secondary check	Detects a dead Prometheus
Alert delivery	Periodic test alert or a watchdog rule	Detects broken notification paths
Data freshness	Alert on stale timestamps	Detects a stuck Oxidized

A watchdog rule is an alert that always fires. If the receiver stops getting it, the alerting pipeline itself is broken. Many Prometheus setups include one by convention, and you should have one too.

Routine Maintenance

1. Review the failing node list weekly and either fix or retire persistent failures.
2. Tune thresholds after the first month, using the real failure patterns you observed.
3. Update the exporter when you upgrade Oxidized, since API fields can change between releases.
4. Test alert routing quarterly by deliberately breaking a lab node.
5. Review dashboard access after staff changes.

Troubleshooting

Troubleshooting Reference

Symptom	Likely cause	First action
oxidized_up is 0	Exporter cannot reach the API	Check Oxidized, port, and the URL setting
Target down in Prometheus	Exporter stopped or port blocked	Check the service and firewall
No node metrics but oxidized_up is 1	API returned an empty list	Check the Oxidized source and inventory
Timestamps missing	Time format differs from the parser	Inspect last.end and update the formats
Dashboard panels show “No data”	Wrong data source or query	Test the query in the Prometheus browser
All nodes appear stale	Oxidized stuck or clock skew	Check Oxidized logs and system time
Duplicate series	Two exporters scraped for the same node	Remove the duplicate scrape target
Alerts never fire	Rule file not loaded	Run promtool check rules and check Prometheus status

Step-by-Step Diagnosis When Metrics Are Missing

1. Run curl against the Oxidized API and confirm it returns node data.
2. Run curl against the exporter’s /metrics endpoint and confirm the oxidized_ metrics appear.
3. Check the Prometheus Targets page to confirm the exporter is up.
4. Run the metric name in the Prometheus expression browser and confirm it returns series.
5. Run the dashboard query in Grafana’s Explore view to confirm it works with the data source.
6. Only then inspect panel settings such as transformations and value mappings.

Working outward from the source in this order isolates the layer that is failing, and it avoids the common mistake of fiddling with Grafana panels when the data never reached Prometheus.

Timezone and Clock Problems

If every node looks hours old or timestamps appear to be in the future, the likely cause is a timezone mismatch between how Oxidized writes times and how the exporter interprets them. The exporter in this article assumes UTC. If your Oxidized timestamps are in local time, adjust the parsing, or better, run Oxidized and the exporter on hosts with synchronized clocks, using NTP, so that this class of error cannot occur.

Common Mistakes to Avoid

1. Alerting on a single failed run. Hourly polling makes one failure normal. Use for durations.
2. Putting status in labels. It increases series churn and complicates queries.
3. Using the Pushgateway for node status. A pull-based exporter is simpler and avoids stale pushed data.
4. Ignoring freshness. A last status of success means little if the run was three days ago.
5. Exposing the Oxidized API for convenience. Keep it local and expose only the exporter if needed.
6. Using high-cardinality labels. Avoid IP addresses, configuration text, or any unbounded values.
7. Not testing alerts. An untested alert path is an assumption.
8. Building the dashboard only in the UI. Without provisioning, dashboards are hard to reproduce after a rebuild.
9. Skipping the fleet-wide alert. Without it, a shared credential failure produces dozens of individual alerts and no clear diagnosis.
10. Hard-coding assumptions about the API. Fields change between versions, so verify your parsing after upgrades.

Best Practices Summary

Implementation Checklist

1. Enable oxidized-web and bind the API to localhost.
2. Confirm the fields in /nodes.json with curl and jq.
3. Deploy the exporter as a sandboxed systemd service.
4. Add a scrape job and validate the config with promtool.
5. Create alert rules for exporter health, API health, failing nodes, stale backups, and fleet failure rate.
6. Route alerts through Alertmanager and test the full path.
7. Provision the Grafana data source from a file.
8. Build panels for totals, failures, success rate, group breakdown, failing list, and backup age.
9. Add variables for group and model filtering.
10. Store dashboards, rules, and provisioning files in version control.
11. Add a watchdog alert to detect a broken alerting pipeline.
12. Review thresholds and failing nodes on a regular schedule.

Decision Framework

Question	If yes	If no
Do you already run node_exporter and want minimal change?	Textfile collector	Custom exporter
Do you need near-real-time status?	Custom exporter	Either works
Is the fleet over several thousand nodes?	Longer scrape interval, response caching	Defaults are fine
Do several teams use the dashboard?	Variables and folder permissions	Single dashboard
Is the environment regulated?	Retention planning, access controls, audit evidence	Standard setup

Expert Recommendations

These come from practical experience, and are my opinions rather than vendor guidance:

1. Measure freshness, not just success. The most valuable alert in this whole setup is the stale backup rule. It catches the case where nothing is failing because nothing is running.
2. Start with fewer, better alerts. Teams that begin with fifteen alerts end up muting all of them. Begin with the five in this article and add only when a real incident shows a gap.
3. Group-level dashboards win. People rarely want a list of five hundred devices. They want to know which site or device class is hurting, then drill in.
4. Treat the exporter as disposable. It holds no state, so keep it simple, keep it in version control, and rebuild it whenever you like.
5. Connect the dots with your change pipeline. Backup failures right after a change window are among the clearest signals that a change broke management access. If you push diffs to a remote repository, correlate the two.

Future Trends

Several directions are worth watching, with the caveat that they are trends and not commitments from any vendor:

1. OpenTelemetry adoption. More teams are moving toward OpenTelemetry for metrics, and Prometheus supports receiving OTLP data in recent releases. Your exporter could eventually be replaced by a collector pipeline.
2. Observability for network automation. As network teams adopt software engineering practices, backup systems, pipelines, and validation jobs will be monitored with the same service-level objectives as application services.
3. Unified dashboards. Grafana increasingly acts as a single interface across metrics, logs, and traces, so backup status can sit next to device health, flow data, and change events.
4. Policy-driven alerting. Expect more alert logic to be generated from inventory and source-of-truth data, instead of hand-written per device.

Frequently Asked Questions

Does Oxidized have a built-in Prometheus exporter?

Not as far as the project’s documentation shows. The usual approach is to read node status from the oxidized-web REST API and expose it through a small exporter or the node_exporter textfile collector. Check the current project documentation, since a community or built-in option could appear in later releases.

What is the best way to get Oxidized metrics into Prometheus?

For most environments, a small custom exporter that reads /nodes.json at scrape time is the cleanest option. It stays fresh, removes series for deleted nodes automatically, and needs no extra state. The textfile collector is a good alternative if you already run node_exporter.

Which Oxidized API endpoint should I use?

The /nodes.json endpoint returns the list of nodes with their last job result, which is what you need for status monitoring. Confirm the exact fields on your version with curl before writing parsing code.

How do I alert when a device has not been backed up recently?

Expose the end time of the last job as a Unix timestamp and alert when the difference between time() and that value exceeds your threshold, for example six hours for an hourly polling interval. This is the stale backup rule shown above.

Should I use the Pushgateway for Oxidized?

Generally no. The Prometheus documentation recommends the Pushgateway only for specific cases such as service-level batch jobs, and it has drawbacks including stale metrics. A pull-based exporter fits node status better.

How often should Prometheus scrape the exporter?

One minute is plenty. Backup jobs run on the order of an hour, so faster scraping adds load without adding information. For very large fleets, a two to five minute interval is reasonable.

How do I show backup status per site in Grafana?

Use the group label from Oxidized, which you can map to sites or device classes in your inventory source, and aggregate with sum by (group). Add a dashboard variable so viewers can filter by group.

Can Grafana alert directly instead of Prometheus?

Yes, Grafana has its own alerting system. Prometheus alerting rules with Alertmanager are a common choice because rules live in version-controlled files and keep working if Grafana is down. Pick one system for backup alerts to avoid duplicates.

What should I do if the exporter shows all nodes as failed at once?

Treat it as a shared cause until proven otherwise. Check whether the backup account’s credentials changed, whether the AAA servers are reachable, and whether a firewall change blocks the Oxidized server. Also confirm that the Oxidized service itself is healthy.

Is it safe to expose these dashboards to the whole company?

Probably not without thought. Node names and groups reveal network structure. Use Grafana authentication and folder permissions, and share only what each audience needs.

Key Takeaways

1. Backup systems need monitoring of their own, and freshness matters as much as the last result.
2. Oxidized exposes node status through the oxidized-web REST API, and a small exporter turns it into Prometheus metrics.
3. Use gauges with a 0 or 1 success value and a Unix timestamp, and keep labels to node, group, and model.
4. A fleet-wide failure rate alert quickly reveals shared causes such as credential changes or AAA outages.
5. Provision Grafana data sources and dashboards from files, and keep them in version control.
6. Keep the Oxidized API on localhost and secure the exporter and Grafana, since inventory data is sensitive.
7. Add a watchdog alert, so that a broken alerting pipeline does not go unnoticed.
8. Verify API fields and timestamp formats after every Oxidized upgrade.

Conclusion

A backup that nobody watches gives a false sense of safety. By turning Oxidized’s node status into Prometheus metrics, you replace that false sense with evidence: which devices are covered, which are not, and how the picture changes over time. Grafana makes that evidence readable to everyone from the engineer fixing a failed login to the manager asking whether the network is recoverable.

For next steps, get the exporter running and confirm the metrics by hand. Add the five alert rules and send a test alert through the whole path. Then build the top row of the dashboard, tune your thresholds after a few weeks of real data, and put everything in version control. From there, connect the dashboard to your change pipeline, so that backup health and configuration changes tell one story.