Oxidized with Ansible: Deploying and Managing Oxidized at Scale with Playbooks

Oxidized is an open source network device configuration backup tool, and Ansible is the automation engine that lets you deploy, configure, and operate it the same way every time. Combine them and you get a backup platform that can be rebuilt from a Git repository in minutes, scaled across regions without hand-editing configuration files, and audited through ordinary code review.

Quick answer: To deploy Oxidized with Ansible, create a role that installs the Ruby dependencies and the oxidized, oxidized-script, and oxidized-web gems (or runs the official container image), renders the Oxidized configuration and the device list from Jinja2 templates, manages credentials with Ansible Vault, installs a systemd service, and triggers a node reload through a handler whenever the inventory changes. Run the playbook with check mode and diff first, then roll it out with serial so a bad change never reaches every collector at once.

This guide walks through that approach end to end. It starts with the concepts, builds a complete role step by step, then moves into inventory generation, secrets, scaling, security, testing, and troubleshooting. If you have already followed the earlier posts in this series on syncing Oxidized with LibreNMS, pushing diffs to GitLab and GitHub, and visualizing backup status in Grafana, this article is the piece that makes all of them repeatable.

Summary box: what you will build

1. An Ansible project with separate production and staging inventories.
2. A reusable oxidized role covering packages, service account, gems, configuration, device list, and systemd unit.
3. A container-based alternative using the official image.
4. A device list generated from Ansible inventory, with notes on NetBox as a dynamic source.
5. Vault-protected credentials, a hardened service, and post-deployment health checks.
6. A rolling-update and testing workflow suitable for CI.

Why Automate Oxidized with Ansible

A single Oxidized server installed by hand works fine for a lab. The trouble starts when the deployment has to survive staff turnover, a second data center, a rebuilt VM, or an audit question like “who changed the backup credentials and when?” Hand-built servers drift, and the knowledge of how they were built lives in someone’s shell history.

Ansible fixes this by turning the build into declarative, version-controlled code. It is agentless, so the only requirement on the target host is SSH and Python. Its YAML syntax is readable by network engineers who do not write software for a living, and its template engine maps neatly onto Oxidized’s own YAML configuration and its simple colon-delimited device list.

What Oxidized Does

Oxidized logs in to network devices over SSH or Telnet, runs the commands defined for each device model, cleans the output, and stores the result. Its recommended output backend is Git, which creates a commit on every configuration change, so you get a full history and a diff of every change. The project describes itself as a RANCID replacement and supports a large catalog of operating system models, from Cisco IOS and NX-OS to Juniper Junos, Arista EOS, Fortinet, MikroTik, and Linux-based platforms.

Oxidized reads its list of devices from a source, which can be a CSV file in RANCID router.db style, an SQLite database, or an HTTP endpoint returning structured data. It also exposes a REST API and web interface through the separate oxidized-web gem, and it supports hooks that run after events such as a configuration being stored.

What Ansible Adds

Ansible contributes five things that matter for Oxidized specifically:

• Idempotent builds. Running the playbook twice produces the same server, and a second run only changes what has drifted.
• Templated configuration. The Oxidized config file and the device list are generated from variables rather than edited by hand.
• Secret handling. Ansible Vault keeps device credentials and API tokens encrypted in the repository.
• Controlled rollouts. Keywords such as serial and max_fail_percentage let you update collectors gradually.
• A single source of truth. The same inventory that describes your network can drive the Oxidized device list.

When This Approach Is Worth It

Not every team needs a role-based deployment. The table below is a quick decision aid.

Situation	Hand install	Ansible role	Ansible plus dynamic inventory
Lab or proof of concept, under 20 devices	Reasonable	Optional	Overkill
One production server, stable device list	Acceptable but fragile	Recommended	Optional
Multiple sites or collectors	Hard to keep consistent	Strongly recommended	Recommended
Frequent device onboarding and decommissioning	Error prone	Recommended	Best fit
Audit or change-control requirements	Weak evidence trail	Good	Best

Architecture Overview

Before writing a playbook, it helps to see the moving parts. An Ansible-managed Oxidized deployment has two layers: the automation layer that builds and configures, and the runtime layer that actually backs up devices.

Core Components

Component	Layer	Role in the deployment
Ansible control node	Automation	Runs playbooks, holds the vault password, reads the inventory
Inventory	Automation	Lists Oxidized servers and the network devices to back up
Oxidized role	Automation	Installs packages, renders configuration, manages the service
Oxidized server	Runtime	Polls devices on an interval and stores configurations
Device source	Runtime	The router.db file, SQLite database, or HTTP endpoint listing devices
Git output repository	Runtime	Stores one commit per configuration change
oxidized-web REST API	Runtime	Provides the web UI, node status, and reload endpoint
Remote Git hosting	Integration	Optional off-box copy through a hook

How It Works

The flow below is the sequence that repeats on every playbook run and on every Oxidized polling cycle.

1. An engineer changes the inventory or a variable and commits it to Git.
2. The playbook runs from the control node, or from a CI job, against the Oxidized servers.
3. The role ensures packages, user, directories, and gems are in place.
4. Templates render the Oxidized configuration and the device list on the target.
5. If either file changed, a handler restarts the service or asks the running instance to reload its node list.
6. Oxidized polls each device on its interval, connects with the configured credentials, collects the output for that device model, and commits any change to the Git repository.
7. Hooks optionally push the repository to a remote and notify other systems.

Deployment Patterns

Pattern	Best for	Trade-offs
Native install with systemd	Teams that want simple troubleshooting and direct control of Ruby gems	You own the Ruby and gem lifecycle
Container with the official image	Teams already running Docker or Podman	You must map volumes and the REST listener correctly
One Oxidized instance per region	Large or segmented networks	More servers to patch, but shorter, safer network paths
Multiple instances on one host	Separating customers or security zones cheaply	Needs distinct home directories and ports

Because the role is parameterized, the same code can drive every pattern. Only the variables change.

Prerequisites and Planning

Control Node Requirements

You need a Linux or macOS control node with a currently supported Ansible release, Git, and network reachability to the Oxidized servers over SSH. Install the collections the role depends on from a requirements.yml file so the dependency list is versioned alongside the code:

yaml

--- collections: - name: community.general - name: community.docker - name: netbox.netbox

Install them with ansible-galaxy collection install -r requirements.yml. The community.general collection provides the gem module, community.docker is only needed for the container variant, and netbox.netbox is only needed if you adopt NetBox as a dynamic source later. Pin versions in CI so a collection update cannot surprise you on deployment day.

Sizing and Capacity

Oxidized uses a pool of worker threads and an interval. Its default behavior is to add and remove threads automatically to meet the configured retrieval interval. The practical consequence is that the interval, the number of devices, and the time each device takes to answer determine how many concurrent SSH sessions your server opens. Start with a modest threads value, watch the logs, and raise it gradually. The project documentation does not publish a universal sizing formula, so treat any number you find elsewhere as a starting point to validate against your own devices.

Device Accounts and Access

Plan the access model before you write any code, because it shapes your variables:

1. Create a dedicated read-only account for Oxidized on every device, ideally through TACACS+ or RADIUS so it is centrally managed.
2. Restrict that account to the show commands Oxidized needs where your platform supports command authorization.
3. Decide whether enable passwords are required for specific models, and plan to pass them as per-node variables rather than globally.
4. Allow SSH to devices only from the Oxidized server addresses.

Designing the Ansible Project Structure

A clean layout makes the project understandable to the next engineer. The paths below form a conventional structure, listed as a plain list so it stays readable in any publishing platform.

1. ansible.cfg holds defaults such as the inventory path and the roles path.
2. requirements.yml lists the required collections.
3. site.yml is the top-level playbook.
4. inventories/production/ and inventories/staging/ each contain an inventory file plus a group_vars directory.
5. inventories/production/group_vars/all/vault.yml holds encrypted secrets.
6. roles/oxidized/defaults/main.yml holds overridable defaults.
7. roles/oxidized/tasks/main.yml holds the tasks.
8. roles/oxidized/handlers/main.yml holds handlers.
9. roles/oxidized/templates/ holds config.j2, router.db.j2, and oxidized.service.j2.

Keeping staging and production inventories separate lets you test every change against a staging Oxidized server that polls a handful of lab devices before production sees it.

Inventory Design

The inventory describes two kinds of hosts: the Oxidized servers that Ansible connects to, and the network devices that Ansible only reads about. The devices are never connected to by Ansible in this design. They exist in the inventory purely as data that feeds the device list template.

This YAML inventory defines one collector and a few devices with the variables the templates need:

yaml

--- all: children: oxidized_servers: hosts: oxidized-01.example.net: ansible_host: 10.20.0.15 network_devices: hosts: core-sw-01.example.net: oxidized_model: ios oxidized_group: core core-sw-02.example.net: oxidized_model: nxos oxidized_group: core edge-fw-01.example.net: oxidized_model: fortios oxidized_group: edge ansible_host: 10.30.1.1

Each device carries an oxidized_model and an optional oxidized_group. If a device has an ansible_host, the template uses it as the address to poll; otherwise it uses the inventory name, which must then resolve in DNS. Verify each model name against the model list in the Oxidized documentation, because a typo here produces a node that fails on every cycle.

Group Variables

Global settings live in group_vars/all. Secrets go in a separate file that you encrypt, as covered in the Vault section later. Non-secret settings look like this:

yaml

--- oxidized_interval: 3600 oxidized_threads: 30 oxidized_timeout: 20 oxidized_retries: 3 oxidized_rest_listen: "127.0.0.1:8888" oxidized_git_user: "Oxidized" oxidized_git_email: "oxidized@example.net"

An interval of 3600 seconds means every device is polled hourly. Shorter intervals give finer change tracking but increase load on both the server and the devices, so match the interval to how quickly you actually need to detect change.

Building the Oxidized Role

With the structure in place, the role comes together one file at a time. Create the skeleton with ansible-galaxy role init roles/oxidized and then fill in each file.

Role Defaults

Defaults define every variable the role understands, so anyone reading the role can see what is configurable in one place.

yaml

--- oxidized_user: oxidized oxidized_group_name: oxidized oxidized_home: /etc/oxidized oxidized_data_dir: /var/lib/oxidized oxidized_apt_packages: - git - ruby - ruby-dev - libsqlite3-dev - libssl-dev - pkg-config - cmake - libssh2-1-dev - libicu-dev - zlib1g-dev - g++ oxidized_gems: - name: oxidized - name: oxidized-script - name: oxidized-web oxidized_default_model: ios oxidized_interval: 3600 oxidized_threads: 30 oxidized_timeout: 20 oxidized_retries: 3 oxidized_rest_listen: "127.0.0.1:8888" oxidized_ssh_secure: true oxidized_scope_group: network_devices oxidized_username: "" oxidized_password: ""

The package list mirrors the build dependencies named in the upstream README for Debian-family systems. Confirm it against the README for your distribution release, since package names change between releases. To pin a gem version, add a version key to an entry in oxidized_gems. Setting oxidized_ssh_secure to true enables host key verification, which is the safer default, but it means each device’s host key must already be known to the service account. If that is not practical on day one, set it to false deliberately and document the decision.

Tasks: Packages, Account, and Directories

The first block of tasks prepares the host. It installs build dependencies, creates a dedicated service account with no login shell, and builds the directory structure with strict permissions.

yaml

--- - name: Install packages required to build and run Oxidized ansible.builtin.apt: name: "{{ oxidized_apt_packages }}" state: present update_cache: true cache_valid_time: 3600 - name: Create the Oxidized group ansible.builtin.group: name: "{{ oxidized_group_name }}" system: true - name: Create the Oxidized service account ansible.builtin.user: name: "{{ oxidized_user }}" group: "{{ oxidized_group_name }}" system: true home: "{{ oxidized_data_dir }}" shell: /usr/sbin/nologin create_home: true - name: Create the Oxidized home directory ansible.builtin.file: path: "{{ oxidized_home }}" state: directory owner: "{{ oxidized_user }}" group: "{{ oxidized_group_name }}" mode: "0750"

Expect these tasks to report changes on the first run and “ok” afterward. The most common pitfall is a Red Hat family host: the package names and the module differ, so swap ansible.builtin.apt for ansible.builtin.dnf and adjust the package list. If you manage mixed distributions, put the package list in vars/Debian.yml and vars/RedHat.yml and load them with include_vars keyed on ansible_os_family.

Tasks: Installing the Gems

Oxidized is distributed as a Ruby gem, along with oxidized-script for running ad hoc commands and oxidized-web for the REST API and web interface. The upstream documentation notes that if you do not install oxidized-web you must remove the rest option from the configuration, so keep the two consistent.

yaml

- name: Install Oxidized gems community.general.gem: name: "{{ item.name }}" version: "{{ item.version | default(omit) }}" state: present user_install: false loop: "{{ oxidized_gems }}" notify: Restart oxidized

This installs the gems system-wide. With state: present and no version, an existing install is left alone, which makes upgrades a deliberate act. To upgrade intentionally, either bump the pinned version or run the playbook with an extra variable that switches the state to latest. Gem compilation can take several minutes on small VMs, so allow for that in CI timeouts.

Templates: The Oxidized Configuration

The configuration template turns variables into the YAML file Oxidized reads. The upstream documentation shows that configuration is sourced from /etc/oxidized/config and then ~/.config/oxidized/config, and that you can set the OXIDIZED_HOME environment variable to change the home directory. This role uses OXIDIZED_HOME pointing at /etc/oxidized.

yaml

--- # {{ ansible_managed }} username: {{ oxidized_username | to_json }} password: {{ oxidized_password | to_json }} model: {{ oxidized_default_model }} interval: {{ oxidized_interval }} threads: {{ oxidized_threads }} timeout: {{ oxidized_timeout }} retries: {{ oxidized_retries }} debug: false prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ rest: {{ oxidized_rest_listen }} input: default: ssh, telnet debug: false ssh: secure: {{ oxidized_ssh_secure | lower }} output: default: git git: user: {{ oxidized_git_user }} email: {{ oxidized_git_email }} repo: "{{ oxidized_data_dir }}/devices.git" source: default: csv csv: file: {{ oxidized_home }}/router.db delimiter: !ruby/regexp /:/ map: name: 0 model: 1 group: 2

Two details deserve attention. First, the to_json filter wraps the username and password in double quotes and escapes special characters, which makes the value valid YAML even if a password contains a colon, hash, or quote. Second, the keys shown follow the structure in the project’s sample configuration and README; run the installed version once without a configuration to let it generate its own default file, and compare the two when you adopt a newer release.

Because this task renders credentials, mark it with no_log: true so passwords never appear in playbook output or diffs:

yaml

- name: Render the Oxidized configuration ansible.builtin.template: src: config.j2 dest: "{{ oxidized_home }}/config" owner: root group: "{{ oxidized_group_name }}" mode: "0640" no_log: true notify: Restart oxidized

The file is owned by root and readable by the service group, so the Oxidized process can read its credentials but cannot rewrite its own configuration. The trade-off of no_log is harder debugging, so temporarily remove it only on a non-production host when a template error needs investigation.

Templates: The Device List

The device list is where Ansible earns its keep. This template emits one line per device in the colon-delimited format the CSV source expects, using the name, model, and group columns mapped in the configuration above.

jinja

{% for host in groups[oxidized_scope_group] | sort %} {{ hostvars[host].ansible_host | default(host) }}:{{ hostvars[host].oxidized_model | default(oxidized_default_model) }}:{{ hostvars[host].oxidized_group | default('default') }} {% endfor %}

Ansible’s template module trims block lines by default, so the loop tags do not leave blank lines in the output. Sorting the host list keeps the file stable between runs, which matters because an unstable order would make Ansible report a change every time. The task that deploys it should notify a reload handler rather than a full restart:

yaml

- name: Render the Oxidized device list ansible.builtin.template: src: router.db.j2 dest: "{{ oxidized_home }}/router.db" owner: root group: "{{ oxidized_group_name }}" mode: "0640" notify: Reload oxidized nodes

If you need a different address than the inventory name, set ansible_host on the device. If you need to skip a device temporarily, move it out of the network_devices group instead of commenting it out, so the intent is visible in Git history.

The systemd Service

A systemd unit gives you automatic restarts, boot-time startup, and logging through the journal. This template runs Oxidized as the service account with a few low-risk hardening options.

ini

# {{ ansible_managed }} [Unit] Description=Oxidized network device configuration backup After=network-online.target Wants=network-online.target [Service] User={{ oxidized_user }} Group={{ oxidized_group_name }} Environment=OXIDIZED_HOME={{ oxidized_home }} ExecStart=/usr/local/bin/oxidized Restart=on-failure RestartSec=10 NoNewPrivileges=true PrivateTmp=true [Install] WantedBy=multi-user.target

The path /usr/local/bin/oxidized is where system-wide gem executables commonly land on Debian-family systems, but it is not guaranteed. Run command -v oxidized on a built host and adjust the template, or expose the path as a variable. Stronger options such as ProtectSystem can break Oxidized because it writes its repository, log, and process files, so introduce them one at a time in staging.

Deploy it and manage the service with these tasks:

yaml

- name: Install the Oxidized systemd unit ansible.builtin.template: src: oxidized.service.j2 dest: /etc/systemd/system/oxidized.service owner: root group: root mode: "0644" notify: - Reload systemd - Restart oxidized - name: Enable and start Oxidized ansible.builtin.systemd: name: oxidized enabled: true state: started

Handlers

Handlers run once at the end of a play, no matter how many tasks notified them. That behavior is exactly what you want here, because a configuration change and a unit change in the same run should produce one restart, not two.

yaml

--- - name: Reload systemd ansible.builtin.systemd: daemon_reload: true - name: Restart oxidized ansible.builtin.systemd: name: oxidized state: restarted - name: Reload oxidized nodes ansible.builtin.uri: url: "http://{{ oxidized_rest_listen }}/reload" method: GET status_code: [200, 302]

The upstream documentation describes a REST call for reloading the node list without restarting, which is why device list changes use a reload instead of a restart and avoid interrupting a polling cycle. The accepted status codes are deliberately permissive because redirect behavior can differ between versions of oxidized-web; tighten them after you confirm what your version returns. This handler requires the REST listener to be bound to an address reachable from the target host itself, which the loopback default satisfies.

The Top-Level Playbook

Tie everything together with a short playbook. Setting serial: 1 makes Ansible finish one Oxidized server before touching the next.

yaml

--- - name: Deploy and configure Oxidized hosts: oxidized_servers become: true serial: 1 roles: - role: oxidized tags: [oxidized]

Running It Safely

Always preview before you apply. Check mode with diff shows what would change without changing it:

bash

ansible-playbook -i inventories/staging site.yml --check --diff --ask-vault-pass

Review the output, then run it for real against staging, confirm devices are being backed up, and only then target production:

bash

ansible-playbook -i inventories/production site.yml --ask-vault-pass --limit oxidized-01.example.net

Check mode has limits. Tasks that depend on earlier changes, such as starting a service whose unit file does not exist yet, can report failures in a dry run of a brand-new host. Treat check mode as a strong signal on existing servers and a rough one on fresh builds.

Container Deployment Variant

If your organization standardizes on containers, the same inventory and templates can drive the official image. The project publishes the image as oxidized/oxidized and documents mounting a host directory into the container as its configuration directory, with a CONFIG_RELOAD_INTERVAL environment variable to reload configuration periodically.

yaml

- name: Run Oxidized as a container community.docker.docker_container: name: oxidized image: "oxidized/oxidized:{{ oxidized_image_tag }}" state: started restart_policy: unless-stopped published_ports: - "127.0.0.1:8888:8888" volumes: - "{{ oxidized_home }}:/root/.config/oxidized" env: CONFIG_RELOAD_INTERVAL: "600"

Three cautions apply. First, pin oxidized_image_tag to a specific release rather than latest so a rebuild cannot silently change behavior. Second, the container mount path shown follows the upstream README examples, but images change over time, so verify the path against the documentation for the tag you pin. Third, inside the container the REST listener must bind to all interfaces for the published port to work, while the published port itself stays bound to loopback on the host. Set oxidized_rest_listen to 0.0.0.0:8888 for the container case and keep the host-side publish on 127.0.0.1.

Generating the Device List from Your Inventory

The real payoff of this design is that onboarding a device becomes a pull request. Add the host to the inventory with a model, merge, run the playbook, and the device appears in Oxidized without anyone logging in to the backup server.

Static Inventory as the Source

The static approach shown earlier is enough for many teams. Keep one file per site under inventories/production/, and let group_vars carry shared settings. Use naming conventions consistently so you can filter by site or role later. The built-in ansible-inventory --graph command prints the group structure, which is a fast way to confirm the devices landed in the groups you intended.

Dynamic Inventory with NetBox

If NetBox already holds your device records, you can avoid maintaining a second list. The netbox.netbox.nb_inventory plugin builds an Ansible inventory from the NetBox API, and Oxidized models can then be assigned from NetBox attributes such as platform.

yaml

--- plugin: netbox.netbox.nb_inventory api_endpoint: https://netbox.example.net validate_certs: true group_by: - sites - device_roles - platforms

Save this as a file ending in nb_inventory.yml inside the inventory directory and supply the API token through the environment rather than the file. The plugin creates groups from the attributes you list in group_by, and the exact group names depend on your NetBox slugs and plugin version. Run ansible-inventory -i inventories/production --graph to see what you actually get, then attach the Oxidized model to the platform groups with a group variable file:

yaml

--- oxidized_model: ios

Name that file after the generated group, for example group_vars/platforms_cisco_ios.yml, and repeat for each platform. The advantage of mapping at the group level is that a single line covers every device on that platform, and a new platform needs one new file instead of a model on every host. Treat the NetBox plugin options as something to confirm against the collection documentation for the version you install, since option names evolve.

HTTP Source as an Alternative

Oxidized can also read devices from an HTTP endpoint that returns one object per device, with a map from your field names to the Oxidized fields. This is useful when another system already exposes an API. In that case you template the source section of the configuration instead of a CSV file:

yaml

source: default: http http: url: https://inventory.example.net/api/devices scheme: https delimiter: !ruby/regexp /:/ map: name: hostname model: os

Use this when the authoritative list lives outside Ansible. Use the CSV approach when Ansible’s inventory is the source of truth, because it removes a network dependency from the backup path. The upstream documentation also notes that the configuration can be reloaded automatically when an HTTP source changes, which is the reason the container variant sets a reload interval.

Mapping Inventory Variables to Oxidized

Inventory variable	Oxidized field	Purpose	Notes
ansible_host or inventory name	name	Address or hostname Oxidized connects to	Must be reachable from the Oxidized server
oxidized_model	model	Selects the command set and output cleaning	Must match a supported model name
oxidized_group	group	Organizes devices and repositories	Also enables per-group credentials
oxidized_username (optional)	username	Per-node credential override	Prefer group-level credentials

Per-Group Credentials

Real networks rarely share one account everywhere. Oxidized supports a groups section that can override credentials for devices in a group, so edge firewalls can use a different account than core switches. Add this block to the configuration template:

yaml

{% if oxidized_group_credentials | length > 0 %} groups: {% for name, creds in oxidized_group_credentials.items() %} {{ name }}: username: {{ creds.username | to_json }} password: {{ creds.password | to_json }} {% endfor %} {% else %} groups: {} {% endif %}

Define oxidized_group_credentials: {} in the role defaults and override it from encrypted variables. When groups are in use, the Git output can create one repository per group in the same directory, as the upstream documentation describes, so confirm that layout is what you want before you rely on it for retention or access control.

Managing Secrets with Ansible Vault

Device credentials are the most sensitive data in the whole deployment. Anyone who reads them can log in to your network. Keep them out of plain-text variable files from the first commit.

1. Create an encrypted file: ansible-vault create inventories/production/group_vars/all/vault.yml.
2. Inside it, define variables with a vault_ prefix, such as vault_oxidized_username and vault_oxidized_password.
3. In a plain group_vars/all/main.yml, map them: oxidized_username: "{{ vault_oxidized_username }}".
4. Provide the vault password at runtime with --ask-vault-pass, or with --vault-password-file pointing to a file readable only by the automation account.

The prefix convention keeps the variable names searchable in plain text while the values stay encrypted, so a reviewer can see which settings exist without being able to read them. In CI, store the vault password in the platform’s secret store and write it to a temporary file for the run. Rotate the Oxidized device credentials on a schedule, and because the password flows from a single variable, rotation is a vault edit plus one playbook run.

If you also push backups to a remote Git repository, store the deploy key the same way and deploy it with restrictive permissions:

yaml

- name: Install the Git deploy key ansible.builtin.copy: content: "{{ vault_oxidized_deploy_key }}\n" dest: "{{ oxidized_home }}/deploy_key" owner: "{{ oxidized_user }}" group: "{{ oxidized_group_name }}" mode: "0600" no_log: true when: oxidized_remote_repo is defined

Expect a changed result only when the key content changes. A common pitfall is a missing trailing newline in a private key, which causes SSH to reject it as invalid, so keep the explicit newline shown.

Pushing Configurations to Remote Repositories

Local Git history protects against bad changes but not against losing the server. A remote copy closes that gap. Oxidized hooks subscribe to events such as post_store, and the githubrepo hook runs a git push to a configured remote. Append this to the configuration template, guarded by a condition so servers without a remote skip it:

yaml

{% if oxidized_remote_repo is defined %} hooks: push_to_remote: type: githubrepo events: [post_store] remote_repo: {{ oxidized_remote_repo }} username: git publickey: {{ oxidized_home }}/deploy_key.pub privatekey: {{ oxidized_home }}/deploy_key {% endif %}

The upstream hook documentation warns that the remote name must not match the name of the local repository configured under the Git output, so give the two distinct names. If the private key has a passphrase, the documentation describes an environment variable for supplying it. Our dedicated article on GitLab and GitHub webhooks covers hook behavior, authentication choices, and failure handling in depth, so use this section only for the Ansible side of the wiring.

Scaling Oxidized with Ansible

Scaling Oxidized is mostly about three things: how fast one instance can poll, how many instances you run, and how safely you can change them.

Tuning Threads, Interval, and Timeout

Setting	What it controls	Raise it when	Risk of raising it
interval	Seconds between polling cycles	Backups finish well within the cycle and you want fewer sessions	Slower detection of changes
threads	Maximum concurrent device sessions	Cycles overrun the interval	More load on devices and AAA servers
timeout	Seconds to wait on a device	Slow or distant devices fail spuriously	Slow devices tie up threads longer
retries	Attempts before marking a node failed	Flaky links cause intermittent failures	Longer failure detection

Because these are role variables, you can set tighter or looser values per Oxidized server using host_vars, which is how you give a high-latency remote site a longer timeout without penalizing the rest of the network.

One Instance per Region

A single central collector is simple, but it makes every polling session cross your WAN and forces broad firewall rules. Regional collectors keep sessions local and shorten blast radius. The role supports this through the oxidized_scope_group variable. Give each collector a scope group in its host_vars, and change the device list template so it only emits the devices in that scope:

jinja

{% for host in groups[oxidized_scope_group] | sort %} {{ hostvars[host].ansible_host | default(host) }}:{{ hostvars[host].oxidized_model | default(oxidized_default_model) }}:{{ hostvars[host].oxidized_group | default('default') }} {% endfor %}

The template is unchanged from earlier. What changes is the value of oxidized_scope_group, set per server in the inventory, for example site_london for the London collector. Make sure every device belongs to exactly one regional group, or you will back it up twice, and consider a check task that fails the play if any device is in no scope group.

Rolling Updates

Changing every collector at once is how a typo becomes an outage. The serial keyword on the play processes hosts one at a time, and max_fail_percentage stops the rollout if too many hosts fail. Pair it with a health check so a collector must prove it is working before Ansible moves on:

yaml

- name: Wait for the Oxidized REST API ansible.builtin.wait_for: host: 127.0.0.1 port: 8888 timeout: 60 - name: Read the node list ansible.builtin.uri: url: "http://{{ oxidized_rest_listen }}/nodes.json" return_content: true register: oxidized_nodes - name: Confirm the expected number of nodes loaded ansible.builtin.assert: that: - oxidized_nodes.json | length == groups[oxidized_scope_group] | length fail_msg: "Oxidized loaded a different number of nodes than the inventory defines"

Place these in a verify.yml task file and include it at the end of the role. The node list endpoint and its JSON structure come from oxidized-web, so confirm both against the version you run before depending on the assertion. A failed assertion stops that host and, with serial in place, prevents the rest from being touched.

Security Considerations

A backup system holds the keys to the network and a copy of every configuration, which makes it an attractive target. Build the controls into the role rather than adding them later.

Risk	Control	Where it lives
Credential exposure in the repository	Ansible Vault for all secrets, no_log on tasks that render them	Vault files and task keywords
Over-privileged device account	Read-only role, command authorization, central AAA	Device and AAA configuration
Man-in-the-middle on device SSH	Enable host key verification with secure: true	oxidized_ssh_secure variable
Unauthenticated REST API	Bind to loopback and place a TLS reverse proxy with authentication in front	oxidized_rest_listen and proxy configuration
Service compromise	Dedicated unprivileged user, NoNewPrivileges, restricted file modes	Role tasks and systemd unit
Sensitive data in stored configs	Review model support for secret removal and restrict repository access	Oxidized configuration and Git hosting permissions
Lateral movement from the server	Firewall rules allowing SSH to devices only from Oxidized	Network and host firewall

Treat the Git repository itself as sensitive. Configuration backups routinely contain SNMP communities, key material, and topology details, even when Oxidized strips some secrets for certain models. Restrict who can read the repository, and prefer a private remote.

Testing and Validation

Automation without testing only makes mistakes faster. Three layers of testing catch most problems.

1. Static checks. Run ansible-lint and ansible-playbook --syntax-check on every change.
2. Staging deployment. Apply the change to a staging collector that polls a few lab devices and confirm commits appear in its repository.
3. Role-level tests. Use Molecule with a container or VM driver to converge the role on a clean image and verify that the service is running and the configuration file has the expected permissions.

A minimal continuous integration job runs the static checks on every pull request:

yaml

name: ansible-checks on: pull_request: jobs: lint: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install tooling run: pip install ansible ansible-lint - name: Install collections run: ansible-galaxy collection install -r requirements.yml - name: Lint run: ansible-lint - name: Syntax check run: ansible-playbook -i inventories/staging site.yml --syntax-check

The job needs no vault password because linting and syntax checks do not decrypt secrets, though a syntax check may need a placeholder if your play references vault variables at parse time. Keep deployment jobs separate and protect them behind manual approval for production.

Monitoring and Maintenance

A backup tool that silently stops is worse than no backup tool, because people assume it works. Add monitoring to the role’s definition of done.

Operational Checks

1. Confirm the service is active with systemctl status oxidized and read recent logs with journalctl -u oxidized.
2. Query the REST API or web interface for nodes whose last status is not success.
3. Verify that commits are still landing in the repository for devices you know changed.
4. Alert if no commit has occurred in a period longer than your longest expected quiet window.

For dashboards and alerting on backup status, the companion article on Grafana and Prometheus shows how to expose node status as metrics, and you can deploy that exporter from the same role by adding tasks and a handler.

Upgrades and Housekeeping

Upgrade deliberately. Pin the gem or image version in variables, change it in staging first, read the project’s release notes for configuration changes, and then promote the same change to production through the usual pull request. After each upgrade, run the verification tasks and spot-check a few devices.

Back up the Oxidized server’s repository like any other data, even if you also push to a remote. A remote copy protects you from losing the server, while a periodic archive of the data directory protects you from a force-push or a deleted remote repository. Keep the Ansible repository itself in a highly available Git host, since it is the recovery plan for everything else.

Troubleshooting

Symptom	Likely cause	What to check or do
Gem installation fails during the play	Missing build dependencies	Confirm the package list was installed; read the gem build log for the missing library
Service starts then exits immediately	YAML error in the configuration, or rest set without oxidized-web installed	Run Oxidized in the foreground as the service user and read the error
Devices missing from the web interface	Device list not reloaded, or delimiter and map mismatch	Check router.db contents and call the reload endpoint
Node status stays failed	Wrong credentials, wrong model, blocked path, or host key verification failing	Test SSH manually from the server with the same account; confirm the model name
REST API unreachable	Listener bound to a different address than expected	Compare oxidized_rest_listen with what the process is listening on
Playbook reports a change on every run	Unstable template output	Sort loops and avoid timestamps in templates
Handler did not fire	An earlier task failed, so handlers were skipped	Fix the failure, or use meta: flush_handlers where ordering matters
Vault decryption errors	Wrong password or mixed vault identities	Confirm the vault password file and any vault IDs match

When a device keeps failing, reproduce the problem outside Oxidized first. If a manual SSH session from the Oxidized server cannot log in, no amount of tuning inside Oxidized will fix it. Legacy devices that only offer older SSH algorithms are a frequent cause, and the right response is to upgrade the device or deliberately and narrowly relax the client side rather than weaken everything.

Common Mistakes to Avoid

1. Storing device passwords in plain-text variables. Use Vault from the first commit, because removing a secret from Git history later is painful.
2. Using latest for gems or container images. An unattended rebuild can change behavior in the middle of an audit window.
3. Restarting instead of reloading. A restart interrupts in-flight polls; a reload of the node list does not.
4. Letting the inventory and reality diverge. Decommissioned devices stay in the list and generate failure noise. Remove them in the same change that retires the device.
5. Skipping staging. A role that works on a freshly built VM can still break an existing server with years of drift.
6. Disabling host key verification permanently. It is a reasonable bootstrap step, but record it as a risk with a plan to close it.
7. Ignoring the REST listener. A web interface exposed to the whole network is an easy source of configuration disclosure.
8. Running everything with become and no restraint. Only the tasks that need privilege should have it, and the service itself should never run as root.

Best Practices Summary

Area	Recommendation
Structure	One role, separate staging and production inventories, collections pinned in requirements.yml
Secrets	Vault for all credentials, no_log on rendering tasks, rotate on a schedule
Changes	Check mode and diff first, serial rollouts, verification tasks after every run
Inventory	Single source of truth with models and groups defined as variables
Operations	Monitor commit activity and node status, not just whether the process is running
Recovery	Remote Git copy plus periodic archive; Ansible repo treated as critical infrastructure

Expert Recommendations

A few judgment calls separate a deployment that merely works from one that stays healthy for years.

First, treat the device list as data owned by your source of truth, not as a file someone edits. If you have NetBox or another system of record, generate from it. Second, keep the role small and boring. Resist the urge to embed site-specific logic in tasks; put it in variables so a new region is a new inventory entry, not a new branch of code. Third, decide early which failures should block a rollout and encode them as assertions. A health check that costs ten seconds will save you from discovering a broken collector the next morning. Finally, document the recovery path and rehearse it. Rebuilding a collector from the repository, restoring the Git data, and confirming devices are polled again is the real test of whether your automation delivers on its promise.

Future Trends

Several directions are worth watching, none of which change the fundamentals described here. Source-of-truth platforms such as NetBox are becoming the standard origin for device lists, which makes inventory-driven backup configuration increasingly common. Event-driven approaches are also growing: the upstream project documents using syslog to detect configuration change events and trigger an immediate fetch, which complements scheduled polling by capturing changes closer to when they happen. Teams are also moving toward pipeline-driven operations, where the same pull request that changes a device record deploys the backup configuration and runs the validation. Whatever tooling arrives, keeping configuration backups reproducible and reviewable will remain the sound foundation.

Frequently Asked Questions

What is the best way to deploy Oxidized with Ansible?

For most teams, a dedicated role that installs the gems natively and runs Oxidized as a systemd service is the simplest to troubleshoot. Teams that already operate containers can use the official image with the same templates. In both cases, keep the configuration and device list templated and secrets in Ansible Vault.

Does Oxidized have an official Ansible role?

The upstream project documents installing from gems, from source, and with Docker, and I am not aware of an officially maintained Ansible role. Community roles exist, but their quality and maintenance vary, so review any role before using it. Writing a small role of your own, as shown above, is often easier to audit.

How do I add a new device to Oxidized with Ansible?

Add the device to the inventory with the correct oxidized_model and, if needed, an oxidized_group and ansible_host. Run the playbook, which regenerates the device list and calls the reload handler. The device is picked up on the next polling cycle.

How do I keep device passwords secure in an Ansible deployment?

Store them in an Ansible Vault encrypted file, reference them from plain variables, and mark any task that renders them with no_log. Restrict the rendered configuration file to root ownership with group read access for the service account, and give Oxidized a read-only device account.

Should I restart or reload Oxidized after changing the device list?

Reload. The REST API provides a reload call that rereads the node list without interrupting running polls. Use a restart only when the main configuration file or the service definition changes.

How many devices can one Oxidized server handle?

It depends on device response times, the polling interval, and the thread count, and the project does not publish a fixed limit. Start with modest threads, watch for cycles that overrun the interval, and add regional collectors when a single instance becomes a bottleneck or the network paths become awkward.

Can I run Oxidized in Docker with Ansible?

Yes. Use the community.docker.docker_container module with a pinned image tag, mount your rendered configuration directory, and publish the REST port on loopback. Check the mount path against the documentation for the exact image tag you use.

Key Takeaways

1. Ansible turns an Oxidized server into reproducible infrastructure: rebuildable, reviewable, and consistent across sites.
2. Generate both the Oxidized configuration and the device list from templates and inventory variables.
3. Protect credentials with Ansible Vault and no_log, and give Oxidized a dedicated read-only device account.
4. Reload the node list for device changes and restart only for configuration or service changes.
5. Roll out with serial, verify with assertions, and promote changes from staging to production.
6. Monitor commits and node status, back up the repository, and treat the Ansible repo as your recovery plan.

Conclusion

Oxidized does the hard part of network configuration backup, and Ansible makes that capability dependable at scale. A small role with a handful of templates gives you a collector you can rebuild on demand, a device list that follows your inventory, secrets that stay encrypted, and a rollout process that fails safely. Start by deploying the role to a staging server with a few lab devices, confirm that commits arrive in Git, and then move production over one collector at a time. From there, layer on the integrations covered elsewhere in this series: LibreNMS for inventory sync, remote repositories through webhooks, and Grafana for visibility. Each one becomes easier when the base deployment is already code.