Oxidized with GitLab and GitHub Webhooks: Push Config Diffs to Remote Repos

Quick answer: Oxidized stores every device configuration change as a Git commit. To get those commits into GitLab or GitHub, you use an Oxidized hook on the post_store event, either the built-in githubrepo hook or an exec hook that runs git push. Webhooks then work in the opposite direction. GitLab or GitHub calls your receiver whenever a push arrives, so you can trigger chat alerts, CI validation, and change tickets.

This guide takes you from the basic ideas to a production design. It covers architecture, working configuration, a signature-verifying webhook receiver, security, scaling, and troubleshooting. A note on scope: Oxidized changes between releases, so check the hook options against the documentation for the version you run (the Oxidized GitHub repository and its docs/Hooks.md are the reference). Where I am not certain a behaviour is the same in every version, I say so.

Why Push Network Config Diffs to a Remote Git Repository

Oxidized is an open source network configuration backup tool written in Ruby. It logs in to routers, switches, firewalls, and other devices on a schedule, pulls the running configuration, and saves it. With the Git output enabled, every change becomes a commit, so you get a full history and readable diffs for free.

That local repository is useful, but it has three weaknesses in real environments:

1. It is a single point of failure. If the Oxidized host dies and the disk is gone, so is your configuration history.
2. It is invisible to the rest of the team. Engineers, auditors, and security staff rarely have shell access to the backup server.
3. It cannot trigger anything. A local bare repository does not notify Slack, open a ticket, or start a validation pipeline.

A remote repository on GitLab or GitHub fixes all three. You get an off-box copy, a web interface with diff views and blame, access control, and event hooks. This is the practical core of “configuration management as code” for networks, where the repository becomes the audit trail for what changed, when, and on which device.

What You Will Build

By the end you will have:

1. Oxidized writing configs to a local Git repository.
2. An Oxidized hook that pushes each new commit to a remote on GitLab or GitHub.
3. A webhook configured on the remote that notifies a small receiver service.
4. Secret handling, deploy keys, and monitoring around the whole pipeline.

Key Concepts: Oxidized, Git, Hooks, and Webhooks

Four terms get mixed up constantly, so it is worth separating them before touching any configuration.

Term	What it is	Direction of travel
Oxidized source	Where Oxidized learns which devices to back up (CSV, SQL, HTTP such as LibreNMS)	Inventory into Oxidized
Oxidized output	Where configs are stored (file, Git, HTTP)	Oxidized to local storage
Oxidized hook	An action Oxidized runs when an internal event fires	Oxidized to an external action
Git webhook	An HTTP POST that GitLab or GitHub sends when something happens in a repository	Remote repo to your receiver

The confusion usually comes from the word “webhook”. In Oxidized’s own vocabulary, the feature that reacts to events is called a hook. GitLab and GitHub call their outbound HTTP notifications webhooks. In this article, “hook” means the Oxidized side and “webhook” means the Git platform side.

Oxidized Hook Events

Oxidized hooks attach to events. The ones you will use most often are:

Event	When it fires	Typical use
node_success	A node was fetched successfully	Metrics, heartbeat
node_fail	A node failed after its retries	Alerting on unreachable devices
post_store	After a configuration was stored (and committed, when using Git output)	Push to remote, notify on change
nodes_done	After a full cycle over all nodes finishes	Batch actions, one push per cycle

For pushing diffs, post_store is the natural event. It means a new revision has just been saved, so the remote receives changes close to real time.

Why Git Is a Good Fit for Configurations

Configurations are line-oriented text, and Git is built for line-oriented text. You get content-addressed history, cheap branching, signed commits if you want them, and tooling that every engineer already knows. The weakness is that network configs often contain secrets such as SNMP communities, TACACS keys, and password hashes, and pushing those to a hosted platform is a security decision, not a convenience. The Security Considerations section deals with that directly.

Architecture Overview

The pipeline has four stages, and each stage has its own failure modes.

1. Collection. Oxidized connects to devices over SSH or Telnet (SSH should be the only option you use in production) and fetches configuration using the device model definitions.
2. Local commit. The Git output writes the configuration and commits it with the author name and email you configured.
3. Remote push. A hook fires on post_store and pushes the local branch to GitLab or GitHub.
4. Event delivery. The platform receives the push and calls your webhook URL with a JSON payload. A receiver validates the request and fans it out to chat, ticketing, or CI.

Components and Responsibilities

Component	Responsibility	Typical location
Oxidized service	Scheduling, device login, config fetch, local commit	Dedicated Linux VM or container
Local Git repository	Authoritative history on the Oxidized host	Oxidized data directory
Deploy key or token	Credential that authorizes pushes to one repository	Oxidized host, restricted file permissions
Remote repository	Off-box copy, review UI, access control	GitLab or GitHub
Webhook receiver	Verifies and routes push notifications	Small internal service behind a reverse proxy
Downstream consumers	Chat, ticketing, CI, SIEM	Existing tooling

Placement Decision: Where Should the Receiver Live?

GitHub and GitLab.com send webhooks from the public internet, so the receiver needs a reachable HTTPS endpoint. You have three realistic options:

1. Expose a hardened receiver through a reverse proxy. Simple, and fine if you restrict by path, rate limit, and verify signatures.
2. Use a self-managed GitLab instance inside your network. The webhook never leaves the corporate network. This is the cleanest option for regulated environments.
3. Skip inbound webhooks and use polling or CI. If you cannot expose anything, run a pipeline on the repository instead and let CI do the work.

Do not put the Oxidized REST API on the internet to make this work. The webhook receiver and Oxidized should be separate services.

Prerequisites and Environment Preparation

You need a working Oxidized installation. If you do not have one yet, the project supports gem-based installs on Linux and an official container image. If you are feeding Oxidized from LibreNMS, the inventory side is already handled and this article picks up from the output stage.

Before you start, confirm the following:

1. Oxidized runs as a service and successfully backs up at least one device.
2. The Git output works and git log in the repository shows commits.
3. You have an empty repository on GitLab or GitHub, with no README, to avoid an unrelated-histories conflict on the first push.
4. The Oxidized host can reach the remote over SSH (port 22) or HTTPS (port 443). Many corporate firewalls allow 443 only, and GitHub supports SSH over port 443 through ssh.github.com for that situation.

Install the Git Output Dependencies

Oxidized’s Git output depends on the rugged Ruby gem, which needs build tools and libgit2 headers on many distributions. On a Debian or Ubuntu system the usual preparation looks like this.

This installs the compilers and libraries needed to build the gems that Oxidized uses for Git support.

bash

sudo apt-get update sudo apt-get install -y ruby ruby-dev build-essential cmake pkg-config \ libssl-dev libssh2-1-dev libicu-dev zlib1g-dev libgit2-dev git sudo gem install oxidized oxidized-script oxidized-web

Expect rugged to compile during installation, which can take a few minutes. A frequent pitfall is a missing cmake or libssl-dev, which makes the native build fail. If you run the container image instead, these dependencies are already included.

Configure Oxidized Local Git Output

Start with a minimal, correct local setup. Oxidized reads its configuration from ~/.config/oxidized/config for the user running it, unless you point it elsewhere with the OXIDIZED_HOME environment variable.

The following configuration enables Git output, sets the commit author, and enables the REST API on localhost only.

yaml

--- username: backup password: change-me model: ios interval: 3600 use_syslog: true debug: false threads: 30 timeout: 20 retries: 3 prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ rest: 127.0.0.1:8888 vars: remove_secret: true input: default: ssh ssh: secure: false output: default: git git: user: Oxidized email: oxidized@example.com repo: "/var/lib/oxidized/configs.git" source: default: csv csv: file: /var/lib/oxidized/router.db delimiter: !ruby/regexp /:/ map: name: 0 ip: 1 model: 2 group: 3

A few notes on this file. The remove_secret: true line tells Oxidized to strip secrets that the device model knows how to identify, so passwords and community strings are replaced before the configuration is saved. This is a first line of defense, not a guarantee, because it depends on the model definition recognizing every secret format. The secure: false setting for SSH disables strict host key checking, which is convenient in a lab and a poor idea in production, where you should set it to true and maintain a known hosts file. The repo path points at a bare repository that Oxidized creates on first run.

Group-Based Repositories

By default, Oxidized stores all devices in one repository. If you want one repository per device group (for example, one for core routers and another for firewalls), enable the multi-repository mode.

This setting changes the Git output so that each group gets its own repository under the configured path.

yaml

output: default: git git: user: Oxidized email: oxidized@example.com single_repo: false repo: routers: "/var/lib/oxidized/routers.git" switches: "/var/lib/oxidized/switches.git" firewalls: "/var/lib/oxidized/firewalls.git"

The trade-off is operational. Separate repositories make access control cleaner, because the firewall team can be given access to the firewall repository only, but they also multiply the number of remotes and deploy keys you have to manage. Be consistent with the group names that your source supplies, because a node whose group has no mapped repository can lead to errors.

Verify the Local Repository

Before adding any remote logic, prove the local side works. Run Oxidized, wait for one cycle, then inspect the repository.

This checks that commits exist and shows the most recent change in a readable form.

bash

git --git-dir=/var/lib/oxidized/configs.git log --oneline -n 5 git --git-dir=/var/lib/oxidized/configs.git show --stat HEAD

If the log is empty, nothing has been stored yet. Check the Oxidized log for login failures before going any further.

Method One: The Built-In githubrepo Hook

Oxidized ships a hook named githubrepo. Despite the name, it is implemented on top of Git itself through the rugged library, so the remote does not have to be GitHub. In my experience with this hook, the same configuration pattern works against GitLab and other Git servers that accept the credentials, but treat that as something to test in your own environment rather than assume.

Configuration

This block adds a hook that pushes to a remote every time a configuration is stored.

yaml

hooks: push_to_remote: type: githubrepo events: [post_store] remote_repo: git@gitlab.example.com:netops/device-configs.git username: oxidized publickey: /var/lib/oxidized/.ssh/id_ed25519.pub privatekey: /var/lib/oxidized/.ssh/id_ed25519

The hook name push_to_remote is arbitrary. What matters is type, events, and the remote definition. For HTTPS remotes, the hook also supports a password field, which you would populate with a token rather than a real account password.

If you use group-based repositories, remote_repo can be a mapping from group name to remote URL, matching the repository layout you set up earlier. Check the hook documentation for your Oxidized version for the exact key names, since this is the part of the hook that has changed between releases.

Strengths and Limits of the Built-In Hook

Aspect	Built-in githubrepo hook	exec hook with git push
Setup effort	Low, pure YAML	Medium, needs a script
Credential handling	Key or token in Oxidized config	Whatever Git and SSH on the host provide
Error visibility	Logged by Oxidized	Fully under your control
Flexibility	Push only	Anything a script can do
Dependencies	rugged gem	Git binary

The built-in hook is the quickest route and a reasonable default. Its limits show up when you want retries, custom branches, signed commits, or detailed logging. At that point the exec hook is easier to control.

Method Two: The exec Hook with git push

The exec hook runs a command when an event fires, and it passes context about the event through environment variables. This gives you full control over how the push happens.

Environment Variables Available to exec Hooks

Variable	Meaning
OX_EVENT	The event name that triggered the hook
OX_NODE_NAME	Device name
OX_NODE_IP	Device IP address
OX_NODE_FROM	Where the node was fetched from
OX_NODE_MSG	Message associated with the node, such as an error
OX_NODE_GROUP	Device group
OX_NODE_MODEL	Device model
OX_JOB_STATUS	Result of the job
OX_JOB_TIME	Job time
OX_REPO_COMMITREF	Commit reference of the stored config
OX_REPO_NAME	Repository name

Not every variable is populated for every event, so write scripts that tolerate empty values. Confirm the list for your release in the hook documentation.

Hook Configuration

This hook calls a shell script after each stored configuration, runs it asynchronously, and limits how long it may run.

yaml

hooks: push_git: type: exec events: [post_store] cmd: '/usr/local/bin/oxidized-push.sh' async: true timeout: 120

With async: true, Oxidized does not wait for the script, so a slow remote does not delay the next device. The timeout value protects you from a hung push that would otherwise leave a process waiting indefinitely.

The Push Script

The script below pushes the local bare repository to a remote, uses a dedicated deploy key, and serializes concurrent runs with flock, so two overlapping hook executions cannot collide.

bash

#!/bin/bash set -euo pipefail REPO="/var/lib/oxidized/configs.git" REMOTE="git@gitlab.example.com:netops/device-configs.git" KEY="/var/lib/oxidized/.ssh/id_ed25519" LOCK="/var/lock/oxidized-push.lock" export GIT_SSH_COMMAND="ssh -i ${KEY} -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/var/lib/oxidized/.ssh/known_hosts" ( flock -w 60 9 || { logger -t oxidized-push "lock timeout"; exit 1; } git --git-dir="${REPO}" push --quiet "${REMOTE}" master:master \ && logger -t oxidized-push "pushed ${OX_NODE_NAME:-unknown} ${OX_REPO_COMMITREF:-}" \ || logger -t oxidized-push "push failed for ${OX_NODE_NAME:-unknown}" ) 9>"${LOCK}"

Make it executable with chmod 750 and owned by the Oxidized user. Three details deserve attention:

1. Branch name. Older Oxidized versions commit to master, and this is what the script assumes. Run git --git-dir=/var/lib/oxidized/configs.git branch to confirm what your installation uses and adjust the refspec to match.
2. Host key verification. StrictHostKeyChecking=yes requires the remote’s key in known_hosts beforehand. Add it once with ssh-keyscan and compare the fingerprint to the one published by GitLab or GitHub instead of trusting it blindly.
3. No force push. The script never uses --force. If the remote diverges, the push fails loudly, which is what you want from an audit repository.

Method Three: Remote Mirroring Without a Hook

There is a third approach that moves the work off the Oxidized host entirely. GitLab can mirror repositories, and push mirroring is available on all tiers, while pull mirroring is limited to paid tiers on self-managed and GitLab.com instances at the time of writing. Check GitLab’s current documentation to confirm what your tier includes, since plan packaging changes.

With pull mirroring, GitLab fetches from a repository that is reachable from it. That requires your Oxidized repository to be served over a network endpoint, which most teams do not want. For that reason, hook-based push from Oxidized is the more common design.

Choosing Between the Methods

Situation	Recommended method
Small environment, quick win	Built-in githubrepo hook
Need retries, logging, branches	exec hook with a script
Strict network segmentation, no outbound from Oxidized	Server-side mirroring or a separate sync host
Many groups, many repositories	exec hook with a mapping table in the script

Set Up GitLab as the Remote

The GitLab side has two jobs: accept pushes securely, and send webhooks when pushes arrive.

Create the Project and a Deploy Key

1. Create an empty project, for example netops/device-configs, and set its visibility to Private.
2. Generate a dedicated key pair on the Oxidized host, with no other privileges.
3. In the project, open Settings, Repository, Deploy keys, add the public key, and enable Grant write permissions to this key.

This generates a modern SSH key used only for pushing configuration backups.

bash

sudo -u oxidized ssh-keygen -t ed25519 -f /var/lib/oxidized/.ssh/id_ed25519 -N "" -C "oxidized-push@configs" sudo -u oxidized ssh-keyscan -t ed25519 gitlab.example.com >> /var/lib/oxidized/.ssh/known_hosts

A deploy key is scoped to one project, which is the main reason to prefer it over a personal SSH key. If the key leaks, an attacker can push to one repository and nothing else. The empty passphrase is a pragmatic choice for an unattended service, so protect the private key with file permissions (chmod 600) and, where you can, restrict it at the network level.

Protect the Branch

Because Oxidized is the only legitimate writer, protect the default branch and allow pushes only from the deploy key’s role. In GitLab, deploy keys with write access can be allowed on a protected branch under Settings, Repository, Protected branches. Disable force push. This keeps history intact even if somebody with developer access tries to rewrite it.

Create a GitLab Webhook

1. Open Settings, Webhooks in the project.
2. Enter the receiver URL, for example https://hooks.example.com/gitlab.
3. Set a Secret token. GitLab sends it back in the X-Gitlab-Token header.
4. Select Push events. If you only care about the default branch, set a branch filter.
5. Keep Enable SSL verification turned on.

GitLab includes an event type header, X-Gitlab-Event, and the JSON body contains object_kind set to push for push events. The payload carries the branch reference, the before and after commit IDs, and a list of commits with their messages and the files they touched. Those fields are what your receiver uses to build a useful notification.

Set Up GitHub as the Remote

GitHub follows the same pattern with different names for the pieces.

Create the Repository and Deploy Key

1. Create a private repository with no initial files.
2. Open Settings, Deploy keys, Add deploy key, paste the public key, and tick Allow write access.
3. Optionally add a branch protection rule that blocks force pushes and deletions on the default branch.

Deploy keys on GitHub are tied to a single repository, and they are read-only unless you explicitly enable write access. A key can be used on one repository only, so a multi-repository layout needs one key per repository, which is something to plan for if you chose group-based repositories.

Alternative: HTTPS with a Fine-Grained Token

If your firewall allows only HTTPS, use a fine-grained personal access token limited to the single repository, with only the contents write permission. Store it outside the Oxidized configuration where possible, and rotate it on a schedule, because tokens expire. Where an organization allows it, a GitHub App installation token is a stronger choice for automation, because it is not tied to a person’s account.

Create a GitHub Webhook

1. Open Settings, Webhooks, Add webhook.
2. Set the Payload URL to https://hooks.example.com/github.
3. Set Content type to application/json.
4. Enter a Secret.
5. Choose Just the push event.
6. Keep SSL verification enabled and save.

GitHub sends the event name in X-GitHub-Event and an HMAC-SHA256 signature of the raw request body in X-Hub-Signature-256, prefixed with sha256=. When you save a new webhook, GitHub sends a ping event so you can confirm delivery, and the delivery history page in the webhook settings shows each request, its response, and lets you redeliver it. That redelivery feature is extremely useful while you are developing the receiver.

Build a Webhook Receiver That Verifies Signatures

A receiver turns raw push notifications into something your team can act on. The example below is deliberately small: it handles both platforms, verifies authenticity, and extracts the changed device configuration files.

Why Verification Is Not Optional

Anyone who discovers your webhook URL can send it a forged request. Without verification, an attacker could trigger false alarms or, worse, trigger downstream automation. GitHub’s HMAC signature and GitLab’s secret token exist to prevent that, and constant-time comparison prevents timing leaks.

The Receiver Code

This Flask application validates the GitHub signature or the GitLab token, then logs which files changed in each push. It reads secrets from environment variables rather than source code.

python

import hashlib import hmac import os from flask import Flask, request, abort, jsonify app = Flask(__name__) GITHUB_SECRET = os.environ["GITHUB_WEBHOOK_SECRET"].encode() GITLAB_TOKEN = os.environ["GITLAB_WEBHOOK_TOKEN"] def changed_files(commits): files = set() for commit in commits: for key in ("added", "modified", "removed"): files.update(commit.get(key, [])) return sorted(files) @app.post("/github") def github(): signature = request.headers.get("X-Hub-Signature-256", "") expected = "sha256=" + hmac.new(GITHUB_SECRET, request.get_data(), hashlib.sha256).hexdigest() if not hmac.compare_digest(signature, expected): abort(401) event = request.headers.get("X-GitHub-Event", "") if event == "ping": return jsonify(status="pong") if event != "push": return jsonify(status="ignored") payload = request.get_json() files = changed_files(payload.get("commits", [])) app.logger.info("github push ref=%s files=%s", payload.get("ref"), files) return jsonify(status="ok", files=files) @app.post("/gitlab") def gitlab(): token = request.headers.get("X-Gitlab-Token", "") if not hmac.compare_digest(token, GITLAB_TOKEN): abort(401) payload = request.get_json() if payload.get("object_kind") != "push": return jsonify(status="ignored") files = changed_files(payload.get("commits", [])) app.logger.info("gitlab push ref=%s files=%s", payload.get("ref"), files) return jsonify(status="ok", files=files)

Run it behind a production WSGI server such as Gunicorn, not the Flask development server, and put it behind a TLS-terminating reverse proxy. Common pitfalls include parsing the JSON before computing the signature, which can change the bytes, and returning slowly. Both platforms expect a quick response, and GitHub in particular times out slow deliveries, so acknowledge fast and do heavy work in a queue.

Fan-Out Ideas for the Receiver

Once the receiver trusts the request, you can route it:

1. Post a summary to Slack or Microsoft Teams listing the devices that changed.
2. Open or update a ticket in your ITSM tool, tagging the change window.
3. Trigger a CI pipeline that lints or validates the changed configurations.
4. Write an event into your SIEM for correlation with other security data.

Real-World Use Cases

Change Detection and Alerting

An operations team receives a Slack message each time a core router’s configuration changes, with a link to the exact diff in GitLab. Unplanned changes outside a maintenance window stand out immediately. This is the simplest and most valuable application of the pattern, and it replaces a lot of manual log reading.

Compliance Evidence and Audit Trails

Auditors frequently ask who changed what and when. A Git history with named commits and timestamps answers the “what” and “when”. Be careful about the “who”, though. Oxidized records that a change was observed, not who made it on the device. To attribute changes to people, correlate commit timestamps with AAA logs from TACACS+ or RADIUS accounting, or with your change management records. Mapping the repository to a framework such as NIST SP 800-53 configuration management controls (CM-2 baseline configuration and CM-3 change control) or CIS benchmarks is reasonable, but the repository alone is evidence, not compliance.

Configuration Validation in CI

With the repository on GitLab or GitHub, a pipeline can run on every push. Checks might include scanning for forbidden commands, verifying that required NTP and syslog servers are present, or flagging accidental enable password strings. This turns a passive backup into an active guardrail.

Disaster Recovery

When a device fails, you pull its last known good configuration from the remote repository instead of from the backup server. Because the remote is off-box, recovery does not depend on the Oxidized host surviving.

Multi-Team Visibility

Repository permissions let you give the security team read access without giving them any access to network devices or the Oxidized host. The web interface becomes the shared source of truth.

Security Considerations

This is the area where backup pipelines most often go wrong. Network configurations are sensitive, and pushing them to a third-party platform changes your risk profile.

Treat the Repository as Sensitive

Risk	Why it matters	Mitigation
Secrets in configs	Community strings, key material, and password hashes may be stored	Use remove_secret, review model output, scan commits
Public exposure	A mistakenly public repository leaks the network design	Private repositories, organization-level policies
Overly broad credentials	A leaked key could expose more than backups	Per-repository deploy keys, fine-grained tokens
Forged webhooks	Fake events trigger automation	Signature or token verification
Hosted platform trust	Data lives with a third party	Self-managed GitLab for regulated data

Secret Handling in Practice

Oxidized’s remove_secret option depends on what each model definition knows how to mask. Do not assume it covers every vendor and every field. After enabling it, inspect real output from each device model you use and look for anything that resembles a credential. Add a secret scanning step to CI as a second layer. GitHub offers secret scanning and push protection on repositories where it is available, and GitLab offers secret detection in its CI templates, though availability depends on your tier. Check the current documentation for what your plan includes.

Credential Hygiene

1. Use a different key for every repository and never reuse a personal key.
2. Keep private keys readable only by the Oxidized user.
3. Rotate tokens on a schedule and immediately after staff changes.
4. Keep webhook secrets in a secrets manager or environment files with strict permissions, not in code.
5. Log pushes, and alert on a push failure rather than discovering weeks later that the remote is stale.

Webhook Endpoint Hardening

Serve the receiver only over HTTPS, accept only the expected HTTP method and path, and apply rate limits at the reverse proxy. GitHub publishes the IP ranges its webhooks originate from through its meta API, and GitLab.com does the same in its documentation, so you can restrict inbound traffic to those ranges if you want a network-level control on top of signature checks. Treat that as defense in depth, since ranges can change.

Principle of Least Privilege on the Oxidized Host

Run Oxidized as an unprivileged account, not as root. Give it a read-only device account where the vendor supports one, since Oxidized only needs to read configuration. Limit its outbound network access to the devices and the single Git remote.

Performance and Scalability

Pushing a commit is cheap, but large estates multiply small costs.

Push Frequency

With post_store and hundreds of devices, you may trigger many pushes in a short period when a polling cycle completes. Each push is incremental, so the data volume is small, but the connection overhead adds up. Two sensible options:

1. Keep post_store with async: true and the flock serialization shown earlier, so overlapping pushes queue instead of colliding.
2. Switch to nodes_done, which pushes once at the end of each full cycle. This reduces remote load and webhook noise, at the cost of a delay of up to one polling interval before the remote is updated.

Repository Growth

Configurations change rarely, so growth is usually modest. Problems appear when a model includes volatile lines such as timestamps, uptime counters, or rotating hashes, which produce a new commit on every run for no real reason. Fix those at the source by filtering them in the model’s cfg block, because noisy diffs destroy the value of both alerts and audits. The Oxidized documentation covers how to adjust model output, and the same pattern lets you strip lines such as “Last configuration change” banners on devices that include them.

Interval and Thread Tuning

The interval setting (in seconds) and threads control how aggressively Oxidized polls. Start with an interval of 3600 and a thread count your devices and AAA servers can tolerate, then adjust. Aggressive polling can lock out accounts or overload older devices, which will cost you more than slightly staler backups.

Monitoring and Maintenance

A pipeline you do not monitor will fail quietly, and you will find out during an outage.

What to Monitor

Signal	How to observe it	Why it matters
Oxidized service health	systemd status, REST API node list	Detects a stopped collector
Node failures	node_fail hook or the node status API	Finds unreachable devices
Push success	Syslog entries from the push script	Detects a stale remote
Webhook deliveries	GitHub delivery history, GitLab webhook event log	Detects receiver errors
Repository freshness	Last commit date on the remote	Catches silent breakage

A Simple Freshness Check

This command compares the age of the latest remote commit to a threshold, which you could run from any monitoring system.

bash

git ls-remote git@gitlab.example.com:netops/device-configs.git refs/heads/master

The command returns the current head commit of the remote branch. Compare it with the head of your local repository. A mismatch that persists beyond one polling interval means pushes are failing. The exact branch name must match your repository.

Routine Maintenance Tasks

1. Review failed nodes weekly and remove devices that no longer exist.
2. Rotate deploy keys and tokens on your policy schedule.
3. Update Oxidized and its gems regularly, and read the release notes, since model definitions and hook behavior change.
4. Test a restore by cloning the remote and applying a configuration to a lab device.
5. Periodically review the webhook delivery logs for repeated failures.

Automation Opportunities

Once the data flows, a lot becomes possible without much additional code.

Chat Notifications Straight From Oxidized

Oxidized has a hook for sending diffs to Slack, named slackdiff, and similar hooks exist for other messaging platforms. If your only goal is chat alerts, this may be simpler than a webhook receiver. The receiver approach wins when you also need verification, routing logic, or integrations with several systems, and when you want alerts to reflect what actually reached the remote repository.

Pipeline Gates

A GitLab CI job or a GitHub Actions workflow can run on every push to check configurations against policy. This example shows a minimal GitLab pipeline that fails if a forbidden string appears in any configuration file.

yaml

stages: - validate forbidden-strings: stage: validate image: alpine:3.20 script: - | if grep -R -n -E "snmp-server community (public|private)" .; then echo "Insecure SNMP community found" exit 1 fi

The job fails the pipeline when it finds a default SNMP community. Customize the pattern for your vendors’ syntax, and keep in mind that a simple grep catches only the obvious problems. A failed pipeline is a signal for review, since the change has already happened on the device by the time Oxidized records it.

Pull-Based Reconciliation

For a more advanced design, compare the committed configuration against an intended state generated by tools such as Ansible, Nornir, or a source of truth like NetBox. Differences point at configuration drift. Oxidized provides the “actual” side of that comparison.

Troubleshooting

When something breaks, work from the Oxidized side outward. Most issues fall into a few categories.

Troubleshooting Reference

Symptom	Likely cause	First action
No commits locally	Login failure or model mismatch	Check the Oxidized log and node status
Local commits but nothing on remote	Hook not firing or push failing	Check syslog and hook config
Permission denied on push	Deploy key lacks write access	Enable write permission on the key
Host key verification failed	Missing known_hosts entry	Add the verified host key
Non-fast-forward rejection	Remote has commits not in local	Investigate, do not force push
Webhook returns 401	Secret or token mismatch	Compare secrets on both sides
Webhook returns 5xx or times out	Receiver error or slow handler	Check receiver logs, respond faster
Duplicate or constant diffs	Volatile lines in the config	Filter lines in the model

Step-by-Step Diagnosis for “Nothing Reaches the Remote”

1. Confirm the local repository has recent commits with git log.
2. Confirm the hook is defined under hooks: with the event post_store, and that YAML indentation is correct.
3. Look for hook errors in the Oxidized log or syslog output.
4. Run the push script manually as the Oxidized user, since this reveals permission and key problems immediately.
5. Test the key with ssh -T git@gitlab.example.com using the same options as the script.
6. Check whether the remote branch is protected in a way that rejects the push.
7. Verify that the remote repository is not archived or read-only, which rejects pushes silently in some workflows.

Testing the Push Manually

Run the script as the service account to reproduce what Oxidized does. This is the fastest way to separate an Oxidized problem from a credentials problem.

bash

sudo -u oxidized env OX_NODE_NAME=test-node /usr/local/bin/oxidized-push.sh sudo -u oxidized tail -n 20 /var/log/syslog | grep oxidized-push

A successful run logs a pushed message. If you see a failure, the script’s error output and the SSH debug output (ssh -vT) normally point to the exact cause.

Webhook-Specific Troubleshooting

Use the delivery logs. GitHub’s recent deliveries view shows the request headers, payload, and the response your receiver returned. GitLab provides a similar event log with a test button and resend option under the webhook settings. If the platform shows a delivery but your receiver sees nothing, suspect the reverse proxy, firewall, or TLS certificate chain. A certificate that does not validate is the most common reason a webhook silently fails, because both platforms verify TLS by default.

Common Mistakes to Avoid

1. Pushing with a personal credential. If that person leaves, the pipeline breaks. Use deploy keys or service tokens.
2. Using force push to fix a conflict. This destroys the audit trail. Find out why histories diverged.
3. Skipping webhook verification. An unauthenticated endpoint is an open door to your automation.
4. Ignoring secrets in configs. Assume secrets exist until you have proven otherwise for each model.
5. Leaving volatile lines unfiltered. Constant noise makes people ignore alerts.
6. Creating the remote with a README. The first push fails on unrelated histories.
7. Not testing restores. A backup that has never been restored is an assumption.
8. Exposing the Oxidized REST API publicly. It is meant for local or tightly restricted access.
9. Using secure: false in production. Disabling SSH host key checks removes protection against man-in-the-middle attacks on device logins.
10. Treating the repository as proof of who made a change. It records observations, so attribute changes through AAA logs.

Best Practices Summary

Implementation Checklist

1. Confirm Oxidized backs up devices and commits locally.
2. Create a private remote repository without initial files.
3. Generate a dedicated deploy key and enable write access on that one repository.
4. Add the remote host key to known_hosts after verifying its fingerprint.
5. Choose the built-in hook or the exec script and configure post_store.
6. Run the push manually and confirm the remote updates.
7. Protect the default branch and disable force pushes.
8. Create the webhook with a secret and push events only.
9. Deploy the receiver over HTTPS with signature or token verification.
10. Add monitoring for push success and repository freshness.
11. Add secret scanning in CI and review model output for credentials.
12. Document the process and test a restore.

Decision Framework

Question	If yes	If no
Do you need only an off-box copy?	Built-in hook or mirror	Consider exec with a script
Do you need alerts and routing?	Add a webhook receiver	Skip inbound webhooks
Is the data regulated or sensitive?	Self-managed GitLab inside your network	Hosted platform can be acceptable
Do you have many groups?	Multiple repositories with a mapping	Single repository
Can you expose an HTTPS endpoint?	Webhook receiver	CI pipeline or polling

Expert Recommendations

A few judgments from running pipelines like this, offered as practitioner opinion rather than vendor guidance:

1. Start with one repository and one group. Most teams over-engineer the repository layout on day one and regret the key management overhead. Split later when access control demands it.
2. Prefer exec once the pipeline matters. The built-in hook is great for getting started. When pushes become something people rely on, the visibility and retry control of a script are worth the extra file.
3. Make the remote the shared interface, and keep the Oxidized host boring. The less the backup server does, the less there is to break and the less there is to secure.
4. Fix noisy models before you add alerts. If your first Slack notification is a timestamp change, your team will mute the channel and the project loses credibility.
5. Write down what the repository does not prove. Auditors respect precise claims. A repository shows configuration state over time. It does not show intent or authorship.

Future Trends

Network automation is moving toward treating configuration as a managed, reviewable artifact. Several directions are worth watching:

1. Intent-based and source-of-truth driven workflows. Tools such as NetBox and infrastructure-as-code frameworks increasingly define the desired state, with backups acting as the verification layer.
2. Policy as code in CI. More teams validate configurations automatically with pipeline checks before and after changes.
3. Streaming telemetry alongside backups. Telemetry covers operational state, while backups cover configuration. The two complement each other.
4. Stronger supply chain and secret controls. Expect hosted platforms to keep expanding built-in secret detection and token scoping, which suits configuration repositories well.

These are directional observations, and the specifics will depend on the tools and platforms you adopt.

Frequently Asked Questions

What is the difference between an Oxidized hook and a GitHub or GitLab webhook?

An Oxidized hook is an action that Oxidized runs when one of its own events fires, such as after a configuration is stored. A GitHub or GitLab webhook is an HTTP request that the Git platform sends to your URL when something happens in a repository, such as a push. In this design, the hook gets the data to the remote and the webhook tells your systems that new data arrived.

Which Oxidized event should I use to push configurations?

Use post_store for near-real-time pushes after each stored configuration. Use nodes_done if you prefer a single push at the end of each polling cycle to reduce load and noise.

Does the githubrepo hook work with GitLab?

The hook uses Git through the rugged library, so it is not tied to GitHub’s API and should work with any remote that accepts the configured credentials. Because behavior can vary by version, test it against your GitLab instance, and use an exec hook with git push if you want complete control.

Should I use SSH deploy keys or HTTPS tokens?

SSH deploy keys are simple and scoped to one repository, which makes them a good default. HTTPS with a fine-grained token is useful when only port 443 is allowed. In both cases, give the credential the smallest permission set that allows pushes.

How do I verify a GitHub webhook?

Compute an HMAC-SHA256 of the raw request body using the webhook secret, prefix the hex digest with sha256=, and compare it with the X-Hub-Signature-256 header using a constant-time comparison. Reject the request if they do not match.

How do I verify a GitLab webhook?

Set a secret token on the webhook and compare it with the X-Gitlab-Token header on each incoming request, again using a constant-time comparison.

Is it safe to push network configurations to GitHub or GitLab.com?

It can be, but it is a risk decision. Configurations may contain sensitive material, so use private repositories, strip secrets, scan commits, and apply least-privilege credentials. For regulated environments, a self-managed GitLab instance inside your own network is often the better choice.

Why does Oxidized create a commit on every run?

Usually because the device includes a changing value in its output, such as a timestamp, uptime, or an auto-generated hash. Filter those lines in the model definition so that commits represent real changes.

Can I get alerts without writing a receiver?

Yes. Oxidized has hooks such as slackdiff for sending diffs to chat directly. A receiver becomes worthwhile when you need verification, routing, or integration with several systems.

How often should Oxidized poll devices?

A one-hour interval is a common starting point. Shorten it only if you have a real need, and watch for account lockouts or device load when you do.

Key Takeaways

1. A remote Git repository gives your Oxidized backups off-box durability, team visibility, and event-driven automation.
2. Use the post_store event to push after each stored configuration, or nodes_done to push once per cycle.
3. The built-in githubrepo hook is the fastest start, and an exec hook with a script gives you better control and logging.
4. Use per-repository deploy keys or fine-grained tokens, and protect the default branch against force pushes.
5. Always verify webhooks, with X-Hub-Signature-256 on GitHub and X-Gitlab-Token on GitLab.
6. Treat configuration repositories as sensitive, because secrets can slip through even with remove_secret enabled.
7. Monitor push success and repository freshness so that failures do not stay silent.
8. Remember that the repository records what changed, while attributing who changed it needs AAA or change management records.

Conclusion

Oxidized already does the hard part of network configuration backup, which is logging in to many kinds of devices and capturing their configuration reliably. Connecting it to GitLab or GitHub turns that history into something the whole organization can use: an off-box archive, a review interface, an audit trail, and a source of events for automation.

For next steps, get one hook pushing to one private repository and confirm the result by hand. Then add the webhook and a verified receiver, send the first notification to a channel, and fix any noisy models before widening the rollout. Finish with a restore test, because that is the moment the pipeline proves its value.