Oxidized Palo Alto PAN-OS Backup: Handling Candidate vs Running Configs

Palo Alto Networks firewalls behave differently from most network devices when it comes to configuration. On a Cisco router, typing a command changes the live device immediately. On PAN-OS, typing a command changes only a staging copy, and nothing affects traffic until someone runs a commit. That two-stage design is excellent for safety, yet it creates a subtle trap for anyone building automated backups: which version of the configuration are you actually saving?

If your backup tool captures the candidate configuration, your Git history will fill with half-finished edits, abandoned experiments, and changes that never went live. If it captures the running configuration, you get an accurate record of what the firewall enforces, but you lose sight of uncommitted work. Oxidized can handle either approach, and in practice the best answer is often both. This guide explains the distinction in depth, shows how to build a least-privilege backup account, deploys Oxidized with Docker, provides custom models for running and candidate captures, adds XML API exports for restore-grade files, and covers syslog triggers, Git workflows, restores, and troubleshooting.

What Is Oxidized for Palo Alto PAN-OS?

Oxidized for Palo Alto PAN-OS is an automated backup workflow in which the Oxidized application connects to a firewall or Panorama over SSH, runs CLI commands that display the configuration, and commits each changed result to a Git repository. Depending on the model and commands used, it can capture the running configuration, the candidate configuration, or both, giving teams a versioned, searchable record of firewall policy changes.

Key Takeaways

• The running configuration is what the firewall enforces, while the candidate configuration is a shared staging copy that becomes running only after a commit.
• For audit and rollback, back up the running configuration. For spotting uncommitted or abandoned changes, capture the candidate as a separate node.
• Verify what your installed Oxidized model actually retrieves, because behavior can differ between versions. A simple test with an uncommitted change settles the question.
• Set-format output is ideal for readable diffs, while XML is required for a restorable file. Keep both if restores matter.
• Configuration values encrypted with the device master key cannot be restored elsewhere without that key, so protect and document it separately.
• A commit-triggered syslog hook captures every deployment within seconds.

Candidate vs Running Configuration in PAN-OS

Before touching any tooling, it is worth getting the vocabulary exactly right, because the whole article turns on it.

Definitions

The running configuration is the active configuration currently loaded and enforced by the firewall. It changes only when a commit succeeds. The candidate configuration is the working copy that administrators edit through the web interface, CLI, or API. Importantly, the candidate is a shared working copy, so uncommitted changes from several administrators can coexist in it until they are committed, partially committed, or reverted.

Characteristic	Running Configuration	Candidate Configuration
Purpose	What the firewall enforces right now	Staging area for pending edits
Changes when	A commit completes	Any administrator edits an object or setting
Visible to traffic	Yes	No
Reliability as a record	High, reflects deployed state	Variable, may contain unfinished work
Typical backup value	Audit, rollback, disaster recovery	Drift detection, change review
Persistence	Saved to disk on commit	Held until commit, revert, or restart behavior applies

The Commit Lifecycle

The following sequence explains where each version lives and why timing affects what a backup sees.

1. An administrator edits an object, rule, or setting, which modifies only the candidate configuration.
2. The change is visible in the candidate immediately, but the running configuration is untouched.
3. The administrator starts a commit, and PAN-OS validates the candidate for errors and warnings.
4. If validation passes, the firewall applies the candidate and saves it as the new running configuration.
5. A configuration version is recorded, so earlier running configurations can be reloaded later.
6. Config and system logs record who committed and what the result was.

Why the Distinction Matters for Backups

A backup captured at two in the afternoon during a change window can contain a policy that was edited but never committed. In contrast, a backup of the running configuration contains only deployed policy. Consequently, a candidate-based history is noisy and potentially misleading in an audit, since a reviewer might assume everything in it was enforced. On the other hand, a running-only history cannot reveal that an engineer left twenty uncommitted edits sitting in the candidate for a week.

Which Configuration Should You Back Up?

The right choice depends on the question you want your history to answer.

Goal	Recommended Capture	Reason
Audit trail of enforced policy	Running configuration	Matches what actually passed traffic
Rollback and disaster recovery	Running configuration in XML	Restorable and consistent
Detect abandoned or unapproved edits	Candidate configuration, separate node	Reveals uncommitted changes
Change review before commit	Candidate configuration	Shows pending work
Compliance evidence	Running configuration with commit logs	Pairs deployed state with who and when

For most organizations, the running configuration is the primary backup. A second, lower-priority candidate node is a useful extra for teams that want to catch drift between edits and commits.

How Oxidized Works with PAN-OS

Oxidized is built from sources, inputs, models, and outputs. For PAN-OS, the model is the piece that decides which configuration you receive.

Architecture Components

Layer	Component	Function in a PAN-OS Deployment
Source	CSV, SQL, HTTP, or NetBox	Supplies firewall and Panorama addresses and model names
Input	SSH	Opens the PAN-OS CLI session
Model	panos or a custom model	Defines the commands and output cleanup
Output	Git or gitcrypt	Stores each changed configuration as a commit
Hooks	githubrepo, exec, slackdiff	Reacts to stored changes
Interface	REST API and Oxidized Web	Provides node status and manual triggers

The Backup Lifecycle

1. Oxidized reads the node list from the configured source.
2. A worker opens an SSH session to the firewall.
3. The model disables the pager and selects an output format.
4. The model runs its commands, such as a configuration display.
5. Volatile lines are removed from the output.
6. Oxidized compares the result with the last stored version.
7. A changed result is written to Git, and any hook runs.

Verify What Your Model Retrieves

Oxidized ships with a panos model, but the commands it runs have evolved across releases, and some versions display the configuration from configuration mode, which reflects the candidate. Rather than trusting memory or documentation alone, test it on a lab firewall.

1. Run a backup of the lab firewall and note the stored configuration.
2. On the firewall, make a harmless uncommitted change, such as editing an object description, and do not commit it.
3. Trigger a second backup through the API.
4. If the new description appears in the Git diff, your model is capturing the candidate configuration.
5. If nothing changes until you commit, the model is capturing the running configuration.

This five-step check takes ten minutes and removes all ambiguity. Afterward, the custom models later in this guide give you explicit control regardless of how the bundled model behaves.

Prerequisites and Planning

Requirement	Recommendation	Why It Matters
Backup server	Linux VM with 2 vCPU and 2 GB RAM	Light workload, Git history grows steadily
Container runtime	Docker Engine with Compose	Simple upgrades and rollback
PAN-OS access	SSH reachable on the management interface	Oxidized uses the CLI
Service account	Dedicated administrator with a restricted role	Limits impact of a leaked credential
Permitted IPs	Oxidized server only	Blocks use of the credential elsewhere
Time sync	NTP on all systems	Reliable commit and log timestamps
Remote Git	Private repository on GitLab, GitHub, or Gitea	Protects history if the server fails
Master key record	Documented and stored securely	Needed for certain restores

Firewalls, Panorama, or Both

In a Panorama-managed environment, decide what to back up. Each firewall holds its own running configuration, which includes the policy pushed from Panorama. Panorama holds the templates, device groups, and shared objects that generate those pushes. Backing up both is ideal: Panorama shows where a change originated, while the firewalls prove what was actually deployed.

Management Path

Use the dedicated management interface or a management-profile-enabled interface reachable from the Oxidized server. In addition, check the service route configuration, because the source address the firewall uses for syslog, which matters later, depends on those settings.

Preparing PAN-OS for Oxidized

The objective is an administrator who can log in over SSH and read the configuration, with no ability to change policy or commit.

Create an Admin Role and User

The web interface is the most reliable way to build the account, because menu names are stable across releases.

1. Go to Device, then Admin Roles, and create a custom role named oxidized-ro.
2. On the Web UI tab, leave features disabled or read-only, since Oxidized does not use the interface.
3. On the XML API tab, enable only the configuration and export permissions you need, or leave them off if you will use SSH only.
4. On the Command Line tab, choose the read-only option that your role type offers.
5. Go to Device, then Administrators, and add a user named oxidized assigned to that role.
6. Choose public key authentication and paste the Oxidized server’s public key, or set a long random password.
7. Commit the change, because administrator accounts do not take effect until the commit completes.

If you prefer the CLI, the built-in superreader role gives read-only access to the full configuration.

configure set mgt-config users oxidized permissions role-based superreader yes set mgt-config users oxidized public-key <base64-encoded-public-key> commit

Public key formats and available role names differ across PAN-OS versions, so confirm the syntax with tab completion on your release. After committing, test the account from the Oxidized server with ssh oxidized@10.10.1.5 and run show config running to confirm it can read the configuration.

Restrict Management Access

A backup account is only as safe as the paths that reach it. Limit SSH to trusted sources in two places. First, in the management interface settings under Device, Setup, Management, add the Oxidized server to the permitted IP addresses. Second, if you reach the firewall through a data-plane interface, attach a management profile that allows SSH and lists the Oxidized server as a permitted address.

High Availability Pairs

In an active-passive pair, the peers synchronize configuration, yet each retains its own management address and some device-specific settings. Add each peer’s management address as a separate Oxidized node. As a result, you confirm that both units are reachable, and you capture local differences that a cluster address would hide.

Panorama-Managed Firewalls

A firewall managed by Panorama shows pushed policy in its own running configuration. Therefore, a commit on Panorama followed by a push generates a commit on each firewall, and each of those produces a new running configuration. Expect a burst of Git commits across the fleet after every push, which is normal and useful evidence of the rollout.

Installing Oxidized with Docker

Directory Layout

sudo mkdir -p /srv/oxidized/ssh /srv/oxidized/model sudo chown -R 30000:30000 /srv/oxidized

The model subdirectory is where custom models live. Because the official image runs as an unprivileged user, ownership matters, and the private key should be readable only by that user.

The Core Configuration File

Save this as /srv/oxidized/config.

username: oxidized password: ChangeThisLongRandomPassword model: panos interval: 3600 use_syslog: false debug: false threads: 10 timeout: 60 retries: 3 rest: 127.0.0.1:8888 vars: ssh_keys: /home/oxidized/.config/oxidized/ssh/id_rsa input: default: ssh ssh: secure: false output: default: git git: user: Oxidized email: oxidized@example.com repo: /home/oxidized/.config/oxidized/paloalto.git source: default: csv csv: file: /home/oxidized/.config/oxidized/router.db delimiter: !ruby/regexp /:/ map: name: 0 ip: 1 model: 2 group: 3

The timeout is deliberately generous at 60 seconds, since large PAN-OS configurations with thousands of rules take a while to render. Furthermore, the thread count is modest because firewalls handle management sessions on a constrained control plane, so flooding them is unwise. Switch secure to true in production once known hosts are populated.

The Device Inventory

fw-pa-edge-01:10.10.1.5:panos:running fw-pa-edge-02:10.10.1.6:panos:running panorama-01:10.10.1.20:panos:panorama

At this stage, the bundled panos model is used. After the verification test described earlier, you can replace the model column with a custom one.

Start the Container

docker run -d --name oxidized \ --restart unless-stopped \ -p 127.0.0.1:8888:8888 \ -v /srv/oxidized:/home/oxidized/.config/oxidized \ oxidized/oxidized:latest oxidized

Watch the first run with docker logs -f oxidized, then confirm with curl -s http://127.0.0.1:8888/nodes.json and git -C /srv/oxidized/paloalto.git log --oneline.

Custom Models for Running and Candidate Captures

Because explicit control beats assumptions, write two small models. One always captures the running configuration, and the other always captures the candidate. Oxidized loads custom models from the model directory, and the model name is the lowercase form of the class name.

A Running Configuration Model

Save this as /srv/oxidized/model/panosrunning.rb. It disables paging, switches the CLI to set-format output for readable diffs, prints the system information as comments with volatile lines removed, and then displays the running configuration.

class PanOSRunning < Oxidized::Model comment '# ' prompt /^[\w.@()-]+[>#]\s?$/ cmd :all do |cfg| cfg.each_line.to_a[1..-2].join end cmd 'show system info' do |cfg| cfg.gsub!(/^(uptime|time|app-version|app-release-date|av-version|av-release-date|threat-version|threat-release-date|wildfire-version|wildfire-release-date|url-filtering-version): .*\n/, '') comment cfg end cmd 'show config running' do |cfg| cfg end cfg :ssh do post_login 'set cli pager off' post_login 'set cli config-output-format set' pre_logout 'exit' end end

The removed fields, such as uptime and content update versions, change constantly and would otherwise create a commit every hour. Meanwhile, the model name for the inventory becomes panosrunning.

A Candidate Configuration Model

The candidate model enters configuration mode and displays the configuration from there, which reflects the candidate. Save it as /srv/oxidized/model/panoscandidate.rb.

class PanOSCandidate < Oxidized::Model comment '# ' prompt /^[\w.@()-]+[>#]\s?$/ cmd :all do |cfg| cfg.each_line.to_a[1..-2].join end cmd 'show' do |cfg| cfg end cfg :ssh do post_login 'set cli pager off' post_login 'set cli config-output-format set' post_login 'configure' pre_logout 'exit' pre_logout 'exit' end end

Entering configuration mode does not commit anything, and a read-only role cannot change policy. Nevertheless, test this model first on a lab unit, because role restrictions on configuration mode vary by version. Models are plain Ruby, so treat them like code: keep them in version control and review changes.

Using the Models in the Inventory

After adding the files, restart the container so Oxidized loads them. Then list the same firewall twice with different node names.

fw-pa-edge-01:10.10.1.5:panosrunning:running fw-pa-edge-01-cand:10.10.1.5:panoscandidate:candidate fw-pa-edge-02:10.10.1.6:panosrunning:running

Git stores each node as a separate file, so running and candidate histories stay cleanly separated. In addition, the group column places them in different directories, which makes later reporting simpler.

Git Versioning for Firewall Policy

Why Git Fits PAN-OS

Set-format output prints one command per line, for example a security rule’s source, destination, application, and action each on its own line. Consequently, adding a rule or changing its action appears as a small, precise diff, even in a configuration with thousands of lines. Git adds integrity, compact storage, and tools that auditors recognize.

Reading the History

git clone /srv/oxidized/paloalto.git ~/paloalto-configs cd ~/paloalto-configs # Last five changes to one firewall git log -5 --stat -- running/fw-pa-edge-01 # What changed in the latest commit git show HEAD -- running/fw-pa-edge-01 # Compare two points in time git diff HEAD~10 HEAD -- running/fw-pa-edge-01

Because the group column places each node under its group directory, the paths above include the group name. Commit messages record the node, so git log --grep quickly finds every change to a given firewall.

Push to a Remote Repository

hooks: push_to_remote: type: githubrepo events: [post_store] remote_repo: git@git.example.com:netops/paloalto-configs.git publickey: /home/oxidized/.config/oxidized/ssh/deploy.pub privatekey: /home/oxidized/.config/oxidized/ssh/deploy

This hook pushes to any Git server reachable over SSH. Use a repository-specific deploy key and protect the main branch so history cannot be rewritten.

Detecting Uncommitted Drift Between Candidate and Running

Having both histories in Git creates a capability that PAN-OS itself makes awkward: a record of work that was started but never deployed. Once the running node and the candidate node are both being collected, the comparison becomes a simple review habit.

A Practical Drift Workflow

1. Collect the running node hourly and the candidate node on the same schedule.
2. After each cycle, compare the latest candidate file with the latest running file.
3. If the files are identical, no uncommitted work exists.
4. If they differ, list the differing lines and identify the administrator through the config log.
5. Contact the owner, then either commit the change through your normal approval process or revert the candidate.

cd ~/paloalto-configs git pull --quiet diff <(grep -v '^#' running/fw-pa-edge-01) <(grep -v '^#' candidate/fw-pa-edge-01-cand) | head -40

The comment filter removes the system information header, which exists only in the running model, so the comparison focuses on configuration lines. In practice, this check catches forgotten edits, half-completed change tickets, and the occasional unauthorized experiment. Moreover, it protects against a quieter risk: the next administrator to commit may unknowingly deploy someone else’s pending work, because the candidate is shared.

PAN-OS also provides its own tools for inspecting pending changes, including a config audit in the web interface and CLI commands that compare candidate and running versions. Those tools are excellent for a human at a console, whereas the Git-based approach adds history and automation across an entire fleet.

Taming Noise in PAN-OS Backups

A useful history depends on commits that mean something. Fortunately, PAN-OS noise comes from a short list of predictable sources.

Noise Source	What Happens	Remedy
System information fields	Uptime, time, and content versions change constantly	Strip those lines in the model, as shown earlier
Content updates	Application and threat versions advance on schedule	Exclude from the captured text
Panorama pushes	Every firewall commits after a push	Accept as legitimate evidence of rollout
Candidate churn	Edits appear and disappear as administrators work	Keep candidate in a separate, lower-priority history
Output format changes	Switching between set and XML rewrites every line	Choose one format per node and keep it
Pager or prompt artifacts	Stray lines enter the output	Verify the model’s login steps and prompt pattern

After any change to a model, run two consecutive backups of a quiet lab firewall. If the second run produces a commit, noise remains. Repeat until two consecutive runs produce identical files, and only then deploy the model widely.

Choosing Between Set Format and XML

Format	Strengths	Limitations
Set format	One command per line, very readable diffs, easy to paste for partial rollbacks	Cannot be loaded as a full configuration file
XML	Native format, supports full configuration import	Verbose, diffs are harder to read

Since each format excels at a different job, the strongest design uses set format in Oxidized for review and XML from the API for restore. The next section shows how to add the XML side.

Adding XML API Exports for Restore-Grade Files

PAN-OS exposes an XML API that can export the running configuration as a complete file suitable for import. Combining that export with Git gives you a restorable archive alongside the readable Oxidized history.

Understanding the API Verbs

The XML API distinguishes between configuration states in a way that mirrors the candidate and running split. A configuration request with the show action returns the active, running configuration, whereas the get action returns the candidate configuration. The export request, meanwhile, downloads the running configuration file.

API Request	Returns
type=export with category=configuration	The running configuration file in XML
type=config with action=show	The active (running) configuration
type=config with action=get	The candidate configuration
type=keygen	An API key for the requesting administrator

A Scheduled Export Script

Generate an API key for a dedicated administrator, store it in an environment file with restricted permissions, and run a script from cron or a systemd timer. Validate the firewall certificate with your internal CA rather than disabling verification.

#!/bin/bash set -euo pipefail FW="fw-pa-edge-01.example.com" DIR="/srv/panos-xml" mkdir -p "$DIR" curl -s --cacert /etc/ssl/certs/corp-ca.pem \ "https://${FW}/api/?type=export&category=configuration&key=${PANOS_API_KEY}" \ -o "${DIR}/${FW}.xml" cd "$DIR" git add -A git diff --cached --quiet || git commit -m "Export ${FW} $(date -u +%FT%TZ)"

This script downloads the running configuration, then commits it only when the content has changed. As a result, the XML repository mirrors the behavior of the Oxidized repository, and both can be pushed to a remote. For many firewalls, loop over an inventory file, and store API keys in a secrets manager instead of plain files.

PAN-OS also supports scheduled configuration export to SCP or FTP servers from Panorama and, on newer releases, directly from firewalls. If that feature is available in your version, it is a reasonable alternative to a custom script, since it needs no external scheduler.

Real-Time Backups with Syslog Triggers on Commit

Because the running configuration changes only at commit, commits are the natural trigger. A syslog-driven hook captures each deployment within seconds.

The Trigger Sequence

1. An administrator or Panorama push commits the configuration.
2. PAN-OS writes a config log entry and a system log entry for the commit.
3. A syslog server profile forwards the log to the Oxidized host.
4. rsyslog passes the sender address and message to a script.
5. The script recognizes a successful commit and calls the Oxidized API.
6. Oxidized fetches the running configuration and commits the diff to Git.

Configure the Firewall

In the web interface, create a syslog server profile under Device, Server Profiles, Syslog, pointing at the Oxidized host on UDP or TCP 514. Then create a log forwarding configuration or a Device, Log Settings entry that sends config and system logs to that profile, and commit. Remember that the source address of the syslog traffic depends on the service route settings, so confirm which interface and address the firewall uses.

Configure rsyslog and the Script

template(name="ipmsg" type="string" string="%fromhost-ip% %msg%\n") module(load="imudp") input(type="imudp" port="514" ruleset="pan") ruleset(name="pan") { action(type="omprog" binary="/usr/local/bin/oxidized-trigger.sh" template="ipmsg") }

#!/bin/bash while read -r ip msg; do case "$msg" in *CONFIG*commit*Succeeded*|*CONFIG*commit*succeeded*) curl -s "http://127.0.0.1:8888/node/next/${ip}" > /dev/null ;; esac done

Log field layouts and wording differ across PAN-OS releases and custom log formats, so capture sample messages from your own firewalls and adjust the match patterns. Note also that Oxidized resolves the address to the first node with that IP. Since the running node is listed before the candidate node in the inventory, commits trigger the running capture, which is exactly what you want. Finally, restrict the listener to firewall addresses, and consider a short debounce so that a Panorama push to many devices does not queue redundant work.

Using the Oxidized REST API and Web Interface

Endpoint	Purpose
/nodes	Lists nodes with status and last run time
/nodes.json	Same data in JSON for automation
/node/fetch/NAME	Returns the latest stored configuration
/node/next/NAME	Queues an immediate backup
/node/version?node_full=GROUP/NAME	Lists stored versions
/reload	Reloads the inventory

The JSON node list is a convenient monitoring source. An alert on any node that stays failed for more than two intervals catches expired credentials, changed permitted IP lists, and firewalls that have quietly become unreachable.

Securing the Backup Pipeline

Firewall configurations are among the most sensitive documents a network team owns. They reveal rule bases, address plans, VPN peers, and, in encrypted form, secrets.

Control	Recommendation
Service account	Restricted role, key authentication, permitted IP limited to Oxidized
SSH settings	Use a strong SSH service profile on supported releases and keep Oxidized current
API exposure	Bind to localhost, publish only through an authenticated TLS proxy
Repository	Private remote, deploy key, protected main branch
Encryption at rest	Use gitcrypt or disk encryption on the backup server
API keys	Store in a secrets manager and rotate on a schedule
Master key	Record and escrow separately, never in the same repository
Monitoring	Alert on failed nodes and unexpected commits

The Master Key Matters

PAN-OS encrypts sensitive values in the configuration, such as private keys and shared secrets, using a device master key. Consequently, a configuration backup contains encrypted blobs that only a firewall holding the same master key can decrypt. If you restore such a file to a replacement firewall with a different master key, those values will fail to decrypt, and the commit may fail or the affected features may break. Therefore, document each firewall’s master key configuration, store the key securely outside Git, and, where policy allows, use a common master key across a device group so replacements are straightforward.

Restoring a PAN-OS Firewall from a Backup

Restore procedures vary with the file you hold. A set-format file is a review and partial-rollback tool, while an XML file supports a full restore.

Full Restore from an XML File

1. Confirm the target runs a compatible PAN-OS version and has the correct master key.
2. Obtain the XML export from your archive and verify its commit time.
3. Import it through Device, Setup, Operations, Import named configuration snapshot, or by SCP from the CLI.
4. Load the named configuration into the candidate.
5. Review the candidate for expected differences, such as interface assignments.
6. Commit and verify traffic, routing, VPN tunnels, and management access.

scp import configuration from backupuser@10.10.50.25:/srv/panos-xml/fw-pa-edge-01.xml configure load config from fw-pa-edge-01.xml # Review the candidate before committing commit

Syntax and prompts differ slightly by release, so use tab completion to confirm each command. The key insight is that loading a file affects only the candidate. Nothing changes on the wire until you commit, which gives you a chance to inspect the result.

Partial Rollback from Set-Format History

When only a rule or object needs to return to its earlier state, copy the relevant set lines from Git, enter configuration mode, paste them, review the candidate, and commit. This approach is faster and safer than a full restore for single-policy mistakes.

Restore Pitfalls

Several problems recur. Master key mismatches break encrypted values. Hardware differences change interface names and zones. Dynamic content, including application and threat databases, is not part of the configuration and must be updated separately. Licenses and certificates may need reapplying. Finally, the target firewall must be able to commit cleanly, so check for commit locks and pending jobs before starting.

Change Auditing and Compliance Evidence

Commit history in Git, together with config logs that identify the administrator, creates a complete change trail.

Framework	Relevant Area	How the Workflow Helps
NIST SP 800-53	CM-2, CM-3, CM-6	Baselines, tracked changes, documented settings
CIS Controls	Secure configuration of network devices	Versioned, reviewable firewall policy
ISO 27001:2022	Configuration management	Retrievable configuration history
PCI DSS 4.0	Network security control reviews and change approval	Diffs that simplify periodic review and evidence of approved changes
SOC 2	Change management criteria	Timestamped records of production changes

Because the running history reflects only deployed policy, it is the right artifact to hand an assessor. Pair each commit with its change ticket, and use the candidate history internally to demonstrate that unapproved edits are detected quickly.

Scaling Across Many Firewalls and Panorama

Groups and Credentials

Use groups to separate datacenter, branch, and customer firewalls, and to apply different credentials where needed. For service providers, this prevents one customer’s credential from reaching another’s device. Git stores each group in its own directory.

Source of Truth

A CSV file suffices for a pilot. At larger scale, generate the inventory from NetBox or an HTTP endpoint so that new firewalls enter the backup rotation automatically, and retired ones leave it.

Timing and Load

PAN-OS management planes are deliberately modest, so keep thread counts conservative and timeouts generous. Meanwhile, stagger polling by group if hundreds of firewalls share a management network, and rely on syslog triggers for timely capture instead of shortening the interval.

Troubleshooting PAN-OS Backups in Oxidized

Symptom	Likely Cause	Fix
Node shows no_connection	Permitted IP list or management profile blocks SSH	Test with ssh from the Oxidized host and adjust access
Authentication failed	Wrong credential, missing commit after creating the user, or key mismatch	Verify the login manually and confirm the admin commit completed
Unauthorized or invalid command	Read-only role cannot run a command in the model	Test the command manually and adjust the role or model
Output contains pager prompts	Pager not disabled	Confirm post_login commands and model loading
Prompt mismatch or hang	Prompt pattern misses the configuration-mode prompt	Allow both operational and configuration prompts in the pattern
Key exchange or cipher error	SSH service profile stricter than the Ruby library supports	Update the Oxidized image and review the profile
Commit on every run	Volatile lines in output	Remove system info fields and verify two runs match
Candidate model shows others’ changes	Candidate is shared by all administrators	Expected behavior, use for drift detection
Timeout on large configuration	Very large rule base or slow management plane	Increase timeout and reduce threads
Custom model not found	File name, class name, or ownership issue	Match the lowercase class name and check file permissions

Debugging a Single Firewall

Isolate the failing node, enable debug temporarily, and restart the container. Then run the same commands manually over SSH with the same account and compare the transcripts. If the manual session works and Oxidized does not, the cause is almost always the prompt pattern or a command the role is not allowed to run.

After a PAN-OS Upgrade

Major upgrades can change banners, SSH algorithms, or CLI output. When backups start failing afterward, update the Oxidized image, retest your custom models, and compare a manual session with the model’s expectations. If the bundled model is at fault, report it to the project with the PAN-OS version and a sanitized debug log.

Common Mistakes to Avoid

• Assuming the bundled model captures the running configuration without testing.
• Treating a candidate-based history as proof of what was deployed.
• Forgetting to commit after creating the backup administrator.
• Using a full superuser account for backups.
• Skipping the permitted IP restriction on the backup account.
• Keeping only set-format output and then discovering it cannot be loaded as a full restore.
• Ignoring the master key until a replacement firewall refuses to commit.
• Letting volatile system fields create a commit every hour.
• Exposing the Oxidized REST interface without authentication.
• Never rehearsing a restore on a lab firewall.

Oxidized Compared with Other PAN-OS Backup Options

Method	Cost	Version Control	Multi-Vendor	Best Fit
Oxidized	Free	Native Git	Yes	Teams wanting history, diffs, and an API
Panorama config versions	Included with Panorama	Built-in revisions	Palo Alto only	Central management and rollback
Native scheduled export	Included	File-based	Palo Alto only	Simple off-box copies
Custom API scripts	Free	You build it	Palo Alto only	Restore-grade XML archives
Ansible with PAN-OS collections	Free	You build it	Yes	Teams already invested in Ansible
Commercial NCM suites	Licensed	Yes	Yes	Organizations wanting vendor support

In practice, these options complement one another. Panorama manages deployment, API exports give you restorable XML, and Oxidized provides a readable, multi-vendor history that sits alongside your routers, switches, and other firewalls.

Real-World Scenarios

Scenario One: Finding the Rule That Broke an Application

An application team reports that a service stopped working shortly after lunch. The security engineer reviews the running history for the firewall and finds a commit twenty minutes earlier that moved a deny rule above an allow rule. The diff shows the exact lines, the config log names the administrator, and the fix is a single reorder rather than an afternoon of packet captures.

Scenario Two: Catching Forgotten Candidate Changes

During a weekly review, the drift comparison shows eight uncommitted rule edits on a datacenter firewall. The owner had started a change two weeks earlier and was reassigned. Because the team caught it, nobody accidentally deploys stale work during the next unrelated commit, and the change is either completed properly or reverted.

Scenario Three: Replacing a Failed Firewall

A branch firewall suffers a hardware fault. The replacement is staged with the same PAN-OS version and the documented master key, then loaded with the latest XML export. After reviewing the candidate for interface differences, the engineer commits, and the branch returns to service in under an hour.

Optimization Recommendations

Area	Recommendation	Benefit
Capture strategy	Running node for audit, candidate node for drift	Accurate history plus early warning
Formats	Set format in Oxidized, XML from the API	Readable diffs and restorable files
Polling	Hourly interval plus commit-triggered syslog	Fresh data, low load
Models	Keep custom models in version control	Reviewable, reproducible behavior
Repository	Off-site mirror and periodic garbage collection	Resilience and smaller storage
Secrets	Escrow the master key separately	Reliable restores
Alerting	Notify on failed nodes and on commits to critical firewalls	Faster response
Documentation	README with restore steps and version notes	Faster recovery under pressure

Frequently Asked Questions

Does Oxidized support Palo Alto PAN-OS?

Yes. Oxidized includes a panos model that connects over SSH and stores the configuration in Git or another output. You can also write custom models for finer control.

Does Oxidized back up the candidate or the running configuration?

It depends on the model and version. Test by making an uncommitted change and checking whether it appears in the next backup. Custom models let you capture either one explicitly.

What is the difference between candidate and running configuration in PAN-OS?

The running configuration is what the firewall enforces. The candidate is a shared staging copy of pending changes that becomes running only after a successful commit.

Which configuration should I back up for disaster recovery?

Back up the running configuration, preferably as an XML file, because it reflects deployed policy and can be loaded onto a replacement firewall.

Can I restore a PAN-OS firewall from Oxidized output?

Set-format output supports partial rollbacks by pasting commands. A full restore requires an XML configuration file, which you can collect through the XML API alongside Oxidized.

What permissions does the Oxidized account need?

A read-only role with SSH access is the goal, restricted by permitted IP address. Test the exact commands your model runs, since role capabilities vary by version.

Does Oxidized work with Panorama?

Yes. Add Panorama as a node. Backing up both Panorama and the managed firewalls shows where changes originate and what was actually deployed.

Why does my backup change every hour?

Volatile system information such as uptime and content versions is usually the cause. Strip those fields in the model and confirm that two consecutive runs match.

How do I trigger a backup after every commit?

Forward config and system logs to a syslog receiver, and use a script that calls the Oxidized next-node endpoint when it sees a successful commit.

Is the master key stored in the backup?

No. The configuration contains values encrypted with the master key, but the key itself must be recorded and stored separately for restores to succeed.

Conclusion

Backing up Palo Alto firewalls with Oxidized comes down to one idea: know exactly which configuration you are saving. The running configuration is the authoritative record of what was deployed, the candidate configuration reveals unfinished work, and a well-designed pipeline captures each deliberately. Furthermore, pairing readable set-format history from Oxidized with restorable XML from the API gives you both review and recovery, while commit-triggered syslog hooks keep the repository current within seconds.

The practical next step is small. Test your current model with an uncommitted change on a lab firewall, add the running model for your production nodes, and rehearse one restore. After that, introduce the candidate node for drift detection and the XML export for disaster recovery. Together, those pieces turn firewall change control from a matter of trust into a matter of evidence.