Oxidized for F5 BIG-IP and Load Balancers: Backing Up UCS and Config Files

Load balancers occupy an uncomfortable position in most networks. They sit in front of every important application, they accumulate years of virtual servers, pools, iRules, certificates, and profiles, and they are often configured by application teams as much as by network engineers. When one fails, or when a well-meaning change breaks a checkout page at the worst possible moment, the question is always the same: what did the configuration look like an hour ago, and how fast can we get it back?

F5 BIG-IP answers that question in several ways, none of which is a complete solution on its own. A UCS archive is the gold standard for full recovery, yet it is binary, large, and impossible to diff. The text configuration files are readable and versionable, but they reflect only what has been saved. Oxidized fills the gap by collecting readable configuration on a schedule and committing it to Git, while a separate, scheduled UCS process protects everything else. This guide explains how to build both layers, how to handle the running-versus-saved distinction that trips up many teams, how to write a custom Oxidized model when the bundled one is not enough, and how to restore a failed unit. Later sections extend the same pipeline to other load balancers.

What Is Oxidized for F5 BIG-IP?

Oxidized for F5 BIG-IP is an automated backup workflow in which the Oxidized application connects to BIG-IP systems over SSH, runs tmsh or shell commands that display the configuration, and commits each changed result to a Git repository. It records a readable, versioned history of virtual servers, pools, profiles, and system settings, and it complements, rather than replaces, UCS archives that provide full-fidelity disaster recovery.

What Is a UCS File?

A UCS file is an F5 BIG-IP user configuration set, a compressed archive created with the tmsh save sys ucs command. It contains the system configuration files, the license, SSL certificates and keys, the master key, and other data needed to restore a BIG-IP system to a previous state. Because it is binary and often large, a UCS is stored as a file rather than diffed in Git.

Key Takeaways

• A complete F5 backup strategy uses two layers: readable configuration text in Git for history and review, and encrypted UCS archives for full disaster recovery.
• BIG-IP distinguishes running configuration, held in memory, from saved configuration, written to files. A backup that reads files misses unsaved tmsh changes, while a backup that reads the running configuration shows them.
• Oxidized includes an f5 model, but the exact commands it runs differ between releases, so test what it captures before trusting it.
• UCS archives contain private keys and the master key, so encrypt them with a passphrase and store them outside Git.
• A UCS can generally be restored only on a compatible software version, and restoring to different hardware may require extra options.
• Audit logging forwarded by syslog lets you trigger a capture within seconds of a change.
• The same pipeline extends to other load balancers such as Citrix NetScaler, which also have running and saved configuration states.

Understanding F5 BIG-IP Backup Artifacts

Before building anything, it helps to be precise about what each artifact contains, because teams frequently assume one covers the others.

The Main Backup Artifacts

Artifact	Format	Contains	Diff-Friendly	Best Use
UCS archive	Compressed binary	Configuration, license, certificates, keys, master key	No	Full disaster recovery
Configuration files	Text under the config directory	Saved configuration objects	Yes	Change history and review
SCF (single configuration file)	Text	Entire configuration in one file	Yes	Portable configuration copy
Running configuration output	Text from tmsh list	Objects currently in memory	Yes	Seeing unsaved changes

The Configuration Files That Matter

BIG-IP splits its saved configuration across several files, and partitions add more.

File	Typical Contents
/config/bigip.conf	Traffic management objects such as virtual servers, pools, and profiles in the Common partition
/config/bigip_base.conf	Base network and system settings such as VLANs, self IPs, and interfaces
/config/bigip_user.conf	Local user accounts and related settings
/config/partitions/NAME/bigip.conf	Objects belonging to additional administrative partitions
/config/ssl	Certificate and key files referenced by the configuration

Running Versus Saved Configuration

Every change made through tmsh or the iControl REST interface goes into the running configuration held in memory by the configuration daemon. It is written to the configuration files only when someone runs the save command. The web interface typically saves automatically after each change, whereas tmsh does not.

Characteristic	Running Configuration	Saved Configuration
Location	Memory of the configuration daemon	Files on disk
Reflects current behavior	Yes	Only after a save
Survives a reboot	Only if saved	Yes
Included in a UCS	No, the UCS captures saved files	Yes
Visible through	tmsh list commands	Configuration files

This distinction creates a trap. A backup that reads files will not show an unsaved tmsh change, and a UCS created at that moment will not include it either. Conversely, a backup that reads the running configuration shows work that a reboot would discard.

Which Backup Should You Use?

The right choice depends on the question you want the backup to answer.

Goal	Recommended Capture	Reason
Rebuild a failed unit completely	UCS archive	Includes license, certificates, keys, and master key
Review what changed and when	Readable configuration in Git	Produces diffs and history
Catch unsaved changes	Running configuration capture	Reveals work in memory
Move configuration to another device	SCF or selected configuration stanzas	Portable and readable
Prove approved changes to an auditor	Git history paired with audit logs	Shows what, when, and who

For most organizations, the answer is both a scheduled UCS and an Oxidized-driven Git history, with an optional second Oxidized node that watches the running configuration.

How Oxidized Works with BIG-IP

Oxidized is built from sources, inputs, models, and outputs. For BIG-IP, the model determines which configuration you receive.

Architecture Components

Layer	Component	Function in a BIG-IP Deployment
Source	CSV, SQL, HTTP, or NetBox	Supplies BIG-IP addresses, models, and groups
Input	SSH	Opens a tmsh or shell session
Model	f5 or a custom model	Defines which commands run and how output is cleaned
Output	Git or gitcrypt	Stores each changed configuration as a commit
Hooks	githubrepo, exec, slackdiff	Reacts to stored changes
Interface	REST API and Oxidized Web	Shows node status and allows manual triggers

The Backup Lifecycle

1. Oxidized reads the node list from the source.
2. A worker opens an SSH session to the BIG-IP management address.
3. The model disables paging and prepares the session.
4. The model runs its commands and collects the output.
5. Volatile lines are removed from the output.
6. Oxidized compares the result with the previous version.
7. A changed result becomes a Git commit, and any hook fires.

Verify What the Bundled Model Captures

Oxidized ships with an f5 model, but the commands it runs have changed between releases, and some versions read the running configuration while others read files. Test it rather than assuming.

1. Run a backup of a lab BIG-IP and note the stored output.
2. On the lab unit, make an unsaved change with tmsh, such as modifying a pool description, and do not save.
3. Trigger a second Oxidized backup through the API.
4. If the description appears in the diff, the model reads the running configuration.
5. If nothing changes until you run the save command, the model reads saved files.
6. Repeat after saving to confirm that the change then appears.

This test takes ten minutes and tells you exactly what your history represents. Afterward, a custom model, covered later, gives you explicit control regardless of how the bundled one behaves.

Prerequisites and Planning

Requirement	Recommendation	Why It Matters
Backup server	Linux VM with 2 vCPU and 2 GB RAM	Light workload, Git and UCS storage grow steadily
Container runtime	Docker Engine with Compose	Simple upgrades and rollbacks
BIG-IP access	SSH to the management interface	Oxidized uses the CLI
Service account	Dedicated user with limited role	Limits blast radius
SSH allow list	Management SSH limited to the backup server and admins	Makes stolen credentials less useful
Time sync	NTP on all devices	Reliable logs and commit times
UCS storage	Encrypted, access-controlled file store with retention	UCS archives contain private keys
Remote Git	Private repository on GitLab, GitHub, or Gitea	Protects history if the server fails

High Availability and Device Groups

In a sync-failover pair, configuration synchronizes between the peers, yet each unit keeps its own management address and some device-specific settings. Back up each unit’s management address as a separate Oxidized node. As a result, you confirm that both units are reachable, and you capture local differences that a floating address would hide. In addition, remember that the prompt on a BIG-IP includes the failover state, which changes during a failover and must be tolerated by any prompt pattern.

Partitions and Route Domains

Administrative partitions create separate folders of objects, each with its own configuration file. A backup that captures only the Common partition omits everything application teams built in their own partitions. Therefore, whichever method you choose must cover all partitions, and the file-based custom model later in this guide does so explicitly.

Preparing BIG-IP for Oxidized

Create the Service Account

Create a dedicated user whose role and shell match what the backup method needs. A tmsh-based capture can often use a read-oriented role with a tmsh shell, whereas a file-based capture needs the advanced shell, which generally requires an administrative role. The commands below create an auditor-role user with tmsh access.

tmsh create auth user oxidized password ChangeThisLongRandomPassword shell tmsh partition-access add { all-partitions { role auditor } } tmsh save sys config

Role capabilities and shell options differ by software version, so test the account manually before wiring it into Oxidized. If the account cannot run the commands your model needs, move it to the smallest role that can, and document why.

Restrict SSH Access

BIG-IP can restrict which sources may reach the SSH daemon. Include the Oxidized server and your administrators.

tmsh modify sys sshd allow add { 10.10.50.25 } tmsh save sys config

Because an allow list that omits an administrator can lock people out, confirm the complete list before applying it, and keep console access available.

Use SSH Key Authentication

A dedicated key pair keeps the password out of your Oxidized configuration. Place the public key in the user’s authorized keys file according to your software version’s procedure, then reference the private key in Oxidized through the ssh_keys variable.

Confirm the Account Manually

ssh oxidized@10.10.3.11 "tmsh show sys version"

If the command returns version information, the account and path work. Otherwise, fix the role, shell, or allow list before continuing.

Installing Oxidized with Docker

Directory Layout

sudo mkdir -p /srv/oxidized/ssh /srv/oxidized/model /srv/ucs sudo chown -R 30000:30000 /srv/oxidized

The Core Configuration File

Save this as /srv/oxidized/config.

username: oxidized password: ChangeThisLongRandomPassword model: f5 interval: 3600 use_syslog: false debug: false threads: 10 timeout: 60 retries: 3 rest: 127.0.0.1:8888 vars: ssh_keys: /home/oxidized/.config/oxidized/ssh/id_rsa input: default: ssh ssh: secure: false output: default: git git: user: Oxidized email: oxidized@example.com repo: /home/oxidized/.config/oxidized/f5.git source: default: csv csv: file: /home/oxidized/.config/oxidized/router.db delimiter: !ruby/regexp /:/ map: name: 0 ip: 1 model: 2 group: 3

A 60-second timeout leaves room for large configurations with thousands of objects. Meanwhile, ten threads is modest because BIG-IP management planes share resources with traffic processing. Switch secure to true after populating known hosts.

The Device Inventory

lb-f5-dc1-01:10.10.3.11:f5:bigip lb-f5-dc1-02:10.10.3.12:f5:bigip lb-f5-dc2-01:10.20.3.11:f5:bigip

Start the Container

docker run -d --name oxidized \ --restart unless-stopped \ -p 127.0.0.1:8888:8888 \ -v /srv/oxidized:/home/oxidized/.config/oxidized \ oxidized/oxidized:latest oxidized

Follow the first run with docker logs -f oxidized, then confirm with curl -s http://127.0.0.1:8888/nodes.json and git -C /srv/oxidized/f5.git log --oneline. If a node fails, jump to the troubleshooting section.

A Custom Model for BIG-IP Configuration Files

When you want explicit control, a file-based model reads the saved configuration files directly, including every partition. It requires a user with shell access to run cat, so it is best paired with key authentication and a strict SSH allow list. Save this as /srv/oxidized/model/f5files.rb.

class F5Files < Oxidized::Model comment '# ' prompt %r{^\r?\[[\w.@:-]+\]\s\S+\s[#$]\s?$} cmd :all do |cfg| cfg.each_line.to_a[1..-2].join end cmd 'tmsh show sys version' do |cfg| comment cfg end cmd 'cat /config/bigip_base.conf' do |cfg| cfg end cmd 'cat /config/bigip.conf' do |cfg| cfg end cmd 'cat /config/partitions/*/bigip.conf 2>/dev/null' do |cfg| cfg end cfg :ssh do pre_logout 'exit' end end

The model name in the inventory is the lowercase class name, so this becomes f5files. The prompt pattern assumes the standard bash prompt, and because the prompt text includes the failover state and hostname, test it on both active and standby units. Restart the container so Oxidized loads the file, then add nodes.

lb-f5-dc1-01-files:10.10.3.11:f5files:bigip-files

This sketch is built on standard Oxidized model syntax, so adapt commands, prompts, and filters to your software version before production use. Notably, the user database file is deliberately omitted here because it contains password hashes. Add it only if your policy allows and your repository is encrypted.

Choosing Between the Bundled and File-Based Models

Consideration	Bundled f5 Model	File-Based Custom Model
Maintenance	Community maintained	You maintain it
Shell requirement	Often tmsh only	Advanced shell required
Partition coverage	Depends on version	Explicit through the partitions path
Running or saved	Verify by test	Saved files by design
Privilege needed	Lower	Higher

Many teams run the bundled model as the main node and the file-based model as a second node for partition-complete saved history, then compare the two.

Scheduled UCS Backups

UCS archives are the foundation of disaster recovery, so automate them. Because Oxidized stores text, the UCS process runs beside it.

Design Decisions

Decision	Recommendation	Reason
Where to create	On the BIG-IP, then copy off-box	UCS creation is a local tmsh operation
Encryption	Always use a passphrase	Archives contain private keys and the master key
Storage	Encrypted file store, not Git	Binary, large, and sensitive
Retention	Keep daily copies for 30 days and monthly copies longer	Balances recovery points and storage
Cleanup	Delete the on-box copy after transfer	Prevents disk fill and exposure
Verification	Periodically restore to a lab unit	Proves the archive works

A Scheduled UCS Script

Run this from the backup server, using a service account that can run tmsh and copy files. Store the passphrase in an environment file with restricted permissions or in a secrets manager.

#!/bin/bash set -euo pipefail HOST="$1" STAMP=$(date -u +%Y%m%dT%H%M%SZ) NAME="${HOST}-${STAMP}.ucs" DEST="/srv/ucs/${HOST}" KEY="/srv/oxidized/ssh/id_rsa" mkdir -p "$DEST" ssh -i "$KEY" "backup@${HOST}" "tmsh save sys ucs /var/local/ucs/${NAME} passphrase '${UCS_PASS}'" scp -i "$KEY" "backup@${HOST}:/var/local/ucs/${NAME}" "${DEST}/" ssh -i "$KEY" "backup@${HOST}" "rm -f /var/local/ucs/${NAME}" find "$DEST" -name '*.ucs' -mtime +30 -delete

The script creates a passphrase-protected UCS, copies it to the backup server, removes the on-box copy, and prunes archives older than thirty days. Schedule it from cron or a systemd timer, and run it for each BIG-IP in an inventory loop. In addition, confirm that the transfer succeeded before the cleanup step by checking file size, and alert when any run fails.

Handling Private Keys in a UCS

The save command offers an option that excludes private keys. Excluding them produces a smaller, less sensitive archive, but it also means that restoring the UCS will not restore certificate private keys, so applications may fail until the keys are reloaded. Most teams include private keys, protect the archive with a strong passphrase, and restrict access to the file store. Whichever you choose, document it in the runbook so the person restoring at three in the morning knows what to expect.

Alternative: iControl REST

BIG-IP also exposes UCS creation and download through iControl REST, which suits teams that prefer API-driven workflows and token authentication over SSH file copies. Large downloads use chunked transfers, so SCP is usually simpler for archives of any size. Choose the approach that fits your automation platform.

Git Versioning for Load Balancer Configurations

Why Git Fits

BIG-IP configuration is structured text, where each virtual server, pool, and profile is a brace-delimited stanza. Consequently, a Git diff shows exactly which pool member or profile setting changed, even in a configuration with tens of thousands of lines. Git also provides integrity, compact storage, and tools that auditors recognize.

Reading the History

git clone /srv/oxidized/f5.git ~/f5-configs cd ~/f5-configs # Last five changes to one load balancer git log -5 --stat -- bigip/lb-f5-dc1-01 # What changed in the latest commit git show HEAD -- bigip/lb-f5-dc1-01 # Compare two points in time git diff HEAD~10 HEAD -- bigip/lb-f5-dc1-01

Because nodes are stored under their group directory, adjust the paths to match. Commit messages record the node, so git log --grep quickly finds every change to one device.

Push to a Remote Repository

hooks: push_to_remote: type: githubrepo events: [post_store] remote_repo: git@git.example.com:netops/f5-configs.git publickey: /home/oxidized/.config/oxidized/ssh/deploy.pub privatekey: /home/oxidized/.config/oxidized/ssh/deploy

The hook works with any SSH-reachable Git server. Use a deploy key limited to this repository, and protect the main branch so history cannot be rewritten.

Taming Noise in BIG-IP Backups

A useful Git history contains commits that mean something. If every hourly run creates a change, reviewers stop looking and the one meaningful edit gets buried. BIG-IP noise comes from a short, predictable list.

Noise Source	What Happens	Remedy
Certificate object metadata	Create and update timestamps, checksums, and updater fields change when files are replaced	Accept as legitimate, since they accompany a real certificate change
Failover state in the prompt	The prompt text changes between Active and Standby	Make the prompt pattern tolerant, and keep prompt text out of stored output
Version and hardware output	Uptime and similar fields change constantly	Strip volatile lines in the model
Unsaved versus saved differences	Running and saved configuration diverge temporarily	Decide which one each node represents
Device-specific values between peers	Hostnames, self IPs, and management settings differ	Keep each peer as its own node
Config sync churn	A change on one peer appears on the other seconds later	Expect paired commits after synchronization

After changing any model, run two consecutive backups of a quiet lab unit. If the second run produces a commit, noise remains. Repeat until two consecutive runs produce identical files, and only then deploy broadly.

Secrets, Keys, and the Master Key

BIG-IP configurations and UCS archives contain sensitive material. Local user hashes live in the user configuration file, certificate private keys are stored under the SSL directory, and encrypted values inside the configuration are protected by the system’s master key.

Item	Where It Lives	Handling
Local user password hashes	User configuration file	Exclude from Git, or encrypt the repository
Private keys	SSL key directory and the UCS	Protect UCS archives with a passphrase and restrict storage access
Encrypted passphrases in the configuration	Inside configuration objects	Useless without the master key
Master key	Stored on the device and included in a UCS	Document escrow procedure, never commit it to Git
SNMP and monitor credentials	Configuration objects	Treat the repository as sensitive

Why the Master Key Matters

BIG-IP encrypts many stored secrets with a device master key. A UCS carries that key with it, so restoring a UCS generally brings the key back as well. In contrast, if you restore only text configuration to a device with a different master key, encrypted values will not decrypt, and objects that depend on them can fail to load. Therefore, treat the UCS as the full-fidelity recovery source, use text backups for review and partial rollbacks, and record who holds the passphrase for every archive.

Real-Time Backups with Audit Log Triggers

Hourly polling leaves a gap between a change and its capture. BIG-IP can log configuration changes to an audit log and forward them through syslog, which lets Oxidized capture a change within minutes or seconds.

How It Works

1. An administrator or automation changes the configuration.
2. BIG-IP writes an audit message describing the change.
3. The message is forwarded to the Oxidized server over syslog.
4. rsyslog passes the sender address and message to a script.
5. The script waits briefly, then calls the Oxidized API.
6. Oxidized fetches the configuration and commits the diff.

Configure Audit Logging and Forwarding

tmsh modify sys db config.auditing value enable tmsh modify sys syslog remote-servers add { oxidized { host 10.10.50.25 remote-port 514 } } tmsh save sys config

The first command enables audit logging of configuration changes, and the second forwards system logs to the Oxidized host. Verbosity options exist for the audit setting, so choose a level that records the information you need without flooding the log.

Configure rsyslog and the Script

template(name="ipmsg" type="string" string="%fromhost-ip% %msg%\n") module(load="imudp") input(type="imudp" port="514" ruleset="f5") ruleset(name="f5") { action(type="omprog" binary="/usr/local/bin/oxidized-trigger.sh" template="ipmsg") }

#!/bin/bash while read -r ip msg; do case "$msg" in *"user oxidized"*) ;; *AUDIT*) ( sleep 60; curl -s "http://127.0.0.1:8888/node/next/${ip}" > /dev/null ) & ;; esac done

Two details deserve attention. First, the script ignores messages mentioning the backup account, which prevents Oxidized’s own session from triggering another backup in a loop. Second, the sixty-second delay gives an administrator time to save the configuration before the capture runs. That matters for any node that reads saved files, since audit messages fire when a change is made, not when it is saved. Message wording varies between releases, so capture real samples from your own units and tighten the pattern. In addition, Oxidized resolves the address to the first node with that IP, so list your preferred node first when two nodes share a management address.

Using the Oxidized REST API and Web Interface

Endpoint	Purpose
/nodes	Lists nodes with status and last run time
/nodes.json	Same data in JSON for automation
/node/fetch/NAME	Returns the latest stored configuration
/node/next/NAME	Queues an immediate backup
/node/version?node_full=GROUP/NAME	Lists stored versions
/reload	Reloads the inventory

The JSON node list is a convenient monitoring feed. Alert on any node that stays failed for more than two intervals, which catches expired passwords, changed SSH allow lists, and units that have quietly become unreachable.

Securing the Backup Pipeline

Load balancer configurations reveal application topology, pool members, certificates, and sometimes credentials, so protect the pipeline accordingly.

Control	Recommendation
Service account	Dedicated user, smallest workable role, key authentication
SSH allow list	Restrict to the backup server and named administrators
Shell access	Grant the advanced shell only if a file-based model or UCS transfer requires it
UCS archives	Always passphrase protected, stored in an encrypted, access-controlled location
Passphrases	Held in a secrets manager, separate from the archives
Host key checking	Enable once known hosts are populated
API exposure	Bind to localhost, publish only through an authenticated TLS proxy
Repository	Private remote, deploy key, protected main branch
Monitoring	Alert on failed nodes, failed UCS jobs, and unexpected commits

These controls follow CIS Controls guidance on secure configuration and the configuration management and cryptographic key controls in NIST SP 800-53. Moreover, because the UCS service account needs elevated access, rotate its key regularly and watch its logins.

Restoring a BIG-IP from Backups

Restoration differs depending on which artifact you hold. A UCS rebuilds a whole system, while Git history supports precise rollbacks of individual objects.

Restoring from a UCS

1. Confirm that the replacement runs the same software version as the archive, or a version the vendor documents as compatible.
2. Make sure the license is active on the replacement, or plan to use the option that skips loading the license from the archive.
3. Copy the UCS file to the standard UCS directory on the device.
4. Load the archive with the passphrase.
5. Wait for the configuration daemon to finish loading, then check system state and synchronization status.
6. Verify virtual server availability, pool member health, certificates, and failover.

tmsh load sys ucs /var/local/ucs/lb-f5-dc1-01-20261024T020000Z.ucs passphrase 'ChangeThisPassphrase'

Several options modify the load. One skips the license in the archive, another resets device trust for a different device, and others relax platform checks. Because the right combination depends on whether you are restoring to the same unit, a replacement of identical hardware, or a different platform, consult the vendor documentation for your version and rehearse on a lab unit first.

Restoring Part of the Configuration from Git

When only a pool, virtual server, or profile needs to go back, copy the relevant stanza from Git into a small file, then merge it.

tmsh load sys config file /var/tmp/pool-fragment.conf merge verify

The verify option checks the file without applying it, which gives you a dry run. After a clean verification, run the same command without it and save the configuration. Confirm the exact syntax with tab completion on your version.

Restore Pitfalls

Pitfall	Consequence	Prevention
Version mismatch	UCS refuses to load or loads incompletely	Record software versions with every archive
Missing license	Modules and features stay unavailable	Reactivate licenses before restoring
Wrong master key	Encrypted values fail to decrypt	Use a UCS, which carries the key
Forgotten passphrase	Archive cannot be opened	Store passphrases in a secrets manager
Device trust and sync state	Peer shows disconnected after restore	Reset trust and re-establish the device group as documented
Unsaved changes at backup time	Recent work missing from the archive	Save before creating the UCS

Beyond F5: Other Load Balancers in the Same Pipeline

The pattern generalizes. Most application delivery controllers separate running from saved configuration, offer a full-system backup, and expose a text configuration that Oxidized can collect.

Platform	Text Configuration	Full Backup	Oxidized Support
Citrix NetScaler ADC	Running configuration display	System backup archive	Bundled model available in many releases
A10 Thunder ADC	Running configuration display	Backup to a remote server	Model available in many releases
Kemp LoadMaster	Limited CLI, mostly API and GUI	Backup file through API	Often handled by scripts
Radware Alteon	Configuration dump	Configuration export	Check the model list
HAProxy and NGINX	Plain configuration files	The files themselves	Store directly in Git through CI
Cloud load balancers	Infrastructure as code definitions	State files and templates	Keep definitions in Git

Because model coverage changes between releases, check the list your installation provides, and use the custom model pattern from earlier when a platform lacks one. For file-based proxies such as HAProxy and NGINX, the better practice is to keep configuration in Git as the source of truth and deploy from it, which makes backup a byproduct of the workflow.

Change Auditing and Compliance Evidence

Because every commit records what changed and when, Git history combined with BIG-IP audit logs forms a complete trail.

Framework	Relevant Area	How the Workflow Helps
NIST SP 800-53	CM-2, CM-3, CM-6	Baselines, tracked changes, documented settings
CIS Controls	Secure configuration of network devices	Versioned, reviewable configurations
ISO 27001:2022	Configuration management	Retrievable configuration history
PCI DSS 4.0	Network security control reviews and change approval	Diffs that simplify periodic review
SOC 2	Change management criteria	Timestamped records of production changes

To put this to work, send a daily digest of commits to the application and network teams, and require that each commit correlate with an approved change ticket. As a result, unapproved changes to virtual servers or certificates surface within a day.

Scaling Across Many BIG-IP Systems

Groups and Credentials

Use groups to separate data centers, environments, and business units, and to carry different credentials. Git stores each group in its own directory, which keeps large repositories navigable.

A Source of Truth for the Inventory

A CSV file is fine for a pilot but drifts over time. Switching to NetBox or an HTTP source means the inventory follows your documentation, and retired units leave the rotation automatically.

Timing and Load

BIG-IP management planes share resources with traffic processing, so keep thread counts conservative, stagger UCS jobs across the night, and avoid running them during peak traffic. Meanwhile, rely on audit triggers for timely capture instead of shortening the polling interval.

Troubleshooting BIG-IP Backups in Oxidized

Symptom	Likely Cause	Fix
Node shows no_connection	SSH allow list or routing blocks the server	Test manually from the Oxidized host and adjust the allow list
Authentication failed	Wrong password, expired password, or remote authentication delay	Verify the same login manually and check password policy
Permission denied on commands	Role or shell too limited for the model	Test each command manually and adjust role or model
Hang after login	Prompt pattern does not match	Allow for hostname, failover state, partition, and shell type in the pattern
Output interrupted by more prompts	Pager enabled	Disable paging for the session, and confirm the model runs it
Missing objects from other partitions	Capture limited to the Common partition	Use recursive capture or a model that reads partition files
Commit on every run	Volatile lines in output	Strip them and verify two consecutive matching runs
Backup loop after audit trigger	Oxidized’s own session generates audit messages	Exclude the service account in the trigger script
UCS save fails	Disk space or a wrong passphrase argument	Check free space, clear old archives, and test the command manually
UCS copy fails	SCP not permitted for the account	Grant the required shell and review the allow list

Debugging a Single Unit

Isolate the failing node, enable debug temporarily, and restart the container. Then run the same login and commands manually with the same account and compare transcripts. If the manual session works and Oxidized does not, the culprit is almost always the prompt pattern, the shell type, or a command the role cannot run.

After a Software Upgrade

Major upgrades can change prompts, SSH algorithms, or output formatting. When backups begin failing afterward, update the Oxidized image, retest custom models, and rerun the two-backup noise test. Also review UCS compatibility, since archives from one major version may not load on another.

Common Mistakes to Avoid

• Assuming a text backup covers every partition.
• Treating files as the running configuration, or the reverse, without testing.
• Storing UCS archives in Git or on an unprotected share.
• Creating UCS archives without a passphrase.
• Forgetting to save the configuration before creating a UCS.
• Ignoring the standby unit, then discovering it has diverged.
• Using an administrator account for the text backup when a lower role would work.
• Skipping the SSH allow list on the service account.
• Letting Oxidized’s own login trigger audit-driven backups in a loop.
• Never rehearsing a UCS restore on a lab unit.

Oxidized Compared with Other F5 Backup Options

Method	Cost	Version Control	Multi-Vendor	Best Fit
Oxidized	Free	Native Git	Yes	Readable history and API across vendors
Manual UCS or GUI archive	Included	File-based	No	Occasional manual backups
Scheduled UCS script	Free	File-based	No	Full-fidelity disaster recovery
BIG-IQ	Licensed	Backup sets	F5 only	Centralized F5 management and backups
Ansible with F5 collections	Free	You build it	Yes	Teams invested in Ansible
AS3 and Declarative Onboarding in Git	Free	Native Git	F5 only	Infrastructure as code for application services
Commercial NCM suites	Licensed	Yes	Yes	Vendor support and reporting

In practice, these options layer rather than compete. UCS archives provide full recovery, BIG-IQ or automation tools manage deployment, and Oxidized provides the readable, multi-vendor history that sits next to your routers, switches, and firewalls.

Real-World Scenarios

Scenario One: Finding the Change Behind a Checkout Failure

An e-commerce site starts returning errors after lunch. The engineer opens the Git history for the production load balancer and finds a commit from twenty minutes earlier that changed a pool monitor and removed a member. The diff shows the exact lines, the audit log names the administrator, and the fix is a short, targeted merge from the previous version.

Scenario Two: Catching an Unsaved Change Before a Reboot

The running-configuration node shows a virtual server change that never appears in the saved-file node. A quick check reveals that an engineer used tmsh and never saved. Because the team noticed, they save the change through the proper process instead of losing it during the next maintenance reboot.

Scenario Three: Rebuilding a Failed Unit

A data center BIG-IP suffers a hardware fault. The replacement is staged with the same software version, licensed, and loaded with the latest passphrase-protected UCS. After the load completes, the engineer verifies sync status and pool health, and traffic returns within the hour using a procedure the team had rehearsed.

Optimization Recommendations

Area	Recommendation	Benefit
Two-layer strategy	Oxidized for history, scheduled UCS for recovery	Review and recovery covered
Running versus saved	Test the bundled model, add a second node if needed	Accurate history
Partitions	Capture all of them explicitly	No silent gaps
UCS handling	Passphrase, encrypted storage, retention, and restore drills	Reliable recovery
Polling	Hourly interval plus audit triggers	Fresh data, low load
Models	Keep custom models in version control	Reviewable, reproducible behavior
Repository	Off-site mirror and periodic garbage collection	Resilience and smaller storage
Documentation	README with restore steps and version notes	Faster recovery under pressure

Frequently Asked Questions

Does Oxidized support F5 BIG-IP?

Yes. Oxidized includes an f5 model that connects over SSH and stores the configuration in Git. For explicit control, you can write a custom model that reads the configuration files directly.

Can Oxidized back up a UCS file?

No. Oxidized stores text configuration. UCS archives are binary and large, so create them with a scheduled tmsh job and store them in an encrypted file store alongside the Git history.

What is the difference between a UCS and a config file backup?

A UCS is a complete archive that includes the license, certificates, keys, and master key. Config file backups are readable text that show settings and changes but cannot rebuild a system on their own.

How do I create a UCS on BIG-IP?

Run the tmsh save sys ucs command with a file name, and add a passphrase to encrypt it. Then copy the archive off the device and delete the local copy.

Does a UCS include unsaved changes?

No. A UCS captures the saved configuration, so save the running configuration first if you want recent tmsh changes included.

Which BIG-IP configuration files does Oxidized need to capture?

At minimum the base configuration and the main configuration file, plus the configuration file for every additional partition. The user file contains password hashes, so include it only if your policy allows.

What permissions does the Oxidized account need on BIG-IP?

A tmsh-based capture can often use a read-oriented role with a tmsh shell. A file-based capture and UCS transfer need the advanced shell, which generally requires an administrative role. Test each command manually.

Can I restore a UCS to different hardware?

Often yes, within the vendor’s compatibility rules, but it may require extra options and matching software versions. Consult the documentation for your versions and rehearse on a lab unit.

How do I trigger an Oxidized backup after every F5 change?

Enable audit logging, forward syslog to the Oxidized server, and run a script that calls the next-node endpoint after a short delay, excluding the backup account to avoid loops.

Should I store UCS files in Git?

No. They are binary, large, and contain private keys. Use an encrypted, access-controlled file store with retention, and keep only readable text in Git.

Does Oxidized work with other load balancers?

Yes, for many platforms such as Citrix NetScaler and A10 Thunder, through bundled or custom models. File-based proxies like HAProxy and NGINX are better managed with configuration in Git directly.

Conclusion

Backing up F5 BIG-IP with Oxidized works best as a two-layer design. Scheduled, passphrase-protected UCS archives provide the full-fidelity recovery that text cannot, while Oxidized and Git provide the readable history, diffs, and audit trail that archives cannot. The key to getting both right is understanding what each artifact actually captures: running versus saved configuration, every partition and not just the default one, and secrets and master keys that must be protected rather than committed. Furthermore, audit-log triggers and rehearsed restores turn a periodic snapshot into a living, trustworthy record.

The practical next step is small. Test your bundled model with an unsaved change on a lab unit, add the file-based model if you need partition-complete history, schedule one encrypted UCS job, and rehearse a restore. After that, extend the same pipeline to your other load balancers and let the repository become the record of truth for your application delivery layer.