Backups and Recovery

This page distinguishes Proxmox VM backups from the services appdata archive. They solve different recovery problems.

Coverage

Asset	Current protection	History/retention	Restore status
Ubuntu Cloud-Init template	Manual Proxmox vzdump to pve-compute	Depends on retained dump files	Procedure documented below
Active VMs	Manual Proxmox backup	Depends on Proxmox storage policy	Procedure documented below; IDs must be verified
Ardougne appdata	Timer publishes a metadata-preserving archive when deployed	One current archive; TrueNAS snapshots provide history	Guarded Ansible restore for fresh VM only
Compose definitions	Recreated from Ansible	Repository history only	Run Ansible
Generated secret environment files	Recreated from Ansible Vault	Vault copy/repository protection	Run Ansible
Media library	Stored on TrueNAS, outside VM backup	Not documented here	Use TrueNAS protection
TrueNAS configuration and datasets	Not documented	TBD	TBD
Proxmox host configuration	Not documented	TBD	Reinstall and repeat Proxmox

Ansible keeps one current archive, not a history of backups. The prior current archive is retained only while the next archive is uploaded and verified. TrueNAS snapshots provide retention and recovery from accidental changes or corruption.

Cloud-Init Template

Back up template 9000:

vzdump 9000 \
  --storage pve-compute \
  --mode stop \
  --compress zstd
 
pvesm list pve-compute --content backup

Restore it after confirming VM ID 9000 is free:

pvesm list pve-compute --content backup
qmrestore \
  /mnt/pve/pve-compute/dump/vzdump-qemu-9000-YYYY_MM_DD-HH_MM_SS.vma.zst \
  9000 \
  --storage local-lvm
 
qm config 9000
qm list

If the restored object is not marked as a template, verify its configuration and run qm template 9000 before cloning it.

Back Up a VM in Proxmox

VM backups are currently manual. First obtain the authoritative IDs and assign the VM being backed up:

qm list
VMID=101

For the cleanest backup, stop the VM or use the Proxmox UI backup workflow. A CLI backup to pve-compute is:

vzdump "$VMID" \
  --storage pve-compute \
  --mode snapshot \
  --compress zstd

Use --mode stop instead when downtime is acceptable and application consistency is more important. Verify the resulting dump:

pvesm list pve-compute --content backup

The VM’s qemu-guest-agent should be active, but a Proxmox VM backup does not replace application-aware backup verification.

Restore a VM in Proxmox

Choose an unused target ID. Restoring over an existing VM is not part of this runbook.

qm list
pvesm list pve-compute --content backup
 
BACKUP=/mnt/pve/pve-compute/dump/vzdump-qemu-101-YYYY_MM_DD-HH_MM_SS.vma.zst
NEW_VMID=101
 
qmrestore \
  "$BACKUP" \
  "$NEW_VMID" \
  --storage local-lvm
 
qm config "$NEW_VMID"
qm start "$NEW_VMID"

After restore:

1. Verify the NIC, bridge, Cloud-Init settings, and DHCP reservation.
2. Reattach the GPU PCI device for Varrock if it is absent.
3. Verify SSH and qemu-guest-agent.
4. Update inventory/hosts.yml if the IP changed.
5. Run the appropriate Ansible host-group play to reconcile configuration.
6. Verify the services listed in Inventory and Services.

Services Appdata Backup

When the backup role is deployed, Ansible installs service-appdata-backup.timer on Ardougne. It runs daily at 04:30 in the America/Toronto timezone configured by the common role.

Source:      /srv/docker/appdata/
Staging:     /srv/docker/backup-staging/appdata/
Destination: /mnt/service-backups/ardougne/appdata/current/appdata.tar.zst
Marker:      /mnt/service-backups/ardougne/appdata/.last-success

The script:

1. Verifies /mnt/service-backups is mounted over CIFS.
2. Takes an exclusive flock lock to reject overlapping backups.
3. Requires the arr, downloads, reverse-proxy, and monitoring Compose projects to exist.
4. Records running services in those projects.
5. Stops only services that were running.
6. Mirrors appdata to local staging.
7. Restarts the services.
8. Creates a compressed tar.zst archive with numeric ownership, modes, ACLs, extended attributes, and symlinks preserved inside the archive.
9. Generates appdata.tar.zst.sha256 and manifest.yml.
10. Uploads and verifies an incoming archive before publishing it as current.
11. Writes .last-success only after publication succeeds.

The published backup layout is:

/mnt/service-backups/ardougne/appdata/
├── current/
│   ├── appdata.tar.zst
│   ├── appdata.tar.zst.sha256
│   ├── manifest.yml
│   └── success.yml
└── .last-success

manifest.yml records the backup ID, source host, archive checksum, appdata size, and the required top-level service directories. success.yml repeats the published backup ID and checksum. Restore requires the archive, checksum, manifest, and success marker to agree. TrueNAS snapshots should protect this dataset because Ansible deliberately keeps only the latest published archive.

The backup excludes Compose files, generated .env files, Docker images, /mnt/media, and host configuration. Compose files and secret environment files are regenerated from Ansible and Vault; media is protected by TrueNAS separately. The service-backups mount is root-only because the archive can include application secrets such as Uptime Kuma notification webhooks.

Verify the timer and latest run on Ardougne:

timedatectl
systemctl list-timers service-appdata-backup.timer
systemctl status service-appdata-backup.service
journalctl -u service-appdata-backup.service
sudo cat /mnt/service-backups/ardougne/appdata/.last-success
sudo sh -c 'cd /mnt/service-backups/ardougne/appdata/current && sha256sum -c appdata.tar.zst.sha256'

Run an on-demand backup before risky service work:

sudo systemctl start service-appdata-backup.service
sudo systemctl status service-appdata-backup.service

Fresh Services VM Without Configuration Restore

Use this when a blank services configuration is desired.

1. Create or restore the Ubuntu services VM in Proxmox, apply its expected IP address, and confirm SSH access.
2. From the Ansible controller, run:

./run.sh --limit services_servers

3. Complete the blank-install work in Manual Steps > Services VM Starting Blank Only. This includes qBittorrent credentials and ARR/Prowlarr configuration.
4. Verify the service catalog in Inventory and Services.
5. Verify service-appdata-backup.timer and create an on-demand backup.

This ordinary deployment does not run service_appdata_restore, even though the role appears in site.yml; it is tagged never and requires explicit recovery authorization.

Recover a Fresh Services VM

Use this when moving to a new services VM or rebuilding after data loss, and the latest TrueNAS appdata archive should become the new VM’s configuration.

This workflow is intentionally limited to a fresh VM. It refuses to merge with, replace, or repair an existing nonempty /srv/docker/appdata directory.

1. Recreate or restore the Ubuntu services VM in Proxmox. Confirm it is reachable at the inventory address and has the Cloud-Init tuero user.
2. On the Ansible controller, install the base OS configuration and TrueNAS mounts only. Do not install Docker or service stacks yet:

./run.sh --limit services_servers --tags common,storage

3. Run the read-only restore preflight. It verifies the CIFS mount, .last-success, archive checksum, manifest, archive contents, local disk space, appdata target, Docker state, and backup-service state:

./run.sh \
  --limit services_servers \
  --tags restore \
  --check \
  -e service_restore_enabled=true \
  -e service_restore_confirmation=RESTORE_ardougne_TO_services

4. If preflight succeeds, run the restore. It extracts into a sibling staging directory under /srv/docker and atomically publishes /srv/docker/appdata only after validating required service directories:

./run.sh \
  --limit services_servers \
  --tags restore \
  -e service_restore_enabled=true \
  -e service_restore_confirmation=RESTORE_ardougne_TO_services

5. Recreate Docker, Compose files, proxy network, generated PIA/Cloudflare environment files, and application containers:

./run.sh --limit services_servers --tags docker,arr,downloads,proxy,monitoring

6. Verify:

ssh tuero@ardougne.tuerolab.ca
docker compose --project-directory /srv/docker/compose/arr ps
docker compose --project-directory /srv/docker/compose/downloads ps
docker compose --project-directory /srv/docker/compose/reverse-proxy ps
docker compose --project-directory /srv/docker/compose/monitoring ps
cat /srv/docker/.last-appdata-restore.yml

Verify restored ARR/Prowlarr settings, qBittorrent login and forwarded port, Gluetun VPN egress, media access, Caddy HTTPS endpoints, Homepage container indicators, Uptime Kuma monitors/notifications, and appdata ownership.

No application Web UI reconfiguration should be required after a successful restore. The archive includes qBittorrent credentials and preferences; Gluetun state; Radarr, Sonarr, Bazarr, and Prowlarr databases/settings; indexers; download clients; media roots; application links; Caddy state; and Uptime Kuma monitors, notifications, and history. PIA and Cloudflare credentials are regenerated from Ansible Vault, so they must still be valid.

7. Enable scheduled backups only after application validation:

./run.sh --limit services_servers --tags backups

8. Trigger one backup manually and verify its checksum on TrueNAS before treating the replacement VM as recovered.

In-Place Services Recovery

The restore role does not support an existing services VM with appdata or running containers. Do not bypass its safety checks.

For an in-place incident, first restore a Proxmox VM backup when available. If that is unsuitable, preserve the affected /srv/docker/appdata, create a separate recovery plan, and test it on a disposable VM before modifying production state.

Recover a Fresh Infra VM

Lumbridge has no VM-local state that requires a backup when AdGuard configuration remains Ansible-managed. A fresh Ansible run recreates AdGuard Home, its administrator account, DNS rewrites, Tailscale, IP forwarding, subnet-route advertisement, and exit-node advertisement.

The intentional manual recovery step is Tailscale enrollment. The Tailscale node identity is local to the destroyed VM, and the replacement VM appears as a new node in the tailnet.

1. Recreate or restore VM 102 from template 9000, retaining 10.0.0.112. See Proxmox > Create a VM. Confirm LAN SSH access or use the Proxmox console.
2. From the Ansible controller, configure the complete Infra VM:

./run.sh --limit infra_servers

This installs and configures AdGuard Home immediately. Tailscale is installed but cannot join the tailnet without authentication.

3. From the LAN or Proxmox console, enroll the replacement node:

ssh tuero@10.0.0.112
sudo tailscale up
tailscale status

Open the authentication URL shown by tailscale up and authenticate it with the tailnet.

4. On the controller, rerun the Tailscale role so it applies the subnet and exit-node advertisements:

./run.sh --limit infra_servers --tags tailscale

5. In the Tailscale admin console, approve the replacement node’s advertised subnet 10.0.0.0/24 and exit-node use. Remove the destroyed VM’s old node after confirming the replacement is healthy.
6. Verify the existing Tailscale split-DNS configuration still sends tuerolab.ca queries to 10.0.0.112. If it was also lost, recreate the custom nameserver with Restrict to search domain for tuerolab.ca, enable Use with exit node, and keep MagicDNS enabled.
7. Verify AdGuard DNS and remote reachability:

dig @10.0.0.112 varrock.tuerolab.ca +short
dig @10.0.0.112 ardougne.tuerolab.ca +short
ssh tuero@ardougne.tuerolab.ca

The router’s LAN DNS setting must continue to distribute 10.0.0.112.

Manual changes made through the AdGuard UI, such as upstream resolvers, client groups, filtering lists, or non-Ansible filtering rules, are not recovered by this workflow. Put required settings in Ansible before relying on fresh-VM recovery.

Full Rebuild After Reinstalling Proxmox

Assuming TrueNAS, pve-compute, the Ansible repository, encrypted Vault, and services appdata archive survived:

1. Install and update Proxmox using Proxmox.
2. Reconnect pve-compute and verify Proxmox backups/template images are visible.
3. Restore template 9000, or recreate it from the Ubuntu cloud image.
4. Restore whole VMs when suitable dumps exist; otherwise clone fresh GPU, infra, and services VMs.
5. Recreate DHCP reservations or Cloud-Init addresses.
6. Reconfigure host IOMMU/VFIO and attach the GPU to Varrock.
7. Verify SSH access from the Ansible controller.
8. Run ./run.sh --limit gpu_servers for Varrock. Recover Lumbridge with Backups and Recovery > Recover a Fresh Infra VM.
9. Choose one services path:
   • Use Backups and Recovery > Fresh Services VM Without Configuration Restore for a blank services environment.
   • Use Backups and Recovery > Recover a Fresh Services VM to recover the archived application configuration.
10. Verify DNS, HTTPS, VPN routing, Jupyter/CUDA, timers, and all service state.

Recovery Gaps

• The restore role is fresh-VM-only; there is no automated in-place appdata overwrite.
• Ansible keeps only the current appdata archive. TrueNAS snapshot/replication policy must provide history and disaster protection.
• No Proxmox host configuration backup is documented.
• Infra application state is not backed up by service-appdata-backup; Ansible rebuilds AdGuard and Tailscale, but manual AdGuard UI configuration is not recovered.
• VM backup scheduling and retention are not documented.
• Restore drills and dates are not recorded.