Services

See Inventory and Services for URLs and ports, Routing and Access for DNS and Caddy request flows, and Ansible for the complete deployment order.

Infra VM

Tailscale

Ansible installs Tailscale, enables forwarding, advertises 10.0.0.0/24, and advertises Lumbridge as an exit node.

For a replacement VM, use Backups and Recovery > Recover a Fresh Infra VM. No Infra appdata backup is required when AdGuard remains Ansible-managed.

First-time enrollment and control-plane approval remain manual:

./run.sh --limit infra_servers
ssh tuero@10.0.0.112
sudo tailscale up
tailscale status

Then rerun the role:

./run.sh --limit infra_servers --tags tailscale

In the Tailscale admin console, approve subnet 10.0.0.0/24 and exit-node use. Configure split DNS for tuerolab.ca through nameserver 10.0.0.112, enable use with the exit node, and keep MagicDNS enabled.

AdGuard Home

The Ansible role installs AdGuard Home, binds it to 10.0.0.112, and sets the permanent admin UI to port 3001. Caddy on Ardougne proxies https://adguard.tuerolab.ca to that UI. The direct LAN address http://lumbridge.tuerolab.ca:3001 remains available for recovery. The role sends only adguardhome_dns_rewrites to AdGuard’s set_rules API, so all custom filtering rules are currently Ansible-owned; a role run replaces manually added custom rules.

./run.sh --limit infra_servers --tags adguard
./run.sh --limit services_servers --tags proxy

Verify DNS:

dig @10.0.0.112 varrock.tuerolab.ca +short
dig @10.0.0.112 lumbridge.tuerolab.ca +short
dig @10.0.0.112 ardougne.tuerolab.ca +short
dig @10.0.0.112 falador.tuerolab.ca +short
dig @10.0.0.112 adguard.tuerolab.ca +short

The router should distribute 10.0.0.112 as LAN DNS. Verify the HTTPS route after both roles complete:

curl --fail --head https://adguard.tuerolab.ca

GPU VM

The full gpu_servers play installs development tools, mounts /mnt/backup, installs GitHub SSH credentials, installs NVIDIA drivers and CUDA, configures JupyterLab, and performs post-install repository setup.

./run.sh --limit gpu_servers
ssh tuero@varrock.tuerolab.ca nvidia-smi

Project Workspace Sync

TrueNAS is the canonical, snapshot-protected copy of programming projects at /mnt/projects/<project>. Work locally on Varrock under ~/projects/<project> so builds do not use SMB or fill TrueNAS with generated
artifacts. The projects SMB share must be created on TrueNAS before deploying
the storage,projects tags:

./run.sh --limit gpu_servers --tags storage,projects

Ansible installs directional synchronization commands. They preview the exact operation by default; append --apply only after reviewing the itemized output:

# Start work from the canonical TrueNAS copy.
project-pull name
project-pull name --apply
 
# Save local work back to the canonical TrueNAS copy.
project-push name
project-push name --apply

The commands copy normal files and .git directories, and exclude build and cache paths such as build/, build-*, cmake-build-*, out/, and object files. Both directions use --delete-delay, so an apply operation makes the destination match the source after its transfer succeeds. Do not edit one project from multiple locations between a pull and push; the commands are deliberately directional and do not resolve conflicts.

The SMB mount does not preserve POSIX ownership or permission bits. Git history and Git’s executable-bit metadata are preserved because .git is copied, but an SMB restore can leave local files without their original executable mode. For ordinary source projects, configure git config core.filemode false in the local repository to suppress mode-only status noise. Restore executable scripts from a committed Git revision, or run chmod +x and commit the intended mode, when that metadata matters. The projects mount enables CIFS mfsymlinks; test repositories that rely on symlinks before treating SMB as their recovery path.

A project-root compile_commands.json symlink is synchronized, but its target under an excluded build directory is not. After a pull, the link remains dangling until the local build recreates its target. Other symlinks are also preserved unless their path or filename matches an exclusion. Files imported directly from a TrueNAS root shell must also be owned by the SMB user and inherit the projects dataset ACL. If imported files cannot be changed or deleted while newly created files can, follow TrueNAS > Users to repair their server-side ownership and ACLs.

JupyterLab listens on all interfaces at port 8888 and uses the Vault-provided password hash. Caddy on Ardougne proxies https://jupyter.tuerolab.ca to 10.0.0.111:8888; deploy the AdGuard rewrite and proxy route with:

./run.sh --limit infra_servers --tags adguard
./run.sh --limit services_servers --tags proxy

Until the router distributes AdGuard as LAN DNS, jupyter.tuerolab.ca resolves only on the manually configured Mac. Use http://10.0.0.111:8888 from VMs that do not use AdGuard DNS.

To add a Conda environment as a Jupyter kernel:

conda create -n research python=3.12
conda activate research
conda install ipykernel pytorch numpy scipy
python -m ipykernel install --user --name research --display-name "Python (research)"

Services VM

Layout

/srv/docker/compose/arr
/srv/docker/compose/downloads
/srv/docker/compose/monitoring
/srv/docker/compose/reverse-proxy
/srv/docker/appdata/<service>
/mnt/media
/mnt/service-backups

Compose files and secret environment files are generated by Ansible. Do not treat manual edits under /srv/docker/compose as durable configuration; update the Ansible templates instead.

ARR Stack

./run.sh --limit services_servers --tags arr
ssh tuero@ardougne.tuerolab.ca
docker compose --project-directory /srv/docker/compose/arr ps

Radarr, Sonarr, Bazarr, and Prowlarr use /mnt/media and store configuration in /srv/docker/appdata/<service>. Existing appdata contains the settings created through their Web UIs, including indexer and download-client integration.

Downloads Stack

./run.sh --limit services_servers --tags downloads
docker compose --project-directory /srv/docker/compose/downloads ps

qBittorrent shares Gluetun’s network namespace. PIA credentials come from Ansible Vault, and Gluetun provides the VPN kill switch and port forwarding.

On blank appdata, find qBittorrent’s temporary admin password:

docker compose --project-directory /srv/docker/compose/downloads \
  logs qbittorrent | grep -i -A3 -B3 password

After login, set a permanent password and use /media/downloads as the download directory.

Verify that Gluetun and the host use different public IP addresses:

docker exec gluetun wget -qO- https://ipinfo.io/ip
curl https://ipinfo.io/ip

Check the assigned forwarded port and confirm qBittorrent received the same listening port:

docker logs gluetun 2>&1 | grep -i -E 'port forward|forwarded port'
docker exec gluetun wget -qO- \
  http://127.0.0.1:8080/api/v2/app/preferences \
  | tr ',' '\n' | grep '"listen_port"'

The Compose up-command updates qBittorrent through its local API whenever Gluetun receives a forwarded port. This hook currently depends on qBittorrent allowing unauthenticated API requests from localhost; verify both values after a VPN reconnect.

Perform a basic namespace-availability check:

docker run --rm --network container:gluetun curlimages/curl -s https://ipinfo.io/ip
docker compose --project-directory /srv/docker/compose/downloads stop gluetun
docker run --rm --network container:gluetun curlimages/curl \
  --max-time 10 https://ipinfo.io/ip
docker compose --project-directory /srv/docker/compose/downloads up -d

The second request should fail because Gluetun’s network namespace is unavailable. This confirms the Compose namespace coupling, but it is not by itself a complete VPN leak test; also verify qBittorrent remains configured with network_mode: service:gluetun and has no independent published ports.

Monitoring Stack

Homepage provides service navigation and at-a-glance HTTP reachability indicators. Uptime Kuma provides active HTTP checks, uptime history, and notifications. Caddy restricts both applications to LAN (10.0.0.0/24) and Tailscale (100.64.0.0/10) source addresses:

https://homepage.tuerolab.ca
https://status.tuerolab.ca

Deploy the monitoring DNS, proxy routes, and stack:

./run.sh --limit infra_servers --tags adguard
./run.sh --limit services_servers --tags storage,proxy,monitoring,backups

Homepage configuration is rendered by Ansible under /srv/docker/appdata/homepage; update the monitoring role templates rather than editing those files manually. Its Systems section is a host reference for Varrock, Lumbridge, Ardougne, and Falador; the remaining sections contain service links and HTTP reachability indicators. Homepage intentionally has no Docker socket or Docker API access. This prevents a compromised dashboard from reading container environment variables or application data. Homepage v2.0.0 and Uptime Kuma 2.5.0 are pinned to reviewed image digests; update their tag and digest together through the monitoring role after reviewing a release.

Uptime Kuma stores monitors, notification integrations, status pages, and its administrator account under /srv/docker/appdata/uptime-kuma. This tuero-owned directory has mode 0700, and its state is included in the services appdata archive.

On a blank Uptime Kuma deployment:

1. Open https://status.tuerolab.ca and create the administrator account.
2. Add HTTP(s) monitors using these internal targets:

http://radarr:7878
http://sonarr:8989
http://bazarr:6767
http://prowlarr:9696
http://host.docker.internal:8080
http://10.0.0.112:3001
http://10.0.0.111:8888

The first four names work because Uptime Kuma and the ARR containers share the Docker proxy network. The qBittorrent target reaches its Gluetun-published host port. The final two targets monitor AdGuard and Jupyter on their respective VMs.

3. In Settings > Notifications, add a Discord notification, paste the Discord webhook URL, test it, and attach it to the monitors. The webhook is stored in Uptime Kuma appdata and is restored with the monitoring configuration. Treat it as a secret.
4. Configure monitor intervals, accepted status codes, and notification thresholds to your preference. For endpoints that redirect, allow the appropriate 3xx status or monitor a direct application endpoint.

After a services appdata restore, open Homepage and Uptime Kuma to verify their state. A restored Uptime Kuma should retain its administrator account, monitors, notification integrations, and history; no UI reconfiguration should be required.

Reverse Proxy

Caddy is built with the Cloudflare DNS module. The Vault token requires Cloudflare Zone:Read and DNS:Edit for tuerolab.ca; it is used for ACME DNS-01 certificate validation.

./run.sh --limit services_servers --tags proxy
docker compose --project-directory /srv/docker/compose/reverse-proxy ps

When adding an HTTP service:

1. Add its container and appdata directory to the appropriate Ansible role.
2. Attach it to the external Docker proxy network when Caddy can address it directly.
3. Add its name to adguardhome_dns_rewrites in group_vars/infra_servers/vars.yml.
4. Add its route to reverse_proxy_routes in roles/reverse_proxy/defaults/main.yml.
5. Add its stateful Compose project to service_backup_compose_projects and its appdata directory to both backup and restore expected-directory lists.
6. Run the affected service, AdGuard, proxy, and backup roles.
7. Verify DNS, HTTPS, and backup coverage.