Jake Tuero

Proxmox

5 min read

Proxmox

This page covers the hypervisor layer. Guest configuration begins in Ansible, and recovery order is documented in Backups and Recovery.

Initial Setup

If the installer has display problems with an NVIDIA GPU, press e in the boot menu and add nomodeset to the line containing quiet.

The current Proxmox host is Keldagrim at 10.0.0.110. Its web UI is available at https://keldagrim.tuerolab.ca on the LAN or through Tailscale. Direct recovery access remains available at https://10.0.0.110:8006. Use a <name>.pve hostname on each Proxmox host.

After installation, configure the desired Proxmox repositories and updates. The previously used community post-install script is available from Proxmox VE Helper-Scripts, but it executes remote code and should be reviewed before use.

Update from Node > Updates > Refresh > Upgrade, then reboot when required.

TrueNAS Storage

The current examples use Proxmox storage ID pve-compute, backed by TrueNAS. Verify the configured protocol and content types in Datacenter > Storage; captured output identifies it as CIFS even though an older setup note described NFS.

pvesm status
pvesm list pve-compute

To remove storage, first remove it from Datacenter > Storage, then verify it is no longer mounted before deleting its empty mountpoint:

mountpoint /mnt/pve/pve-compute
rmdir /mnt/pve/pve-compute

GPU Passthrough Host Setup

The current GPU is an NVIDIA GeForce RTX 3090 with graphics and audio PCI IDs 10de:2204 and 10de:1aef. Recheck these IDs after hardware changes.

  1. Enable VT-d or AMD IOMMU in the system firmware.
  2. Add the matching boot parameters to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub:
quiet intel_iommu=on iommu=pt
quiet amd_iommu=on iommu=pt
  1. Run update-grub.
  2. Add these modules to /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
  1. Bind both GPU functions in /etc/modprobe.d/vfio.conf:
options vfio-pci ids=10de:2204,10de:1aef
  1. Blacklist only the host drivers that would otherwise claim this passthrough GPU. The current NVIDIA setup used:
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist rivafb

Do not blacklist unrelated GPU drivers unless that hardware is also being passed through.

  1. Rebuild initramfs and reboot:
update-initramfs -u -k all
reboot
  1. Verify IOMMU, the device IDs, VFIO binding, and isolation:
dmesg | grep -E 'DMAR|IOMMU'
lspci -nn | grep -i nvidia
lspci -nnk -d 10de:2204
lspci -nnk -d 10de:1aef
 
GROUP=$(basename "$(readlink /sys/bus/pci/devices/0000:65:00.0/iommu_group)")
for device in /sys/kernel/iommu_groups/"$GROUP"/devices/*; do
  lspci -nn -s "${device##*/}"
done

The addresses such as 65:00.0 are hardware-specific. Both GPU functions should use vfio-pci, and their IOMMU group must not contain unrelated devices.

If the audio function is still claimed by snd_hda_intel, add this troubleshooting line after the options line in /etc/modprobe.d/vfio.conf, rebuild initramfs, and reboot:

softdep snd_hda_intel pre: vfio-pci

Create the Ubuntu Cloud-Init Template

The current template is Ubuntu 26.04, VM ID 9000, named ubuntu-26.04-cloud-template. Create a separate template for each Ubuntu release, or restore the matching template backup.

Locate the image

Upload the Ubuntu cloud image to pve-compute, then locate it:

pvesm list pve-compute --content iso
pvesm path pve-compute:iso/resolute-server-cloudimg-amd64.img

Build the template

VMID=9000
IMAGE=/mnt/pve/pve-compute/template/iso/resolute-server-cloudimg-amd64.img
 
qm create "$VMID" \
  --name ubuntu-26.04-cloud-template \
  --machine q35 \
  --bios ovmf \
  --cpu host \
  --cores 24 \
  --memory 65536 \
  --scsihw virtio-scsi-pci \
  --net0 virtio,bridge=vmbr0 \
  --agent enabled=1 \ 
  --onboot 1
 
qm set "$VMID" --efidisk0 local-lvm:0,efitype=4m,pre-enrolled-keys=0
qm set "$VMID" --scsi0 "local-lvm:0,import-from=${IMAGE},discard=on,ssd=1"
qm set "$VMID" --boot order=scsi0
qm disk resize "$VMID" scsi0 500G
qm set "$VMID" --ide2 local-lvm:cloudinit
qm set "$VMID" --serial0 socket --vga serial0
qm set "$VMID" --ciuser tuero
qm set "$VMID" --ipconfig0 ip=dhcp

Add the Ansible controller’s SSH public key under the VM’s Cloud-Init settings. Do not start the template VM.

Inspect and convert it:

qm config "$VMID"
qm cloudinit dump "$VMID" user
qm template "$VMID"

The oversized template resources are defaults only; set each clone’s actual CPU, memory, and disk requirements explicitly.

Back Up and Restore the Template

See Backups and Recovery > Cloud-Init Template for the current vzdump and qmrestore procedure.

Create a VM

All three VMs are full clones of template 9000 on local-lvm. Their VM IDs, resource allocations, and static Cloud-Init addresses match Inventory and Services.

VMVM IDvCPUMemoryIP
GPU / Varrock1011232 GiB10.0.0.111
Infra / Lumbridge10244 GiB10.0.0.112
Services / Ardougne103816 GiB10.0.0.113
# GPU / Varrock
qm clone 9000 101 --name gpu-server --full 1 --storage local-lvm
qm set 101 --cores 12 --memory 32768
qm set 101 --ipconfig0 "ip=10.0.0.111/24,gw=10.0.0.1"
 
# Infra / Lumbridge
qm clone 9000 102 --name infra --full 1 --storage local-lvm
qm set 102 --cores 4 --memory 4096
qm set 102 --ipconfig0 "ip=10.0.0.112/24,gw=10.0.0.1"
 
# Services / Ardougne
qm clone 9000 103 --name services --full 1 --storage local-lvm
qm set 103 --cores 8 --memory 16384
qm set 103 --ipconfig0 "ip=10.0.0.113/24,gw=10.0.0.1"

DHCP reservations are an alternative to the Cloud-Init static addresses above, but must resolve to the same addresses configured in Ansible inventory.

Start and verify each VM:

VMID=103
IP=10.0.0.113
qm start "$VMID"
ssh "tuero@$IP"

Add the GPU to Varrock

In VM 101 > Hardware > Add > PCI Device, select the GPU raw device, enable PCI Express and All Functions, and leave Primary GPU disabled.

After the VM is reachable, the Ansible nvidia_cuda role installs and verifies the guest NVIDIA driver and CUDA toolkit. It does not configure host IOMMU or the Proxmox PCI device.

Graph View

Backlinks