Skip to content

Proxmox Recovery

Last Updated: 2026-07-19 Status: ✅ Scenario B tested for real — Pacific's boot disk failed and this procedure was executed end-to-end on 2026-07-19. Scenarios A and C remain unverified. Purpose: Restore Pacific (Proxmox NUC) after hardware or software failure.

Before You Start

  • Proxmox host: Pacific — 192.168.1.10
  • VM/LXC backups stored on: Lotus via Proxmox Backup Server (PBS Docker container)
  • Most critical VM: HomeAssistant (VM 112) — see HA Recovery if that's all that's broken

Scenario A — Restore a Single VM or LXC

Use this when Pacific is running but a specific VM or container needs restoring.

  1. Open Proxmox: https://192.168.1.10:8006
  2. In the left panel, select the PBS storage
  3. Find the relevant VM or LXC backup
  4. Click Restore
  5. Start the restored VM/LXC
  6. Verify the service is accessible

VM/LXC restore priority:

Priority ID Name Purpose
1 VM 112 HomeAssistant Smart home — see HA Recovery
2 LXC 103 TailscaleExitNode External access exit node
3 LXC 108 Frigate Doorbell camera NVR
4 LXC 130 GMUPS-27 UPS monitoring
5 LXC 101 AdGuard DNS (currently inactive)

Scenario B — Full Proxmox Reinstall (Pacific hardware survived)

Use this when Pacific needs a fresh Proxmox install but the hardware is intact. Executed for real on 2026-07-19 (failing boot disk, not dead hardware) — steps below reflect what actually worked, not just the plan.

Before starting: - If HA needs to keep running during the rebuild, fail it over to Lotus first — HA Recovery → Scenario B. Doing this once and leaving Pacific's rebuild un-time-pressured (soak-test as long as needed, single controlled failback at the end) worked much better than trying to minimize downtime with multiple switches. - Verify every guest has a current PBS backup before wiping the disk, not just the ones with scheduled jobs — pvesm list PBS-New | grep backup/ct and cross-check against pct list. On 2026-07-19, two containers (GMUPS-27, TailscaleExitNode) had zero scheduled backups and needed one-off vzdump runs first. - If a disk-space problem caused the failure in the first place, check for other issues hiding behind it — a stale/dead NFS storage mount consuming 45GB of local disk was masking a secondary "no space left on device" backup failure on a different container entirely.

  1. Download Proxmox VE ISO from https://www.proxmox.com/downloads (get the exact current version/checksum — used 9.2-1 on 2026-07-19)
  2. Flash to USB (Rufus — worked fine on Windows/Brabham)
  3. Boot Pacific from USB, select Install Proxmox VE (Graphical)
  4. Double-check the Target Harddisk field before proceeding — on a NUC with both a SATA SSD and an onboard eMMC, the installer can default to the wrong one (/dev/mmcblk0, the 29GB eMMC, instead of the actual SATA SSD). Click into the Target Harddisk field and explicitly select the right drive.
  5. Install Proxmox:
  6. Hostname: proxmox.local (bare proxmox may not satisfy the installer's FQDN validation — .local suffix works)
  7. IP: 192.168.1.10 / Gateway: 192.168.1.1 — these can be entered correctly even if the ethernet cable is currently plugged into the wrong switch port/VLAN; the installer doesn't need live connectivity to complete, it just records the config. Just remember to move the cable to the correct main-LAN port before expecting the host to be reachable afterward.
  8. DNS: use the router (192.168.1.1) rather than AdGuard's own IP if AdGuard is one of the containers being restored — it doesn't exist yet on a fresh install
  9. Post-install, before restoring any guests:
  10. Re-add SSH key: web UI → Datacenter → pacific → Shell → mkdir -p ~/.ssh && echo "<pubkey>" >> ~/.ssh/authorized_keys
  11. Switch from the enterprise repo to no-subscription (the enterprise repo 401s without a paid key and blocks apt-get update/package installs, including kernel headers): 
    mv /etc/apt/sources.list.d/pve-enterprise.sources /etc/apt/sources.list.d/pve-enterprise.sources.disabled
mv /etc/apt/sources.list.d/ceph.sources /etc/apt/sources.list.d/ceph.sources.disabled
cat > /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
apt-get update
  12. Recreate storage: pvesm add pbs PBS-New --server <lotus-ip> --datastore <name> --username backup@pbs --password '<pw>' --fingerprint '<fingerprint>' and the equivalent pvesm add nfs for any NFS-backed storage (e.g. CCTV clips). The backup@pbs password is a separate PBS account from any personal login — see Pacific server doc if it's not in Bitwarden.
  13. If the host has any USB-attached peripherals doing host-level work (e.g. a UPS via NUT), reinstall and reconfigure that now, before restoring guests that depend on it. This is host-OS config, invisible to every VM/CT backup, and gets completely wiped by a fresh install. See Pacific server doc → UPS/NUT for the exact config.
  14. Restore VMs and LXC containers in priority order above (pct restore <id> 'PBS-New:backup/ct/<id>/<timestamp>' --storage local-lvm, qmrestore for VMs)
  15. If any LXC has hardware passthrough (GPU, Coral TPU, USB serial devices, etc.) via a lxc.hook.autodev script, check whether that script actually exists on the new host before starting the container — it's a host-side file, not part of the container backup, and a missing hook script causes the container to fail to start entirely (not just skip the passthrough). If missing, either recreate the script or replace it with static lxc.mount.entry lines in the container's .conf for each device — the latter is more durable. See Pacific server doc → Hardware Notes for a worked example (Coral TPU + GPU).
  16. If an NFS-backed storage (e.g. CCTV clips) shows "directory does not exist" or "stale file handle" on first mount after restore, umount -l it and let Proxmox remount fresh — the underlying export's fileid can change server-side between the backup and restore.
  17. Verify each service

Scenario C — Pacific hardware has died

If the NUC itself is dead and cannot be repaired quickly:

  • Home Assistant → migrate to the standby VM on Lotus (see HA Recovery → Scenario B)
  • Tailscale exit node → can be set up on another device temporarily
  • Frigate → doorbell camera will be offline until Pacific is restored or Frigate is moved

Post-Recovery Checklist

  • [ ] Proxmox UI accessible at https://192.168.1.10:8006
  • [ ] All VMs and LXC containers running
  • [ ] PBS storage connected and showing backups
  • [ ] Home Assistant accessible at http://192.168.1.12:8123
  • [ ] Frigate visible in Home Assistant
  • [ ] Tailscale exit node active
  • [ ] Backup schedule resumed

⚠️ This procedure has not been tested. Test it before you need it.