Backup NAS — Cooper¶

Last Updated: 2026-07-18 Status: ✅ Active Purpose: Backup target. Receives backups from Lotus. Minimal services to reduce unnecessary disk spin-up.

Hardware specs: inventory.md → Cooper

Access¶

Method	Detail
Web UI	http://192.168.1.60
SSH	`ssh root@192.168.1.60 -p 50123` (custom port, not default 22)

Array Configuration¶


Parity	1× 3.6TB
Disk 1	3.6TB
Disk 2	3.6TB
Usable storage	~7.2TB
Cache pool	1× 512GB SSD (2.5" SATA, bay 4)

Cooper now has an SSD cache pool, enabling Docker appdata to run off faster storage. All array data remains on spinning hard drives.

Docker Containers¶

Container	Status	Purpose
unraid-simple-monitoring-api	✅ Running	Exposes basic monitoring endpoint
SWAG	✅ Running	Standby reverse proxy — mirrors Lotus SWAG config
immich-postgres-replica	✅ Running	Streaming replica of Lotus Immich PostgreSQL

SWAG — Standby Reverse Proxy¶

Cooper runs a second SWAG instance configured identically to the primary on Lotus. It acts as a hot spare reverse proxy — if Lotus goes down, a failover script automatically updates Cloudflare DNS to point to Cooper's SWAG within ~10 minutes.

	Lotus (primary)	Cooper (standby)
Tailscale hostname	`swag`	`swag-cooper`
Tailscale IP	100.106.140.33	100.126.183.94
Cloudflare DNS	`*.djchome.uk` points here normally	`*.djchome.uk` points here on failover

Config Sync¶

SWAG config files are rsync'd from Lotus to Cooper hourly via a Lotus User Script (swag-sync-to-cooper). This keeps nginx proxy configs and SSL certificates in sync without manual intervention.

⚠️ Tailscale State Directory Exclusion¶

The rsync job must exclude the Tailscale state directory inside SWAG's appdata:

/mnt/cache/appdata/swag/.tailscale_state/

If this directory is copied across, both SWAG instances will share the same Tailscale node ID, causing a conflict — the primary Lotus instance will be kicked off the Tailnet and will lose external connectivity.

See Lessons Learned for full details.

Note: If Cooper's SWAG ever needs to re-authenticate with Tailscale (e.g. after clearing .tailscale_state), the browser auth flow may hang. Use a pre-generated Tailscale auth key instead: add TAILSCALE_AUTHKEY=tskey-auth-xxxxx as a container environment variable.

Automatic Failover Script¶

A script (swag-failover) runs every 5 minutes on Cooper via User Scripts. It:

Checks whether Lotus SWAG (100.106.140.33) is reachable on HTTPS
After 2 consecutive failures (~10 minutes of downtime), updates the Cloudflare *.djchome.uk wildcard A record to point to Cooper (100.126.183.94)
When Lotus recovers, automatically fails back and restores the DNS record

State and logs are stored at /mnt/cache/appdata/swag-failover/:

File	Purpose
`state`	`primary` or `failover`
`failures`	Consecutive failure counter
`failover.log`	Timestamped event log

Cloudflare record details: - Zone ID: d2e47a211146c145dc79f28a46800b4e - Record ID: a3fe4772cf198a5ac7af9aec1d1e7f38 - The API token for DNS updates is stored inside the failover script on Cooper

Immich PostgreSQL — Streaming Replica¶

Cooper runs a read-only PostgreSQL replica of the Immich database on Lotus. This provides data protection for photos — if Lotus's database is lost or corrupted, Cooper has a continuously-updated copy.


Container	`immich-postgres-replica`
Image	`ghcr.io/immich-app/postgres:16-vectorchord0.3.0-pgvectors0.3.0`
Data path	`/mnt/cache/appdata/immich_postgresql`
Primary	`192.168.1.80:5452` (via socat relay on Lotus — see Primary NAS doc)
Replication user	`replicator`

Replication config lives in /mnt/cache/appdata/immich_postgresql/postgresql.auto.conf and standby.signal. The replica connects to Lotus on port 5452 (the socat relay), not port 5432 directly.

If replication breaks¶

Check container logs: docker logs immich-postgres-replica --tail 50
Verify Lotus socat relay is running: docker ps | grep immich-pg-relay on Lotus
Check connectivity: bash -c 'echo > /dev/tcp/192.168.1.80/5452' && echo OK from Cooper
If the replica has diverged too far, re-run pg_basebackup and restart the container

Backup Role¶

Cooper is the destination for backups from Lotus:

Source	Method	What's backed up
Lotus	LuckyBackup	Data shares
Lotus	Appdata Backup plugin	Docker appdata

Cooper also sends its own Docker appdata to Lotus via Appdata Backup.

Planned: Local Documentation Mirror¶

Cooper is the planned host for the local docs mirror. Once configured:

docs.djchome.uk will resolve to Cooper on the home network (via split DNS)
Cooper will sync from GitHub and serve the MkDocs site locally
Gabriela can use the same URL whether at home or away

See TODO.md for setup tasks.

Recovery¶

See Backup NAS Recovery for restore procedures.

⚠️ Cooper does NOT restart itself after a power cut (confirmed 2026-07-29)¶

Its BIOS is not set to power on after AC restore, and Wake-on-LAN does not work either. After any power interruption Cooper stays down until somebody physically presses its power button — silently, since it is the backup NAS and nothing depends on it minute to minute.

Note that a graceful shutdown is not the same event as a power cut: "restore on AC loss" will not bring back a machine that was cleanly powered off, even if mains is then cycled.
Its power comes via Pw05 (192.168.30.59, Tasmota) — see the Smart Plug Register. Any work on that plug takes Cooper down, so shut the array down gracefully first.
Safe for a long outage: Immich's Postgres on Lotus has no replication slots, so the paused replica cannot accumulate WAL and fill Lotus's cache. But there is no backup target while Cooper is down — the nightly appdata backup, the 07:00 USB flash copy and luckyBackup all fail.
On return, mdcmd status reporting mdNumDisabled=1 / mdNumInvalid=1 is benign — it counts the empty slot 3 (DISK_NP). All real disks read DISK_OK.
Fix worth doing: enable "restore on AC power loss" in Cooper's BIOS.

Troubleshooting¶

Shares disappear — "no exportable user shares" (recurring, root cause found 2026-07-18)¶

Symptom: Cooper's Shares page shows "There are no exportable user shares." Anything mounting Cooper over SMB (e.g. Lotus's /mnt/remotes/192.168.1.60_*) goes stale and fails to remount with NT_STATUS_BAD_NETWORK_NAME. The array itself and individual disks (/mnt/disk1, /mnt/disk2, /mnt/cache) stay healthy throughout — this is not an array or driver failure, despite how it looks from the Shares page.

Root cause: the /mnt/user FUSE mount — the shfs process that pools disk1+disk2+cache into user shares — silently dies at an unpredictable point during uptime, with no corresponding kernel or hardware event logged. mount still lists /mnt/user, but any access returns Transport endpoint is not connected, breaking every user share since they all live under /mnt/user. /mnt/user0 (the same pool minus the cache drive) stayed healthy both times this happened, making the cache drive's involvement in the pool the leading suspect — but the actual trigger for the shfs crash itself is still unconfirmed.

A plain Array Stop/Start from the GUI does not fix it. Unraid's stop sequence doesn't force-unmount a wedged FUSE mount, so on the following start, mkdir /mnt/user fails (directory already exists in the broken transport state) and the new shfs process dies immediately trying to chdir into it. Historically only a full reboot cleared it, since rebooting wipes all mount state.

Faster fix without a reboot (confirmed working 2026-07-18, via SSH):

umount -l /mnt/user
/usr/libexec/unraid/shfs /mnt/user -disks 7 -o default_permissions,allow_other,noatime -o remember=330
/etc/rc.d/rc.samba restart

The -disks 7 bitmask matches Cooper's current 3-disk + cache config — if the disk layout changes, confirm the correct invocation via grep 'shfs /mnt/user' /var/log/syslog (emhttpd logs the exact command it uses on every array start). Verify with df -h /mnt/user and check /var/log/syslog for no further canonicalize_connect_path or Transport endpoint entries.

To catch the actual crash trigger next time it happens: grab a diagnostics zip (Tools → Diagnostics) as soon as it's noticed, before applying the fix above — the in-RAM logs and current FUSE state are the most useful evidence and are lost once the mount is remounted or the box is rebooted.

See Lessons Learned for the full investigation.

Full recovery checklist (updated 2026-07-19 — fixing `/mnt/user` alone is not enough)¶

Doing the shfs fix above gets the mount healthy again, but that alone does not fully restore SMB access. Two more things commonly need fixing on top of it — check all three before assuming the fault has recurred:

/mnt/user FUSE mount — the fix above. Verify with df -h /mnt/user.
Cooper's own Samba share export list can end up empty. Check cat /etc/samba/smb-shares.conf on Cooper — if it's 0 bytes, Samba has no shares defined at all (smbclient -L localhost -U guest -N will show only IPC$), and any client trying to mount a share gets NT_STATUS_BAD_NETWORK_NAME. This happens when emhttpd tries to regenerate the share export list at the exact moment /mnt/user isn't mounted yet (the same race as the shfs crash itself) — e.g. during the Array Stop/Start used to fix #1. Editing a share in the GUI and clicking Done without changing a value does not force a rewrite. The reliable fix is another Array Stop → Start now that /mnt/user is already stable (no race this time) — confirmed working 2026-07-19; it briefly bounces swag and immich-postgres-replica again. Verify with cat /etc/samba/smb-shares.conf (should list [appdata], [backups], [data], [immich]) and smbclient -L localhost -U guest -N.
Client-side stale CIFS mounts don't self-heal. Even after #1 and #2 are fixed on Cooper, a client that had the share mounted through the outage (e.g. Lotus, at /mnt/remotes/192.168.1.60_*) can be left with a wedged mount: mount still lists it, it may show a different SMB dialect than a working mount (e.g. vers=3.1.1 vs vers=3.0), but ls/stat on it returns "No such type of file or directory" and ls -la on the parent shows d????????? for that entry. Unraid's Unassigned Devices "Mount" button sees the stale entry and does nothing, since as far as the OS is concerned it's already mounted. Clear it directly on the client: umount -l /mnt/remotes/192.168.1.60_<share>, then remount.

The UI Mount button is not required — this is doable entirely from the CLI on Lotus (confirmed 2026-07-29):
```
umount -l /mnt/remotes/192.168.1.60_appdata
umount -l /mnt/remotes/192.168.1.60_backups
/usr/local/sbin/rc.unassigned mount "//192.168.1.60/appdata"
/usr/local/sbin/rc.unassigned mount "//192.168.1.60/backups"
```
rc.unassigned mount with no argument fails (Fail: device not defined) — it needs the share path exactly as written in /boot/config/plugins/unassigned.devices/samba_mount.cfg. Verify with stat on each mountpoint, and check the nightly flash backup is present under .../backups/lotus/justtheusb/.

See Lessons Learned for the full investigation.

Auto-recovery watchdog (installed 2026-07-22)¶

Because the shfs crash keeps recurring and used to go unnoticed for days, Cooper now runs a User Script watchdog that self-heals it. It does not fix the underlying bug (still unknown why shfs dies), but it turns a multi-day silent outage into a self-healing blip of a few minutes, and — just as usefully — records the exact time of death so the trigger can eventually be correlated.

Where: Settings → User Scripts → shfs-user-watchdog (file: /boot/config/plugins/user.scripts/scripts/shfs-user-watchdog/script). Runs every 3 minutes.
What it does: checks /mnt/user is a live, readable mountpoint; if it's wedged, logs the timestamp + evidence and runs the Layer-1 recovery (umount -l → shfs restart → rc.samba restart). The shfs command isn't hard-coded — it caches the live invocation to /boot/config/shfs-user-cmd.txt on every healthy run, so it auto-adapts if the disk layout changes.
Safeguards: it won't act during a legitimate array stop or boot (it first checks /mnt/disk1 is mounted), and a 15-minute cooldown stops a failed recovery from hammering Samba in a loop — instead it logs "MANUAL intervention required".
Log to check after any recurrence: /mnt/cache/appdata/shfs-watchdog/watchdog.log — this gives the precise crash timestamp.
Scope: it fixes Cooper (layers 1 + 2 above). It does not touch Lotus's client-side stale mounts (layer 3) — if Lotus still can't see the shares after the watchdog has healed Cooper, clear them manually on Lotus per step 3 above.

Investigation on the 3rd occurrence (2026-07-22) ruled out memory (plenty free, no OOM), kernel/hardware events (none logged), and the cache drive (btrfs error counters all zero) — so the "cache drive is the suspect" theory isn't supported by that occurrence's evidence. A brief UPS on-battery power blip landed ~2 min before the crash window, but a second identical blip 90 min later caused no crash, so it's not a reliable trigger on its own. Still looks like an shfs software bug in Unraid 7.3.x tripped by something in the nightly-backup window.

Notes¶

SSD cache pool added April 2026 — Docker appdata now runs off cache rather than spinning disks
Static HTML docs site would be read-only and fine on spinning disks

Backup NAS — Cooper¶

Access¶

Array Configuration¶

Docker Containers¶

SWAG — Standby Reverse Proxy¶

Config Sync¶

⚠️ Tailscale State Directory Exclusion¶

Automatic Failover Script¶

Immich PostgreSQL — Streaming Replica¶

If replication breaks¶

Backup Role¶

Planned: Local Documentation Mirror¶

Recovery¶

⚠️ Cooper does NOT restart itself after a power cut (confirmed 2026-07-29)¶

Troubleshooting¶

Shares disappear — "no exportable user shares" (recurring, root cause found 2026-07-18)¶

Full recovery checklist (updated 2026-07-19 — fixing /mnt/user alone is not enough)¶

Auto-recovery watchdog (installed 2026-07-22)¶

Notes¶

Full recovery checklist (updated 2026-07-19 — fixing `/mnt/user` alone is not enough)¶