Lessons Learned¶

Purpose: Mistakes, discoveries, and things worth remembering. Future-you will thank current-you.

After solving a problem or learning something valuable, add an entry using the template at templates/lesson-learned.md.

General Principles¶

Documentation: - Document while you work, not after — you'll forget the details - Document WHY you made a decision, not just WHAT you did - If a recovery procedure isn't tested, it's fiction

Unraid: - Always stop the array cleanly before shutting down - Parity checks matter — schedule them and don't skip - Community Applications (CA) plugin versions can drift — pin versions in compose files - Backup the USB boot drive — losing it means reinstalling Unraid config from scratch - Flash Backup plugin is worth using

Proxmox: - VMs are easier to backup and restore than LXC containers - Always enable QEMU guest agent - Snapshots are not backups — they live on the same disk - Give VMs slightly more resources than you think they need

Docker / Containers: - Use named volumes or bind mounts for persistent data — never rely on container storage - Pin image versions — :latest will bite you eventually - docker logs <name> is always your first debugging step

Bash: - Never use ! in passwords passed via docker exec ... psql -c "..." — bash treats ! as a history expansion character inside double-quoted strings, causing event not found errors - Use passwords without ! for anything passed on the command line, or run set +H first to disable history expansion

Home Assistant Notifications: - Always route notifications through script.notify_dan or script.notify_household — never call notify.mobile_app_* directly - Both scripts have sticky: true baked in so notifications stay in the Android tray after tapping — this is intentional so you can read the full message without it disappearing - On Android, sticky: true means the notification persists after tap but can still be manually swiped away - Use trigger.to_state.name (not trigger.to_state.attributes.friendly_name) to get an entity's friendly name in automations — the attributes dict doesn't reliably contain it in modern HA - Capture trigger context in a variables: block before calling scripts — trigger.* is available in automation action templates but not inside script execution context

Networking: - Document your firewall rules — you will forget them - Changing DNS settings breaks things in non-obvious ways — change carefully - Test connectivity after every network change before assuming it worked

Backups: - A backup you haven't tested restoring is not a backup - Offsite matters — local backup doesn't protect against fire, flood, or theft - Retention policies matter — check you're not filling up storage silently

Log¶

2026-08-03: Shelfmark forgot its settings on every update — the rebrand moved the config volume¶

What Happened¶

The book downloader container on Lotus kept losing all of its settings — logins, Prowlarr and Hardcover configuration, download sources — but only intermittently. A plain restart was fine; the settings vanished after Unraid updates.

The container's Docker image is still named ghcr.io/calibrain/calibre-web-automated-book-downloader, but the project has been rebranded to Shelfmark and the GitHub repo renamed calibrain/calibre-web-automated-book-downloader → calibrain/shelfmark. As part of that restructure, all state moved into a new /config volume (CONFIG_DIR, documented in the image's own docs/configuration.md): settings.json, users.db, .flask_secret, and 17 per-plugin JSON files.

The Unraid template predated the change and mapped only the old /var/log/cwa-book-downloader appdata path — which now contains nothing but stale logs. /config had no host mapping at all, so it lived in the container's own writable layer. That layer survives docker restart but is destroyed whenever the container is recreated, which is exactly what an Unraid update does.

The same rebrand had also silently broken the container's icon: the template's <Icon> URL pointed into the old repo path and returned 404.

The Fix¶

Copied the live /config out of the container to /mnt/user/appdata/shelfmark/ (owned 99:100) before touching anything, so the current settings survived.
Added a /config → /mnt/user/appdata/shelfmark/ path to /boot/config/plugins/dockerMan/templates-user/my-Shelfmark.xml, then applied it from the Docker tab (a template edit does nothing until the container is applied in the UI).
Repointed <Icon> at https://raw.githubusercontent.com/calibrain/shelfmark/main/src/frontend/public/logo.png, pre-seeded Unraid's icon cache, and kept a copy on the flash drive at /boot/config/plugins/dockerMan/images/Shelfmark-icon.png — the cache directory under /var/local/emhttp/ is in RAM and is lost on every reboot.

Backups: my-Shelfmark.xml.bak-20260803, my-Shelfmark.xml.bak-20260803-icon.

Follow-up, 2026-08-04 — reverse proxy and dashboard¶

The rebrand had left a third loose end: the SWAG vhost was still cwbd.subdomain.conf, and it was a copy of the calibre-web sample with the port changed. It worked, but under the old name and carrying /opds/ and /kobo/ location blocks that mean nothing to a downloader.

Replaced with a clean shelfmark.subdomain.conf (shelfmark.* → 192.168.1.80:8084); https://shelfmark.djchome.uk verified 200. The old file is kept as cwbd.subdomain.conf.bak-20260804-shelfmark — SWAG only includes *.subdomain.conf, so a .bak suffix is enough to retire a vhost. cwbd.djchome.uk no longer resolves to anything.

Also added to Homepage under Downloads (services.yaml, backup services.yaml.bak-20260804-shelfmark), with the icon pulled from the same repo URL the Unraid template now uses — so if upstream moves the logo again, the Unraid icon and the dashboard icon break together.

No DNS or certificate work was needed: *.djchome.uk is a wildcard record pointing at the Tailscale IP, and SWAG holds a wildcard cert (SUBDOMAINS=wildcard, VALIDATION=dns). Any new subdomain here is one proxy-conf and a reload.

Lesson Learned¶

"Forgets settings between starts" almost always means a volume is missing, not that the app is broken — and the tell is that a restart is fine while an update is not. Container-layer state is invisible until something recreates the container.

When an upstream project rebrands, audit the whole template, not just the bit that broke. One rename moved the config volume and broke the icon URL; both looked like unrelated faults.
The Docker image name staying the same is not evidence that nothing moved.
GitHub redirects a renamed repo, but not old raw.githubusercontent.com asset paths — those just 404.
The image ships its own documentation. docker exec <name> ls /app/docs answered this faster than any web search would have.
General rule, already in the Docker principles above and proved again here: never rely on container storage for anything you would miss.

2026-08-03: A 60-book import took Calibre-Web down, and the sync that "succeeded" silently dropped 10 books¶

What Happened¶

Ingesting 60 ebooks into Calibre-Web Automated triggered a full-library duplicate scan, which deadlocked the web worker. CWA was unreachable for ~10 minutes and needed a container restart. Separately, syncing those books to the Kindle reported success but delivered only 50 of 60 — and a third sync reported "nothing to do".

The Problem¶

The outage. duplicate_scan_frequency was after_import, so every ingest queued a duplicate scan. With 576 books it ran a full scan (min_book_id=None) rather than an incremental one, and wedged. Symptoms: container Up (unhealthy), port still listening, log output completely frozen and CPU at 2.9% — a busy scan looks quite different from a stuck one.

The silent data loss. KOReader's OPDS sync caps downloads at sync_max_dl (default 50), but fillPendingSyncs advances its last_download watermark to the newest feed entry regardless of whether the list was truncated. The overflow ends up behind the watermark and is never fetched. Sync reports success either way.

The Solution¶

Set duplicate_scan_frequency to manual — detection still available on demand, just not on every import
Wound last_download back to the last known-good book and raised sync_max_dl to 100, then verified device count against shelf count

Lesson Learned¶

A background job that reports success is not evidence it did the work — check the resulting count against the source of truth.

Frozen logs plus low CPU means stuck, not busy. Compare log line counts a few seconds apart before deciding to wait it out.
Anything that triggers "on import" should be assumed to run at full scale on the whole dataset, not incrementally.
After any bulk sync, compare item counts at both ends. Two independent bugs here were only visible as a count mismatch.
When comparing two lists of files, compare on a stable identifier (title), not on constructed filenames. Filename comparison produced 41 false "missing" results; a careless len(x) >= 6 guard then produced 2 false negatives in the opposite direction.

Documentation Updated¶

Kindle / KOReader Ebook Sync — full service documentation

2026-08-02: Multiroom audio — four things that each broke it silently¶

What Happened¶

Setting up whole-home audio with Lyrion Music Server (LMS) on Lotus, a Squeezelite player on the Kitchen Pi, and the Sony HT-A9 reached over Chromecast. Four separate problems each produced a system that looked configured but did not play audio, and none of them announced themselves.

The Problems¶

1. LMS in bridge/NAT mode broke Chromecast discovery. Docker bridge networking breaks mDNS, so the Castbridge plugin could not find the Sony at all. The container needs br0 with its own LAN IP (192.168.1.83). That also sidesteps the port 9000 clash with Mealie.

2. Castbridge served audio on a random high port. It runs its own HTTP server, separate from LMS, to feed audio to Chromecast devices — by default on a random high port (e.g. http://192.168.1.83:35845/bridge-1.mp3), which the IoT → main LAN firewall blocked. The Sony connected and then sat silent. Fixed by setting "port starts from" to 35000 and opening that port.

3. The UniFi rule matched on source port. The firewall rule was written with the "LMS Ports" group as the source port, which matches only traffic originating from those ports. Source ports are always ephemeral. Source must be Any; the port group belongs on the destination.

4. The wrong HiFiBerry overlay, and a swallowed config file. The MiniAMP is a PCM5102A, so it needs dtoverlay=hifiberry-dac — hifiberry-dacplus is for the I2C-controlled PCM5122 and produces no audio. dtparam=audio=off is also needed, or onboard audio confuses ALSA device ordering and Squeezelite grabs the wrong device. Separately, Debian's Squeezelite init script silently ignores SL_OPTIONS in /etc/default/squeezelite, so the -s <server> flag never reached the binary — a native systemd unit is required. And -V "" must be passed because the PCM5102A has no hardware ALSA mixer.

Lesson Learned¶

When a media endpoint connects but plays nothing, suspect a second port you didn't know existed. Control channels and audio streams are frequently on different ports, and only the control channel is the one you configured.

Docker containers that need mDNS discovery must be on br0, not bridge
A firewall rule with a port group in the source field matches almost nothing
Check that a distro init script actually reads its /etc/default/ file before trusting it — write a systemd unit instead
The CastBridge plugin is not in the LMS plugin manager (the SourceForge repo URL changed) — install it by hand from the GitHub releases
Snapcast and Music Assistant were tried first and were unreliable; LMS + Squeezelite is the battle-hardened replacement. Do not revert.

Documentation Updated¶

Multiroom Audio

2026-07-22: Lotus rebooted twice with zero trace in its own logs — the smart plug between the UPS and the NAS was crashing¶

What Happened¶

Hours after an Unraid 7.3.1 → 7.3.2 upgrade, Lotus rebooted without warning. The first reboot came ~6 minutes after the upgrade finished (14:54:12); a second, seemingly unrelated one followed ~35 minutes later (~15:38).

Both looked identical from Lotus's side: the syslog simply stopped mid-operation, with no trace of a cause anywhere.

What Was Ruled Out¶

Every one of these came back clean, which is what made the incident so hard to place:

No kernel panic — EFI pstore checked, empty
No OOM — plenty of free RAM, nothing in the log
No USB or disk I/O errors
No hardware errors — mcelog empty, no MCE/EDAC events
No kernel thermal-throttle event
No UPS on-battery event — upsc/NUT logged nothing at all; mains never actually dropped

A CPU-thermal theory was tried first and disproven. Lotus's i3-N305 was observed hitting 80–81 °C during the post-upgrade Docker startup burst, which looked damning. But HA's own Lotus CPU temperature / Lotus CPU usage history showed temps already fluctuating 70–85 °C both before and after the crash moment with no distinguishing spike, and CPU usage was never pegged. The correlation was coincidence.

Root Cause¶

HA's Tasmota sensors for the plug feeding Lotus (sensor.lotusserverplug_restart_reason, sensor.lotusserverplug_last_restart_time) recorded the answer:

A "Software Watchdog" reset at 15:03:16 — 38 seconds before Lotus's kernel began booting again at 15:03:54
A second "Software Watchdog" reset at 15:38:03 — 3 seconds before Lotus's second reboot began at ~15:38:06

The smart plug's own firmware was crashing and resetting, dropping its relay and cutting mains to Lotus.

Why Every Log Was Silent¶

The fault sat between the UPS and the NAS. That single fact explains both halves of the mystery:

The UPS output never dropped, so NUT had nothing to report — the UPS was fine, and so was the mains feeding it
Lotus's PSU lost power outright, so its OS had no opportunity to log anything — the interruption was entirely upstream of anything the kernel could observe

Lesson¶

When a machine dies with total log silence and the UPS reports nothing, suspect everything between the UPS and the machine. A UPS proves mains was present at its output — it says nothing about what happened downstream. Any device in that path (smart plug, extension lead, PDU) is a single point of failure that no log on either side will implicate.

Corollary: power-monitoring smart plugs often expose their own restart reason and last-restart time. Those sensors are the cheapest possible evidence for this class of fault, and they cost nothing to check first.

Resolution¶

2026-07-22 (immediate): the smart plug was bypassed, connecting Lotus directly to the UPS — accepting temporary loss of its power-monitoring entities in HA.
2026-07-28/29 (permanent): replaced with a Third Reality UZ1 (Zigbee, joining via the existing Sonoff bridge / Zigbee2MQTT rather than its own WiFi stack), removing the WiFi-reconnect / MQTT-broker-restart instability class entirely. metering_only_mode: on locks the relay so software can never cut power to Lotus again. See Smart plug failures track load, not age for the wider pattern across this plug batch.
The failed unit was later recommissioned as the spare Pw07, so its old sensor.lotusserverplug_* entities are alive again and must not be deleted.

Unrelated Change Made the Same Day¶

The Docker autostart stagger was applied while the thermal theory still looked plausible. It was not the fix, but it was kept as general hygiene — the upgrade reboot had restarted all 51 containers plus the HA VM simultaneously. /mnt/cache/system/unraid-autostart now spreads startup over ~2 minutes: Swag → AdGuard-Home → adguardhome-sync → Plex first, then all DB/cache containers, then their dependent apps, then everything else at 1s intervals. Two pre-existing ordering bugs were fixed in passing — dawarich/dawarich-sidekiq were starting before their own postgis/redis, and paperless-ngx before redis-paperless. Backup: /mnt/cache/system/unraid-autostart.bak-20260722.

2026-07-29: Smart plug failures track load, not age — and three plugs were not what they claimed to be¶

What Happened¶

After two LocalBytes smart plugs died in one week, the whole plug fleet was audited. Querying every Tasmota device directly over HTTP (Brabham can reach the NoT VLAN, so this was authoritative rather than inferred from Home Assistant) found the fleet was larger than documented, wrongly mapped in the records, and running 2021 firmware with telemetry settings that hid most of what it measured.

The Problem¶

Three plugs were misidentified in the documentation.

IP	UniFi alias	Tasmota `DeviceName`	HA entity ID	Actually
`.52`	DishwasherSmartPlug	Living Room Media Plug Left	`switch.dishwasher`	Living Room switch/media
`.54`	MintNucSmartPlug	brmSmartPlug	`switch.office_nuc_plug`	a shredder
`.57`	Living Room Media Plug Right	Kettle Smart Plug	`switch.kettle`	the kettle
`.59`	TurboTrainerPlug	QNAP Plug	`switch.qnap_plug`	Cooper, the backup NAS

.59 mattered most: the docs called it the turbo trainer, but it feeds the backup NAS — so the junk 146 kWh "today" reading previously blamed on the turbo trainer was actually this plug, and any work on it risked a NAS.

The cause is structural: four independent naming layers, with HA entity IDs frozen at each plug's original role while device names track the current one. Entity IDs actively lie — sensor.living_room_media_plug_left_* belongs to the washer/dryer.

The failure pattern is about current, not age. Every failed unit carried a heavy or continuous load; the light-duty plugs of identical vintage are fine:

Plug	Load	Outcome
Lotus server plug	NAS, continuous	died — rebooted Lotus twice
Boiler plug	boiler	died — 5 h no hot water, then jammed WiFi ch6
Desktop plug (own label: "power cuts when gaming")	gaming PC	died
Kettle	3 kW bursts	flapped, recovered by firmware upgrade
TV / media / Cooper plugs	tens of watts	still healthy

Telemetry was hiding events. Every plug except the kettle still had TelePeriod 300 and Sleep 50 — so any event shorter than five minutes logged as 0 W, and dynamic sleep drove MQTT churn. The TV plug had 3,727 MQTT reconnects against 4 WiFi reconnects in 11 days; the kettle, after its fix, had 2.

The Solution¶

All five WiFi plugs brought to firmware 15.5.0 with TelePeriod 60, PowerDelta1 20, Sleep 0, SetOption56 1 and local NTP. A permanent tagging convention and register was adopted so identity stops drifting.

Practical gotchas met along the way:

The command is PowerDelta1, not PowerDelta on this firmware — the unindexed form silently reports nothing.
A multi-command Backlog silently applied only the first setting. Issue settings individually and verify each.
SetOption57 (periodic roam to the strongest AP) does not work. It was enabled fleet-wide, yet two plugs sat on an AP across the house at -88 dBm indefinitely — one for its entire 147-day uptime. SetOption56 (scan at boot) does work, so after anything that moves an AP, Tasmota devices need a reboot to re-home; they will not do it themselves. Both plugs jumped to -62 and -57 dBm after a reboot.

Lesson Learned¶

Smart plug failures correlate with the load carried, not the age of the plug. A plug on a NAS, a boiler or a gaming PC is doing far harder work than one on a TV, and it fails first. Anything continuous or critical deserves a plug whose relay cannot fail open — a Zigbee unit in metering-only mode, or no plug at all.

A constant load on a critical path shouldn't be metered at all. The living-room network switch drew a steady 6–7 W (19 VA, PF ~0.35), comfortably inside its 12 W rating. Measuring that forever adds a relay in series with the uplink to the whole office — including the NAS running Home Assistant — to re-learn a number that never changes. Model it as a constant instead and plug the switch into the wall.
When identifying a device, trust the device name and area — never the entity ID or the UniFi alias. Both are stale by design.
BootCount is only a health metric for continuously-powered devices. One plug showed 1128 boots against ~250–350 for identical siblings and looked like it was dying. It sits on an office power strip switched with occupancy — ~0.73 boots/day is one per working day. Comparing it against always-on peers produced a completely wrong diagnosis.
Read Rx Rate and AP/Client Signal Balance in UniFi's client detail before condemning a radio. The desktop plug associated at a healthy -47 dBm yet passed no traffic, at a 1.00 Mbps receive rate with balance reported as Poor — the AP could not hear the client even though the client heard the AP. That asymmetry identifies a failing transmitter; neither RSSI alone nor a ping test shows it.
A factory reset that "does nothing" may have worked perfectly. That same plug returned as a blank tasmota-111123-4387 — Tasmota's default hostname — and rejoined WiFi on its own, because a button reset keeps WiFi credentials. It was declared dead twice before the evidence was read properly. The fault was below the configuration layer, where no reset reaches.
Never leave a suspect plug powered "for testing". The boiler plug was left in a living-room socket after it died and jammed 2.4 GHz channel 6 for three days before anyone connected the two. Unplug on diagnosis.

Documentation Updated¶

inventory.md — full register, tagging convention, standard Tasmota settings
CLAUDE.md (Homelab Support project) — audit detail and per-plug history

2026-07-29: Cooper does not come back on its own after a power cut¶

What Happened¶

The plug feeding Cooper (the backup NAS) needed a firmware upgrade, which meant reboots that could drop its relay. Cooper was shut down gracefully first, the plug was flashed, and its relay returned ON — but Cooper stayed dark.

The Problem¶

Cooper's BIOS is not set to power on after AC restore, and Wake-on-LAN does not work on it either. Neither had ever been tested.
The graceful shutdown put it in soft-off (S5). "Power on after AC restore" only fires on a real AC-loss event — and the plug's relay may never have opened during the OTA at all, so Cooper saw nothing to recover from.
Net effect: after any power interruption the backup NAS stays silently down until somebody presses its power button.

Afterwards, Lotus's CIFS mounts to Cooper were wedged exactly as expected — ls -la /mnt/remotes/ showing d????????? for both shares.

The Solution¶

A physical power-button press. Then the mounts were restored entirely from the CLI — previous notes said the Unassigned Devices UI Mount button was required, which is not true:

umount -l /mnt/remotes/192.168.1.60_appdata
umount -l /mnt/remotes/192.168.1.60_backups
/usr/local/sbin/rc.unassigned mount "//192.168.1.60/appdata"
/usr/local/sbin/rc.unassigned mount "//192.168.1.60/backups"

rc.unassigned mount with no argument fails (Fail: device not defined) — it needs the share path exactly as written in /boot/config/plugins/unassigned.devices/samba_mount.cfg.

Lesson Learned¶

Test whether each machine actually restarts itself after a power cut, before you need it to. A UPS protects against short outages; it does not help if the machine never comes back. Cooper is the backup NAS, so a silent failure to restart could go unnoticed for a long time — precisely the scenario backups exist for.

A graceful shutdown and a power cut are different events to a BIOS. "Restore on AC loss" will not resurrect a machine that was cleanly powered off, even if mains is then cycled.
Checked and worth knowing: Immich's Postgres on Lotus has no replication slots, so a paused Cooper replica cannot accumulate WAL and fill Lotus's cache. Cooper can stay off for a long period safely — but there is no backup target while it is down.
On return, mdcmd status reporting mdNumDisabled=1 / mdNumInvalid=1 is benign — it counts the empty slot 3 (DISK_NP). All real disks read DISK_OK.

Documentation Updated¶

inventory.md
CLAUDE.md (Homelab Support project)

2026-07-29: A Powercalc sensor reading `unavailable` usually means its source device is offline¶

What Happened¶

Home Assistant's Powercalc integration was believed to be largely broken. An audit found 26 of 106 Powercalc entities unavailable — but only some of that was Powercalc's fault.

The Problem¶

Four different causes were tangled together:

Sources genuinely offline — the Powercalc sensor was behaving correctly and the fix was to the device, not the config.
Sources renamed or removed — entries still pointing at light.bedside_left_old, light.tv_left, switch.turbotrainer, all long gone.
A wrong mapping — "Gabriela's Room Bedside Left" sourced from light.loft_bedside_left, a light in a different room, silently mis-attributing power.
Duplicates from hardware swaps — a bulb replaced (Hue → innr) left the old entry orphaned beside a working new one.

One thing that looked broken but was not: three entries had entity_id: sensor.dummy, which does not exist. That is Powercalc's internal placeholder for device-based sensors (created against a device rather than an entity, alongside availability_entity and multi_switch). Those sensors worked fine — Hue Bridge 1.74 W, Plus2PM-Boiler 1.32 W.

The Solution¶

Eight dead entries deleted via DELETE /api/config/config_entries/entry/<id> (works with a normal long-lived token, returns {"require_restart":false}). Result: 106 → 92 entities, unavailable 26 → 12, and every remaining unavailable sensor has a deliberately-offline source.

For constant-draw devices, daily_fixed_energy needs no source entity at all and creates both a power and an energy sensor:

powercalc:
  sensors:
    - name: Living Room Switch
      daily_fixed_energy:
        value: 7
        unit_of_measurement: W

Lesson Learned¶

Check the source device before concluding the integration is broken. Powercalc faithfully reports unavailable when its source is unavailable, so a wall of unavailable sensors usually means a wall of offline devices — a completely different fix.

Deleting a config entry also destroys its long-term statistics. Worth pausing over for any entry with real history behind it.
Verify a suspected placeholder before "fixing" it. sensor.dummy looks like an obvious bug and is not.
A device-level audit is cheap: count entities with platform == powercalc in core.entity_registry and compare their states.

Documentation Updated¶

CLAUDE.md (Homelab Support project) — Powercalc section

2026-07-29: SSH to Home Assistant failed with "Corrupted MAC on input" — one MAC algorithm was broken, not the key¶

What Happened¶

ssh homeassistant from Brabham had been failing with:

Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection to 192.168.1.12 port 22: message authentication code incorrect

This had been worked around for months by assuming the key wasn't installed and that the HA SSH add-on was password-only — so config changes were made by pasting YAML into the HA web editor instead.

Root Cause¶

Not authentication, and not the network. The failure happens after key exchange completes, which is the tell. Client and server were negotiating umac-128-etm@openssh.com, which produces corrupted MACs against the Advanced SSH & Web Terminal add-on's OpenSSH 10.3.

The control that cracked it: ssh Lotus from the same machine worked perfectly. That ruled out the SSH client, the key, and the general network in one step.

Fix¶

Pin a known-good MAC in the Host homeassistant block of ~/.ssh/config:

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com

Forcing an AES-GCM cipher (-o Ciphers=aes256-gcm@openssh.com, which has an implicit MAC) also works, and is a good one-liner for diagnosing from a machine without the config entry.

The key was fine all along, and sudo is passwordless.

Lessons¶

"Corrupted MAC on input" is an algorithm-negotiation problem, never an auth problem. If it authenticated far enough to fail here, the key works.
Always test a second host before blaming the client. One working control connection eliminates most of the search space instantly.
ssh -vv and grep for kex: — the negotiated cipher/MAC line tells you exactly what to override.
Piping files to a host via PowerShell injects a UTF-8 BOM. Get-Content -Raw | ssh host "sudo tee -a file.yaml" writes \xef\xbb\xbf at the join and breaks YAML parsing with a confusing expected <block end>, but found '?'. Strip it, or use a here-doc.

2026-07-29: Ring alarm sensor bypass works — but ring-mqtt won't tell you what it bypassed¶

What Happened¶

Wanted to arm the house alarm in Home mode overnight while leaving the bedroom window open, and be told when a sensor had been bypassed or when arming failed outright.

What Was Already There¶

ring-mqtt already exposes bypass as a per-sensor select.<sensor>_bypass_mode entity with three options:

Mode	Behaviour on arm
`Never`	Open sensor blocks arming (default)
`Faulted`	Bypassed only if open at arm time
`Always`	Bypassed unconditionally

Setting the bedroom window to Faulted solved the actual problem in about ten seconds. Because it's a Ring-side setting, it applies to keypad and Ring-app arming too — not just Home Assistant.

The Limitation That Shaped Everything Else¶

ring-mqtt publishes no record of which sensors were bypassed, and no arming-failure event. The fault detail arrives from Ring's websocket but was never exposed as MQTT attributes. The maintainer's recommended pattern is to issue the arm and then check whether the panel actually reached the requested state.

So the reporting had to be reconstructed in Home Assistant from the bypass-mode selects plus contact state at arm time:

bypassed = mode Always, or mode Faulted and sensor open
blocking = mode Never and sensor open
failed = panel didn't reach armed_* or arming within 10s

This mirrors Ring's own rules, so it's accurate — but it is a reconstruction, not a report, and it will drift if a sensor is added or removed without updating the zone list.

Knock-on: a failed arm is only detectable when HA issued the command. From the keypad or Ring app, a rejected arm produces no state change at all, so there is nothing to trigger on. Acceptable, because both of those give their own feedback.

Lessons¶

Check what the integration already exposes before building anything. The whole original problem was a dropdown that already existed.
Read the integration's own issue tracker for what it doesn't expose. Ten minutes there prevented building an automation on an attribute that is never published.
Prefer Faulted to Always for bypass. Faulted self-manages: protection returns automatically on nights the window is shut. Always silently disables a zone forever.
Render templates against live state before trusting them. A batteryStatus != 'ok' health check looked obviously correct and was wrong — Ring uses ok, full and charged as healthy values, so it false-flagged a detector at 85%. Only running it against real data caught it.
Timeouts have a UX cost in voice routines. A failed arm never changes state, so the full wait is spent every time. 30s was "safe" and would have stalled the spoken reply; 10s is the right trade.
Combining several spoken results into one reply forces you to wait for the slowest one. Merging alarm status into the Goodnight power check delayed the whole reply from instant to ~5–6s, because the power figure doesn't exist until after the settle delay. Worth it for one clean sentence, but it is a real trade, not a free win.

2026-07-29: A dead smart plug jammed the living room's WiFi channel for three days — and only the AP's own radio stats could see it¶

What Happened¶

The Shelly Wall Display in the living room dropped off WiFi and would not reconnect, despite being in the same room as a working access point. It reported a neighbour's WiFi as stronger than our own. Rebooting it and updating its firmware changed nothing. Every other 2.4 GHz device in the living room had quietly migrated across the house to the Bedroom AP.

The Problem¶

The Living Room AP's radio was healthy. The channel was not. 2.4 GHz channel 6 was 100% utilised by something that was not WiFi, so the AP's carrier-sense never saw a clear moment to transmit and it fell silent.
This is invisible from the client side and invisible in Home Assistant. Both look identical to "poor signal". The only place it shows up is the AP's own radio statistics:

cu_total    = 100     <-- channel is completely busy
cu_self_rx  = 0       <-- but almost none of it is us receiving
cu_self_tx  = 0       <-- and none of it is us transmitting
tx_packets  = 0       <-- the AP has stopped transmitting entirely

Read via unifi-login-api api/s/default/stat/device → radio_table_stats[].
Both of the "strange" user-reported observations were correct measurements of this: the neighbour's AP genuinely was stronger, because ours had stopped transmitting; and the Shelly genuinely couldn't hold a link, because with the least sensitive radio in the room it ended up on the Bedroom AP at -79 dBm.
The timeline was pinned from stat/report/hourly.ap (num_sta per AP): the Living Room AP held ~25–32 clients for the full 90 days of retained history, then between 21:00 and 22:00 on 07-24 the Bedroom AP jumped 6 → 25 clients while the Living Room collapsed 25 → 5, within a single hour.
Ruled out: no config change (all APs provisioned 2026-05-31), no reboot (128 days uptime), no new client, no alarms, no WiFi neighbour on ch6, and not Zigbee — the Sonoff mesh is on channel 26 and Hue on 25, both at the far top of the band.
The source was a failed Tasmota smart plug. The boiler plug had died on 07-24; that evening it was power-cycled and brought into the living room for testing, and left plugged into a living-room socket. The jam begins in that exact hour. A crash-looping ESP8266 can hold a continuous carrier. It was binned around 07-27.

The Solution¶

Immediate workaround (07-27): moved the Living Room AP's 2.4 GHz radio from ch6 → ch11. Recovery was instant — tx_packets 0 → 3527, clients returned, and the Shelly reassociated to the Living Room AP at -48 dBm, a 31 dB improvement.
Confirmation (07-29): with the plug gone, moved the AP back to ch6 and sampled the radio stats ten times over ~5 minutes: cu_total 27–33, of which ~28 was self-generated, tx_packets 857–1654, 8–9 clients stable. Nothing resembling the jammed state. Channel 6 was clean, so the AP was left there.

Note that rebooting the AP before the channel change did not help — which is precisely what proved the radio was healthy and the interference external.

Lesson Learned¶

cu_total ≥ 95 with cu_self_rx/cu_self_tx ≈ 0 means the channel is being jammed by something external. High cu_total with high cu_self_* is just a busy AP and is fine. The two are indistinguishable from any client, and nothing in UniFi or Home Assistant alerts on the difference — which is why this ran for three days unnoticed.

A dying mains device can take out a radio band. A failed ESP8266 is not just a dead endpoint; treat any crash-looping WiFi device as a potential emitter, and don't leave a suspect plugged in "for testing" near infrastructure.
stat/spectrumscan is not exposed on this UniFi version (Network 10.4.x), so there is no way to survey a channel remotely. Putting an AP on a channel is the only way to observe that channel — which makes the confirmation test inherently disruptive and worth planning.
Believe the user's odd observations. "A neighbour's WiFi is stronger than ours" and "it won't reconnect in the same room as the AP" were both accurate readings of a jammed channel, not confused reporting.
Attribution here is on the balance of evidence, not proven: the bin date and the ch6 → ch11 workaround both fell on 07-27, so the plug's removal cannot be fully separated in time from the channel move. If ch6 ever jams again, re-examine this conclusion first.
The UniFi claude code admin cannot write device config (PUT rest/device/<id> → HTTP 403 api.err.NoPermission); reads are fine, so channel changes must be made in the UI.
Worth building: an alert on the cu_total ≥ 95 + cu_self_tx ≈ 0 signature. Nothing warns on it today.

Documentation Updated¶

network.md — 2.4 GHz channel plan
inventory.md — boiler plug entry
CLAUDE.md (Homelab Support project) — full incident + channel plan

2026-07-29: Zigbee link quality was impossible to measure — 76 LQI sensors had been disabled all along¶

What Happened¶

Deciding whether a WiFi access point could safely use channel 11 required knowing whether it degraded the Zigbee mesh. There was no way to answer that: Home Assistant exposed no link-quality data at all for 75 Zigbee devices. A scan of /api/states returned zero linkquality entities.

The Problem¶

The obvious assumption was that Zigbee2MQTT needed a configuration change to publish link quality. It did not. Z2M was publishing linkquality in every device payload the whole time.
The entities existed in Home Assistant too — all 76 of them — sitting in the entity registry as:

disabled_by:      integration
entity_category:  diagnostic
platform:         mqtt

This is Z2M's deliberate default for diagnostic entities. Disabled entities have no state, which is why they never appeared in /api/states and looked like they didn't exist.
The fix is therefore entirely Home Assistant-side, and the REST API cannot do it — the entity registry is only writable over the websocket API (config/entity_registry/update).

The Solution¶

Enabled all 76 via websocket with disabled_by: null. Script vendored at scripts/enable-lqi.ps1 in the Homelab Support project (PowerShell ClientWebSocket, runs from Brabham, takes an entity list and supports -WhatIf). 76 enabled, 0 failed.
They do not appear immediately — MQTT discovery entities are only (re)created when the MQTT integration reloads or HA restarts. The registry change is persistent, so they arrive at the next restart.
A device paired after the bulk enable (the new boiler plug) got its LQI entity created disabled as well — newly discovered devices need the same treatment.

First baseline, with the Living Room AP on ch6: min 18, max 142, average 69 across 67 reporting devices.

Lesson Learned¶

Before concluding a metric is unavailable, check whether its entities exist but are disabled. "Zero results in /api/states" and "the integration doesn't provide this" look identical, and the difference here was months of unmeasurable Zigbee health versus a five-minute fix.

The entity registry needs the websocket API. The REST API can read states and call services but cannot enable, disable, rename or delete registry entries. This is a recurring gap — the same applies to deleting orphaned entities.
Enabling 76 chatty sensors has a cost: each updates on every Zigbee message, which is significant recorder volume. Watch database growth and exclude the noisiest if needed — sensor.*_last_seen is already excluded for exactly this reason.
Enabling these immediately surfaced information that had never been visible: a handful of devices sitting at LQI 18–25, and several unavailable devices nobody had noticed.
Do not read too much into a device's reported linkquality during a firmware update. The new boiler plug read 25–29 mid-OTA and 65 once the transfer finished — the OTA itself was saturating the link.

Documentation Updated¶

network.md — Zigbee link-quality note
CLAUDE.md (Homelab Support project) — LQI section + ch6/ch11 comparison

2026-07-28: NTP was broken for every device on the NoT VLAN — a firewall rule was matching on SOURCE port¶

What Happened¶

While investigating a smart plug that reported 0 W through a 3.2 kW kettle boil, its clock was found reading 1970-01-01. Checking the rest of the VLAN showed every Tasmota device on NoT had a 1970 date — across three different firmware versions (10.0.0, 12.1.0, 15.5.0). None of them had ever successfully synced time.

The Problem¶

Spanning three firmware versions immediately ruled out a device or firmware fault — this was network-level and VLAN-wide.
The NoT VLAN is deliberately internet-isolated (internet_access_enabled: false on the network, plus a Drop all other NoT catch-all policy), so pool.ntp.org was never reachable. Three of the four plugs were still pointed at it.
But the fourth had already been repointed at a local NTP server (Lotus, 192.168.1.80) and still failed — proving the local path was broken too, not just internet access.
The UniFi policy Allow all Local to NTP (Internal → Internal) looked correct at a glance: source IP group covering all three local subnets, destination port 123, protocol UDP. But its raw definition showed a source port condition as well:

source:       ip_group   = [192.168.1.0/24, 192.168.20.0/24, 192.168.30.0/24]
              port_group = NTP Port [123]      <-- requires SOURCE port 123
destination:  port_group = NTP Port [123]      <-- correct
protocol:     udp

Only a classic ntpd daemon binds local port 123 when acting as a client. Tasmota's NTP client uses a random ephemeral source port, so the rule never matched, and the traffic fell through to Drop all other NoT and was silently dropped.
The rule had therefore been ineffective for every SNTP-style client since it was written. It went unnoticed for years because MainLAN and IoT both have internet access and never needed a local time source — NoT was the only network that actually depended on this rule working.
Two theories were investigated and disproven: Unraid's ntpd restrict line was assumed to be refusing off-subnet queries (it isn't — Lotus serves 192.168.30.x happily), and the UCG was assumed to be a usable fallback time source (it isn't — it does not serve NTP at all).

The Solution¶

Edited the Internal → Internal instance of Allow all Local to NTP (the policy is replicated across four destination zones; only that one is relevant): Source → Port changed from the NTP Port object to Any. Destination port, source IP group and protocol left untouched.
Restarted the one plug that was already pointed at a local server — it synced within seconds.
Repointed the remaining three plugs at Lotus (192.168.1.80) and Cooper (192.168.1.60). All three synced immediately with no restart — an unsynced Tasmota device retries on its own, so no power interruption was needed to the dishwasher, TV or turbo trainer.

Lesson Learned¶

A firewall rule that specifies a source port will silently fail for any client that uses an ephemeral one — and "the rule exists and looks right" is not evidence that it works.

Source port should almost always be Any. Pinning it to the service port only works when the client happens to bind that same port, which for NTP means classic ntpd and essentially nothing else. Removing the constraint is not a weakening — the rule still only permits UDP from local subnets to destination port 123.
A fault that spans multiple firmware versions of the same device class is network-level, not device-level. That single observation collapsed the search space immediately; checking the other three plugs took under a minute and was the highest-value diagnostic step taken.
On an internet-isolated VLAN, check that devices are pointed at a local time source at all — a default of pool.ntp.org is unreachable by definition and will fail silently forever.
The UCG does not serve NTP. Lotus and Cooper (Unraid) both do, and accept queries from other subnets without needing a restrict change.
Symptoms of a dead clock on a metering device are easy to misread as a metering fault: Tasmota's daily energy rollover is clock-driven, so Energy Today never rolls over and accumulates indefinitely. One plug was reporting 146 kWh "today", another 6.5 kWh against a 156 kWh lifetime total. Energy Total is unaffected and stays correct throughout.
Expect one final bad day of data after fixing this: the first real rollover flushes the accumulated garbage into Yesterday, so that figure is junk for one day before everything is clean.

Documentation Updated¶

network.md — NTP policy note
inventory.md — kettle plug entry
CLAUDE.md (Homelab Support project) — full writeup under Known Issues

2026-07-28: A smart plug reported 0 W through a 3.2 kW kettle boil — `TelePeriod` was hiding every short event¶

What Happened¶

Investigating overnight power use, two ~3.2 kW spikes were found at 05:54 and 05:55 (45 s and 30 s long, immediately after kitchen motion at 05:53:35 — clearly a kettle boil). The kitchen kettle's own Tasmota smart plug reported 0 W throughout both, and switch.kettle went unavailable twice within ten minutes.

The Problem¶

The obvious conclusion — a third failing Tasmota plug, after two died the same week — was wrong.
The plug's TelePeriod was 300, meaning it only published telemetry once every 5 minutes. A 45-second event fits entirely between two reports. Every short high-power event on that plug had been invisible to Home Assistant for years.
PowerDelta was unset, so nothing forced an immediate publish when power changed. That setting exists precisely to catch transients between scheduled reports.
Contributing factors found alongside: firmware 10.0.0 built 2021-11-16 (four and a half years stale), Sleep 50 dynamic sleep (a known cause of MQTT instability — the plug showed 114 MQTT reconnects against only 1 WiFi reconnect on a 13-hour link), and a dead clock (see the NTP entry above).

The Solution¶

TelePeriod 60, PowerDelta 20, Sleep 0 — the first two are the actual fix for the invisibility.
Firmware upgraded 10.0.0 → 15.5.0, two-stage as required on a 1 MB device.
Config backed up first via http://<ip>/dl (a 4 KB .dmp) as a recovery path.

Verified the same night with a real boil, sampling sensor.kettle_energy_power every 2 seconds:

23:33:20   kettle=0W       idle
23:33:22   kettle=245W     detected ~2s after switch-on
23:33:27   kettle=3001W    full power
  ...      sustained 2976W for 76 seconds
23:34:44   kettle=48W      off, detected within 2s

A 76-second boil captured start to finish. Under TelePeriod 300 this event — shorter than a single telemetry interval — would have logged 0 W, exactly as the 2026-07-27 boil did.

Lesson Learned¶

Before concluding a smart plug is faulty, check TelePeriod — a plug reporting 0 W during a real load is usually reporting on a schedule that misses it, not failing to measure it.

TelePeriod 60 + PowerDelta 20 is a much better default than the stock TelePeriod 300 for anything that draws power in short bursts (kettle, toaster, microwave, shower). Without PowerDelta, no telemetry interval is short enough to reliably catch a 45-second event.
Two-stage firmware upgrade is mandatory on 1 MB ESP8266 devices. The full tasmota.bin is ~666 KB against ~388 KB reported free, so it must go via tasmota-minimal.bin (~373 KB) first. Attempting the full image directly is a safe failure — the ESP updater refuses before writing — but it wastes a cycle. Bricking requires a power cut mid-write, so the real rule is simply: do not interrupt power during a flash.
The minimal build has a reduced command set — Status returns {"Command":"Unknown"}. That is expected mid-upgrade and is not a failure; read the version from the web UI root instead. Ping and WebQuery are not compiled into the standard release build either, so don't plan diagnostics around them.
All settings survived both stages of the jump from 10.0.0 to 15.5.0, including MQTT config, PowerOnState, and the telemetry settings applied beforehand.
Check PowerOnState before restarting any plug that controls something people rely on. PowerOnState 1 (always on) or 3 (restore last) come back safely; 0 would leave the appliance dead.

Getting Firmware to an Internet-Isolated Device — Without Touching the Firewall¶

The NoT VLAN cannot reach ota.tasmota.com, and a temporary web server on the main LAN is blocked by the Drop all other NoT policy. Adding a firewall exception was considered and proved unnecessary:

The existing Allow NoT to Home Assistant policy permits NoT → 192.168.1.12 on all ports and protocols, and Home Assistant serves /config/www/ at http://192.168.1.12:8123/local/<file> with no authentication.

So the working method is:

Copy the .bin.gz into /config/www/ on Home Assistant
OtaUrl http://192.168.1.12:8123/local/tasmota-minimal.bin.gz then Upgrade 1
Repeat for the full image
Delete the files from /config/www/ and reset OtaUrl — anything left there is served unauthenticated

This works for any IoT or NoT device that needs to fetch a file, not just firmware, and leaves no firewall exception to remember to unpick afterwards.

Documentation Updated¶

inventory.md — kettle plug entry added to Smart Home Hardware
CLAUDE.md (Homelab Support project) — Tasmota fleet section under Known Issues

2026-07-28: "Alexa, Goodnight" moved into Home Assistant — and the house meter can only explain a quarter of the load¶

What Happened¶

The "Alexa, Goodnight" routine (lights off downstairs, alarm to Home, devices off, then a Home Assistant power check) was migrated so that Alexa only hears the phrase and triggers a scene — all logic now lives in Home Assistant as script.goodnight.

The Problem¶

Several things surfaced while rebuilding it:

Two duplicate power-check automations were both enabled, firing on every arm event and double-alerting. They had been silently duplicated at some earlier point.
The "top consumers" alert text was reporting the whole-house meter as the top consumer. It excluded sensor.electricity_meter_power but not sensor.octopus_energy_..._current_demand — a second whole-house meter, which won the sort every time. Several aggregate sensors (all_light_power, all_standby_power, infrastructure_total_power) and one double-counted subset (lotus_cpu_power, part of lotus_plug_power) were also polluting it.
The alert fired before the meter could possibly reflect the shutdown. The Develco ZHEMI101 is an optical pulse counter and updates roughly every 5–6 seconds — fine, but a reading taken immediately after switching things off still includes the load just removed.
The reply had to land within ~5 seconds, because the routine is said on the way upstairs. No meter can reflect a shutdown that fast.
Only ~25% of household load is visible to Home Assistant at all. Correlating six overnight excursions above 500 W against every W sensor: at 23:49 the meter read 570 W while all monitored devices totalled 138 W. The two 3.2 kW kettle spikes were essentially 100% invisible (162 W and 151 W accounted for).

The Solution¶

Capture the load about to be removed before switching, then subtract it from the meter reading — rather than waiting for the meter to catch up. This makes a 5-second verdict both fast and accurate.
Report total / accounted-for / unaccounted rather than a confident top-3 list, so the message says "432 W unaccounted — something without a smart plug" instead of naming the fridge at 40 W.
Voice reply reduced to "using more power than usual, check your phone"; the breakdown goes to the phone notification only.
Both duplicate automations deleted; a single automation now covers keypad arming (with a 90 s delay, since there's no rush), guarded so it can't double-fire when the script arms the alarm itself.
Home Assistant arms the alarm directly — alarm_control_panel.lower_link_alarm is an MQTT panel with code_arm_required: false, so Alexa was never needed for that step.

Lesson Learned¶

Know what fraction of your load is actually measured before building an alert that names culprits — a "top consumers" list is only as honest as your coverage.

Whole-house baseline for reference: overnight min 240 W, median 320 W, p90 570 W, with a permanent ~150–400 W that no smart plug can see (hardwired loads — boiler pump, extractor, immersion, oven clock, network gear). 17.7% of overnight samples exceed 500 W with nothing wrong, so a 500 W alert threshold sits inside normal noise; 700 W is exceeded only 3.6% of the time and still catches a cooker ring at 1000–2000 W.
When filtering sensors by unit for an aggregate, exclude by explicit list, not by hoping the names don't collide — a second utility meter, group/aggregate sensors, and derived subsets of another sensor will all silently corrupt a "biggest consumer" calculation.
To report on the effect of an action faster than the sensor can measure it, measure the inputs beforehand rather than the result afterwards. HA already knew the exact wattage of everything it was about to switch off. Measured evidence for this, from a kettle boil the same night with a per-plug meter and the house meter sampled side by side every 2 seconds:

Time Kettle plug House meter

23:33:27 3001 W 510 W

23:33:31 2976 W 2250 W

23:33:35 2976 W 3260 W

The plug saw the full 3 kW load about 8 seconds before the house meter did. The ZHEMI101 infers power from the interval between optical pulses, so a step change takes several pulses to converge — it ramps rather than steps. A falling step is at least as slow, which is exactly why reading the meter a few seconds after switching things off reports the pre-shutdown load and produces false alarms.
The Octopus Home Mini (..._current_demand) polls on a strict 60-second cycle — useless for anything time-sensitive. The pulse-counter ZHEMI101 is the more responsive of the two despite being the lower-tech device, but it still lags a per-device plug by ~8 s on a large step.
⚠️ Resilience trade-off: the previous Alexa routine ran entirely in Amazon's cloud and worked when the homelab was down. Now that the logic lives in Home Assistant — which currently runs as a VM on Lotus — saying "Goodnight" does nothing at all if Lotus or HA is down: no lights off, no alarm armed. The alarm can still always be armed from the Ring keypad, which is independent of the homelab. This was an accepted consequence of the migration, not an oversight, but it needs to stay visible.

Documentation Updated¶

partner/common-issues.md — corrected: the routine is no longer independent of Home Assistant
CLAUDE.md (Homelab Support project) — full writeup under Known Issues

2026-07-24: HASS.Agent on Brabham lost hardware sensors — Windows Defender removed the WinRing0 driver¶

What Happened¶

HASS.Agent (v2.2.1, running from C:\Users\dan_c.BRABHAM\AppData\Local\HASS.Agent\Client) popped an error dialog on startup: Could not load file or assembly 'LibreHardwareMonitorLib, Version=0.9.1.0...'. The system cannot find the file specified.

The Problem¶

Checked the Client folder directly — every other dependency DLL was present, only LibreHardwareMonitorLib.dll was missing. This ruled out a broad corrupt/incomplete install and pointed at something specifically removing that one file.
LibreHardwareMonitorLib depends on the WinRing0 kernel driver for ring-0 hardware sensor access (CPU/GPU/motherboard telemetry). WinRing0 is on Microsoft Defender's built-in vulnerable-driver blocklist, since it's legitimately risky (any process can request arbitrary ring-0 I/O through it) — exactly the property that makes it useful for hardware monitoring tools.
Confirmed via Get-WinEvent -FilterHashtable @{LogName='Microsoft-Windows-Windows Defender/Operational'; Id=1116,1117}: Defender detected and remediated VulnerableDriver:WinNT/Winring0 on 2026-04-27 and again 2026-04-28 — nearly 3 months before the error was actually noticed, since HASS.Agent doesn't surface the failure until you look at its window.

The Solution¶

Added a Defender exclusion for the HASS.Agent install folder (Add-MpPreference -ExclusionPath "C:\Users\dan_c.BRABHAM\AppData\Local\HASS.Agent") so the driver/DLL won't be stripped again.
Reinstalled HASS.Agent 2.2.1 over the existing install (no uninstall needed) to re-extract the missing DLL and driver.
Confirmed the sensors error was gone on next launch.

Lesson Learned¶

A "file not found" error from a hardware-monitoring tool on Windows is often Defender quietly removing a flagged driver, not a broken install — check the Defender event log before reinstalling.

Get-WinEvent against Microsoft-Windows-Windows Defender/Operational (event IDs 1116/1117) shows exactly what Defender detected and when — much faster than guessing from the app's own vague error message.
Any tool needing WinRing0 (LibreHardwareMonitor, OpenHardwareMonitor, HWiNFO, and by extension HASS.Agent) is a candidate for this same failure mode, since it's a blanket Defender blocklist entry, not something specific to one app.
Add the Defender exclusion before reinstalling/repairing — otherwise the fresh driver just gets removed again on the next Defender scan.
The gap between the actual detection (2026-04-27/28) and Dan noticing the error (2026-07-24) shows this kind of failure can sit silent for months — worth a periodic glance at tools like this that only report failures via an easily-dismissed popup.

Documentation Updated¶

CLAUDE.md in the Homelab Support project — Brabham PC Monitoring backlog item (HASS.Agent/MQTT presence detection work)

2026-07-24: Boiler smart plug failed — 5-hour outage needed a manual power cycle, no hot water¶

What Happened¶

Dan had no hot water going for a shower around 18:50 BST. Investigation via Home Assistant history traced it to the boiler's Tasmota smart plug (switch.boiler, WiFi, NoT VLAN SSID 2SVT-NoT, firmware v10 — an early purchase batch) going completely dead for about 5 hours that afternoon, taking the boiler's power — and the Shelly Plus2PM boiler controller, which is powered from the boiler unit itself — down with it.

The Problem¶

The plug had been crash-looping with firmware Exception restarts every few minutes since at least the previous day (2026-07-23) — a chronic, pre-existing instability that was easy to mistake for "just background noise."
At 13:47 BST it went fully dark: no reboot attempts logged at all during the outage, unlike the routine crash-loop pattern that both preceded and followed it. This is the tell that distinguishes "device genuinely lost power / hung dead" from "device is still crash-looping but very slowly."
Because switch.shellyplus2pm_centralheating (the actual boiler call-for-heat controller) draws its power from the boiler itself rather than a separate socket, it went unavailable at almost the same moment as the smart plug — which initially looked like two independent device failures correlating, but was actually one fault with a downstream effect. Wiring topology matters when reading "two devices died together" — check what powers what before concluding it's a shared external cause (circuit trip, network outage) rather than a single point of failure with a dependent.
The outage did not self-heal. It stayed dead until Dan physically found the plug and cycled power at the wall switch. The plug's own restart-reason sensor confirms this: a genuine "Power On" boot event logged at 18:44 BST, not a WiFi/MQTT reconnect and not an automatic watchdog reboot.
Kitchen motion sensors (binary_sensor.kitchen_motion_detector, Everything Presence One binary_sensor.everything_presence_one_..._occupancy/_pir) independently corroborated the timeline — a walk-through cluster at 18:43–18:45 BST lined up within seconds of the plug's recorded "Power On" event, confirming that was the moment of manual intervention.

The Solution¶

Reconstructed the timeline via HA's REST API (/api/history/period) pulling switch.boiler, sensor.boiler_restart_reason, sensor.boiler_wifi_connect_count, sensor.boiler_last_restart_time, and the Shelly's switch.shellyplus2pm_centralheating — the incrementing last_restart_time timestamps during the crash-loop period were the key signal that distinguished "still trying to boot" from "genuinely dead."
Confirmed the manual-fix theory using kitchen motion sensor history as an independent corroborating source, cross-referenced with Dan's own account of physically cycling power at the wall.
Plug removed from service entirely — on re-test plugged into an isolated wall socket, it appears permanently dead (no LED, no WiFi association).

Lesson Learned¶

A device outage that needs a manual power cycle to end is a hardware/firmware fault, not a network blip — and don't assume two devices failing together means two separate faults without checking what powers what.

A chronic, low-grade crash-loop (frequent brief unavailable blips that self-recover in seconds) can escalate into a full hang with no warning — don't dismiss recurring MQTT/WiFi flakiness as harmless background noise; it can be an early symptom of a device on its way out.
To tell "still trying to reconnect" apart from "genuinely dead" in Tasmota history: check whether a "boot/restart time" sensor keeps advancing during the outage. If it stops advancing entirely for a long stretch and then jumps by hours when it returns, the device was actually powered off/hung, not just flaky.
When two devices lose availability at the same time, check the physical power wiring before concluding it's a shared external cause (circuit trip, VLAN/AP outage) — one device silently powering another (as the Shelly here draws from the boiler) can produce an identical "two independent failures" signature.
Presence/motion sensor history is a useful independent corroborating source for reconstructing "when did I actually do X" when the exact time isn't remembered — it lined up with the technical recovery timestamp to within seconds here.
Second Tasmota WiFi smart plug failure in one week (after the Lotus Server Plug, 2026-07-22 — see Homelab Support CLAUDE.md's Unraid/Lotus section for that incident). If a third fails the same way, treat it as a batch/firmware-version pattern (this one was firmware v10, an early purchase batch) rather than two unrelated coincidences, and audit remaining Tasmota plugs' firmware versions.

Documentation Updated¶

inventory.md — Boiler smart plug marked failed/removed in Smart Home Hardware, Shelly Plus2PM note updated to record its power source
CLAUDE.md (Homelab Support project) — Key Devices table + full incident writeup and Zigbee replacement backlog item under Known Issues

2026-07-23: Electricity meter replaced — Develco ZHEMI101 needed pulse-rate reconfig, a reseated jack, and a total reset¶

What Happened¶

Dan's electricity meter was physically replaced. The Develco ZHEMI101 (a Zigbee optical pulse-counter clipped to the meter's flashing LED, exposed in Zigbee2MQTT/HA as sensor.electricity_meter_power/energy) kept reporting wrong values afterwards — first roughly half the real load, then briefly erratic/zero, until it stopped reporting altogether.

The Problem¶

Three independent issues stacked on top of each other: 1. Wrong pulse rate. The new meter's label states 2000 imp/kWh; the ZHEMI101 was still configured for the old rate (Z2M's pulse_configuration default is 1000), so every pulse was double-counted as energy — power/energy read at roughly half the real value. 2. Loose optical sensor jack. After correcting the pulse rate, readings still went briefly erratic then flatlined at 0. Root cause: the ZHEMI101's 3.5mm jack (which the optical sensor plugs into) hadn't been fully pushed home after the meter swap — intermittent contact meant it caught some flashes and missed others, then stopped reporting new values for minutes at a time. 3. No live HA access to verify any of it. The previously-issued HA long-lived access token had expired, so there was no way to pull live entity states from Brabham to confirm the fix was actually working, rather than just trusting the config change.

The Solution¶

Zigbee2MQTT frontend → device "Electricity Meter" (model ZHEMI101) → Exposes tab → set Pulse configuration to 2000.
Generated a fresh HA long-lived access token (Settings → Profile → Security → Long-lived access tokens) and saved it to C:\Users\dan_c.BRABHAM\Temp\ha-api-token.txt on Brabham, matching the existing pattern used for UniFi credentials (unifi-login.json in the same folder). This let curl.exe hit HA's REST API directly (/api/states/<entity_id>, /api/services/<domain>/<service>) to read and set entity values going forward.
Fully reseated the 3.5mm optical sensor jack. Confirmed fixed once sensor.electricity_meter_power started updating every ~30–60s and tracking within a couple of percent of the Octopus Energy smart-meter integration's own live demand reading (sensor.octopus_energy_electricity_..._current_demand) — a second, independent live reading of the same real-world load, invaluable for confirming the fix rather than just hoping.
Reset the corrupted cumulative total. sensor.electricity_meter_energy had drifted to ~10,100 kWh, partly inflated by the wrong pulse rate — not a number worth trusting. Forced it to the actual registered meter total via number.electricity_meter_unit_summation, using sensor.octopus_energy_electricity_..._current_total_consumption (180.585 kWh) as the ground-truth source, since that's the supplier's own reading of the new physical meter. sensor.electricity_meter_energy now correctly reads 180.59 kWh.

Lesson Learned¶

A meter swap breaks three separate things on a Develco ZHEMI101, not just the pulse rate — check all three.

Read the imp/kWh figure printed on the new meter and set Z2M's pulse_configuration to match — never assume it's the same as the old meter (mains electricity meters commonly differ, e.g. 1000 vs 2000 imp/kWh).
Physically confirm the 3.5mm optical sensor jack is fully seated after any meter swap — a half-inserted jack causes intermittent/zero readings that look exactly like a leftover config problem but aren't. If readings are erratic or intermittently blank after fixing pulse_configuration, check the physical connection before further config changes.
The cumulative total is unrecoverable across a meter swap by fixing config alone — force it to a trustworthy independent source (e.g. a supplier smart-meter integration's own total-consumption sensor) via unit_summation, rather than leaving it wrong or guessing a number.
state_class: total_increasing energy sensors tolerate a forced downward reset fine going forward, but expect a visible discontinuity in the historical Energy dashboard graph at the reset point — this is cosmetic, not a fault.
Having a second, independent live reading of the same physical quantity (here, Octopus's own smart-meter demand sensor) made this whole diagnosis fast — without it, "is the fix actually right?" would have been a guess.

Documentation Updated¶

inventory.md — added the Develco ZHEMI101 to Smart Home Hardware
CLAUDE.md (Homelab Support project) — Key Devices table + HA API token note

2026-07-22: Lotus's Unraid boot USB failed mid-OS-upgrade — recovered to a new stick + TPM licensing¶

What Happened¶

Dan kicked off an in-place Unraid 7.3.1 → 7.3.2 OS update from the web UI. It appeared to hang for a long time at writing flash device - please wait... while extracting the new bz* files. Investigation showed the boot USB flash was dying, and the update aborted itself. Recovered by restoring a daily flash backup onto a new USB stick and re-licensing against the TPM.

The Problem¶

The update's unzip writing to /boot stalled. dmesg showed the boot flash (/dev/sda1, Verbatim "STORE N GO", internal USB port) resetting continuously with usb 1-3: device descriptor read/64, error -110 (timeouts), and unzip stuck in uninterruptible D state on rq_qos. Unraid's own updater then failed with *** bad sha256 on bzmodules → *** The upgrade failed, but no changes were made to your configuration. *** Your USB Flash is likely failing.
It did not damage the running system. Unraid runs from RAM once booted, so the array, Docker (32 containers), the HA VM, and the web UI all kept running the whole time the flash was flapping. The installed 7.3.1 /boot/bz* files were untouched (the updater only writes a staging folder, then copies on success — which never happened).
Three diagnostic traps worth remembering:
SSH login hung while the web UI stayed fine. Unraid's SSH login sources init from /boot, so a saturated/failing flash blocks new SSH sessions; the already-running (RAM-resident) web server is unaffected. Ironically, "monitor the update via SSH" is the one thing a flash write blocks.
A frozen browser tab looked like "UI down" but wasn't. The update-progress popup keeps a live stream to emhttpd that stalls under heavy flash I/O; a fresh load of the local IP (http://192.168.1.80, not the SWAG hostname / not a reload of the dead tab) worked instantly.
The box stayed pingable and served the web UI throughout — a failing boot flash is not an immediate outage.

The Solution¶

Preserved data first (while the flash was still barely readable): emergency-copied /boot/config (license keys, super.dat, network, shares) to the array + Brabham.
The real saviour — the Appdata Backup plugin's daily flash backup. It was already configured with flashBackup: yes (daily 05:00, verifyBackup: yes) writing lotus-v7.3.1-boot-backup-YYYYMMDD-HHMM.zip into /mnt/user/backups/lotus/ab_*/. Today's copy (05:22, hours before the failure, 1.33 GB, unzip -t clean) was the ideal restore source — a full, verified, pre-failure flash image.
Restored onto a new USB (Lexar 28.9 GB, external port) using the Unraid USB Flash Creator → Operating System → "Use custom" → the flash-backup zip (formats + makes bootable automatically). Stayed on 7.3.1 deliberately — change only the hardware, not the OS version, so any problem has one cause.
First boot hit "Multiple License Keys Present" (the restored config carried three .key files) and Unraid flagged the license device as the TPM. Removed Starter.key + Trial.key (kept Unleashed.key), then via Tools → Registration bound the Unleashed license to the TPM (Unraid 7.3 TPM licensing). Array reassembled from super.dat with zero disk errors, no rebuild; a correcting parity check ran automatically (expected after the unclean stop).
USB port layout: the boot stick took the native USB-A port; the UPS moved to the USB-C port via an A-to-C adapter (the boot device gets the most reliable, adapter-free connection; the UPS is low-stakes and non-boot-critical). NUT had to be restarted to re-bind to the moved device.

Lesson Learned¶

A failing Unraid boot flash is a recoverable, non-emergency event if you have a current flash backup — the box keeps running from RAM, so don't reboot it in a panic; secure a backup, prep a new stick, then swap once.

The Appdata Backup plugin's "flash backup" option is not optional — it's what turned a potential reinstall-from-scratch into a ~20-minute restore. Confirm it's enabled and that a copy lands off-box.
USB flash death is about write-cycles + heat, not calendar age — a 1+ GB sustained OS-upgrade write is exactly the stress that kills a marginal stick. Consider updating the OS with the box already healthy, and/or move boot off USB entirely (see below).
Diagnose a failing flash via dmesg (USB resets + error -110), the updater's own bad sha256 ... USB Flash is likely failing, and the tell-tale combo of SSH-login hangs + a live web UI.
On restore, expect "Multiple License Keys Present" if the backup carried more than one .key — remove all but the one to keep, then register.
Unraid 7.3 TPM licensing decouples the license from the USB GUID — the right move on TPM-2.0 hardware, and a prerequisite for going fully USB-free via internal boot. Licensing is independent of boot method; the first transfer is free, then one per 12 months automated.

Deferred — Phase 2: move Lotus boot off USB entirely (internal boot)¶

Unraid 7.3 supports internal boot (NVMe/SSD) with a mirrored boot pool + TPM licensing = fully USB-free. Lotus is a good candidate (TPM 2.0, UEFI, 2× NVMe ZFS mirror) and the licensing half is already done (TPM). Blocker: the NVMe cache mirror consumes the whole of both drives (nvme0n1p1/nvme1n1p1 span the entire disk — no free space for a boot partition), so internal boot requires recreating the pool: back up the cache (only ~116 GB actually used, not the 966 GB capacity) → rebuild as a boot+data pool via Tools → Onboarding Wizard → restore → retire the external USB. Unhurried project; the external stick is a fine interim boot device.

Documentation Updated¶

Primary NAS — Lotus — new Boot Device & Licensing section, flash-backup note, UPS port note
Primary NAS Recovery — flash-failure restore procedure marked tested 2026-07-22

2026-07-19: Coral TPU + GPU passthrough broke after the Proxmox 9 rebuild¶

What Happened¶

Immediately after rebuilding Pacific on Proxmox VE 9.2 (see previous entry) and restoring the Frigate container from backup, it failed to start at all, then — once started — crash-looped on both AI detection and the doorbell camera's video stream.

The Problem¶

Two separate, stacked issues, both root-caused to the same thing: a host-side script (/var/lib/lxc/108/mount_hook.sh, referenced via lxc.hook.autodev in the container's .conf) that isn't part of any container backup, because it lives outside the container filesystem on the Proxmox host itself.

1. Container wouldn't start at all. lxc_init: Failed to run lxc.hook.pre-start / exit 116 — the referenced hook script simply didn't exist on the fresh host. Removing the lxc.hook.autodev line let the container start (device mounts marked optional in the config no-op gracefully; the hook script itself does not).

2. Once started, two hardware-dependent features were broken, because the missing script's actual job was dynamically bind-mounting devices into the container at start: - Coral TPU (/dev/apex_0) — Frigate logged No EdgeTPU was detected ... Failed to load delegate from libedgetpu.so.1.0 and the whole detector process crashed (no CPU fallback was configured, so this wasn't a graceful degradation — object detection was completely dead). - GPU/VAAPI (/dev/dri/card0, /dev/dri/renderD128, used for hardware-accelerated video decode) — ffmpeg crash-looped on the doorbell camera: Option hwaccel ... cannot be applied to output url vaapi. This took the camera stream down entirely, which is arguably worse than the detector issue since there's nothing to detect on without a stream.

Separately, /dev/apex_0 itself didn't exist on the host either — the community gasket-dkms package (installed via Coral's official apt repo) failed to build against Proxmox 9.2's kernel (7.0.2-6-pve, based on Linux 6.12) with four distinct kernel-API incompatibilities: - eventfd_signal(ctx, 1) — signature changed to a single argument - .llseek = no_llseek — the no_llseek symbol was removed from the kernel entirely, replaced by noop_llseek - class_create(driver_desc->module, driver_desc->name) — signature changed to a single argument (name only) - MODULE_IMPORT_NS(DMA_BUF) — now requires a quoted string, MODULE_IMPORT_NS("DMA_BUF")

Also had to switch Pacific from the enterprise apt repo (401s without a paid subscription) to pve-no-subscription before pve-headers-$(uname -r) — needed to build any kernel module at all — was even installable.

The Solution¶

Removed gasket-dkms, switched to pve-no-subscription repo, installed pve-headers-7.0.2-6-pve.
Cloned https://github.com/google/gasket-driver directly (not the packaged/apt version) to /usr/src/gasket-driver-git — the eventfd_signal and class_create fixes were already present upstream on GitHub, version-gated by kernel version macros, just not yet in the packaged release. Only had to manually patch the remaining two:
```
sed -i 's/\.llseek = no_llseek,/.llseek = noop_llseek,/' src/gasket_core.c
sed -i 's/MODULE_IMPORT_NS(DMA_BUF);/MODULE_IMPORT_NS("DMA_BUF");/' src/gasket_page_table.c
```
Built with plain make (the repo has no dkms.conf — it's meant as an out-of-tree module, not a DKMS package) against the pve headers, copied the resulting .ko files to /lib/modules/7.0.2-6-pve/extra/, depmod -a, modprobe gasket apex. Registered for boot auto-load via /etc/modules-load.d/gasket.conf. This is not DKMS-integrated — a future kernel upgrade will break it again and need the same two patches reapplied and a rebuild. The patches themselves are trivial (2 one-line sed fixes) if this happens again; check google/gasket-driver's GitHub issues first in case upstream has caught up further by then.
Removed the broken lxc.hook.autodev reference from 108.conf entirely, and replaced its device-passthrough job with static lxc.mount.entry lines for /dev/apex_0, /dev/dri/card0, and /dev/dri/renderD128 directly in the config — the cgroup2 device-allow rules for these were already present in the original config (major numbers 120 for Coral, 226 for DRM), only the actual bind mounts were missing. This is arguably more durable than the original dynamic hook script, since it doesn't depend on an external file surviving a rebuild.
Also hit (and fixed) a stale NFS file handle on the CCTV clips mount on first access after the restore — NFS: server 192.168.1.80 error: fileid changed in dmesg. Fixed with umount -l /mnt/pve/cctv_clips and letting Proxmox remount fresh; the underlying Lotus export's fileid had changed since the mount was first established (unrelated to the migration itself, just first noticed then).

Lesson Learned¶

A lxc.hook.autodev script referenced by an LXC config is a host-side dependency, invisible to that container's own backup — check it actually exists on the new host before restoring/starting a container that references one, and prefer static lxc.mount.entry config over a dynamic hook script where possible, since the config itself is backed up.

If a hardware-passthrough LXC fails to start with Failed to run lxc.hook.pre-start after a restore, check whether the referenced hook script file actually exists on the host — it's very likely a host-side script that didn't come back with the restore
A community DKMS kernel driver package that hasn't been updated in a while is a real risk on any Proxmox major-version upgrade (new kernel = new internal APIs) — check the upstream GitHub repo's main branch (not just the packaged release) for newer compatibility fixes before assuming a build failure means the hardware is unsupported
When a build fails with multiple kernel-API errors, fix and rebuild rather than giving up after the first — this hit 4 separate incompatibilities across 2 files, not just 1
Missing hardware acceleration can fail hard (crash the whole pipeline) rather than gracefully degrading, depending on how the consuming application is configured — check whether a CPU/software fallback is actually configured, don't assume "slower" is the worst case
A stale NFS file handle (fileid changed) after restoring a container that mounts network storage is a different, separate problem from the storage/container restore itself — umount -l + remount, don't assume the backup/restore itself is at fault

Documentation Updated¶

Proxmox NUC — Pacific — Hardware Notes section rewritten with the driver fix and passthrough details
Proxmox Recovery — added a step covering lxc.hook.autodev host-side script verification, and the stale-NFS-handle gotcha

2026-07-19: Pacific's boot disk failed — full rebuild on Proxmox 9.2¶

What Happened¶

Routine SSH health check on Pacific (prompted by nothing specific — general "is everything healthy" check) turned up a boot/VM disk with active, worsening hardware failure. What started as a health check turned into planning and executing a full disk replacement and OS reinstall.

The Problem¶

/dev/sda (SanDisk SDSSDHP256G, 238.5GB — Pacific's only disk, hosting root plus every VM/LXC via LVM-thin) showed SMART Reported_Uncorrect at 824+ and climbing, Total_Bad_Block at 251, and an ATA error count that grew from 785 to 788 within a single day. Kernel log confirmed real, active corruption: Medium Error / Unrecovered read error - auto reallocate failed — critically, the drive's own firmware had already tried and failed to reallocate the bad sector to a spare, meaning this was not going to self-heal.

This was already causing real damage: LXC 101 (AdGuard) and LXC 108 (Frigate) had both been silently failing their nightly PBS backups for weeks — LXC 101 since 2026-06-18 (a month), LXC 108 since 2026-04-25 (three months) — both failing with Input/output error (os error 5) reading specific files from the bad sectors. Nobody had noticed because the Proxmox scheduler doesn't surface backup failures loudly unless someone checks.

A second, unrelated problem was masking part of this: the root filesystem was at 96% full (2.6GB free), which turned out to be caused by 45GB of orphaned vzdump backups (dated mid-2025) sitting in a unraid-lotus NFS storage whose mount had been dead for 11+ days — Proxmox was reading/writing to the local directory underneath the failed mount instead. This alone was severe enough to cause a separate backup failure (No space left on device) on LXC 108, unrelated to the disk hardware issue.

The Solution¶

Immediate fixes (no reboot required): 1. Decommissioned the dead unraid-lotus storage (removed from storage.cfg, /etc/fstab, cleared the failed systemd mount unit) and deleted the 45GB of orphaned data — root disk went from 96% to 14% used. 2. LXC 101's backup was hitting the bad sector via one specific file (/usr/lib/x86_64-linux-gnu/gconv/IBM280.so, part of libc6) — apt-get install --reinstall libc6 rewrote the file to a different, healthy block, immediately fixing that backup without touching the disk itself. This is a stopgap for one file, not a fix — the disk has 251 recorded bad blocks and climbing; other files could hit the same wall at any time. 3. Took fresh one-off backups of the two containers that had zero scheduled backup jobs at all (GMUPS-27, TailscaleExitNode) before doing anything further.

Full migration (single-failover strategy — see HA Recovery for the general pattern): 1. Failed HA over to the Lotus standby VM once, left it running as primary for the full duration of the rebuild — no time pressure on any of the following steps. 2. Physically swapped the failing SanDisk for a Samsung 860 PRO 512GB (repurposed from a workstation, secure-wiped first via Clear-Disk -RemoveData -RemoveOEM in Windows — not a true ATA Secure Erase, since USB-SATA bridges generally don't pass that command through, but sufficient since the drive was going straight into a fresh install regardless). 3. Fresh Proxmox VE 9.2 install (upgrading from 8.4.17 in the process, rather than doing an in-place major-version upgrade on the failing disk). 4. Restored all guests from PBS, fixed the Coral TPU/GPU passthrough fallout (see next entry), fully soak-tested every service. 5. Single controlled failback to Pacific once satisfied (not yet done as of this writing — Pacific is soak-testing).

Lesson Learned¶

A routine health check is worth doing periodically even with no specific complaint — this disk failure had been silently breaking backups for up to 3 months before anyone looked. And when planning a disk replacement, don't stop investigating once you've found one full-disk cause — a second, unrelated space-consuming problem (stale NFS storage) was independently causing a different failure on a different container.

SMART Reported_Uncorrect climbing + kernel log showing auto reallocate failed means the drive's own bad-sector remapping has stopped working — this is a hard signal to replace the disk, not just monitor it
A backup job silently failing for months doesn't announce itself — check pvesm list <storage> and eyeball the actual backup dates/sizes per guest, don't just trust that a scheduled job exists
Before wiping/replacing a disk, verify every guest has a current backup, not just the ones with scheduled jobs — it's easy to have containers nobody remembered to add to the backup schedule
A single-drive-out-single-drive-in disk swap plus a full disk-space crisis are good opportunities to just do the OS major-version upgrade at the same time, rather than doing it again later
When migrating a long-running HA setup that needs to be down for an extended period, fail over once and remove all time pressure rather than trying to minimize each individual outage — see the HA Recovery doc for the specific pattern
Host-level config that isn't captured by any VM/CT backup (in this case, NUT/UPS setup — USB-attached hardware managed directly by the Proxmox host OS) needs to be identified and captured before wiping the disk, or it's gone. Worth doing an inventory of "what host-level config exists outside of Proxmox's own guest backups" before any planned reinstall.

Documentation Updated¶

Proxmox NUC — Pacific — hardware, storage, and OS version updated; new UPS/NUT section added (previously undocumented entirely)
Proxmox Recovery — Scenario B rewritten with the real steps that worked, marked tested
HA Recovery — added the single-failover strategy for extended rebuilds, and a note on standby VM disk sizing
inventory.md — Pacific's disk, OS version, and NUC model corrected (was incorrectly listed as NUC6CAYH)
current-state.md — Pacific/Lotus sections updated to reflect the temporary HA failover state

2026-07-19: AdGuard Home on Pacific went unreachable — LXC disk filled from an unrotated query log¶

What Happened¶

Dan reported AdGuard Home's primary instance (Pacific LXC 101, 192.168.1.11) unreachable by IP — no obvious network fault, but the admin UI and DNS wouldn't respond.

The Problem¶

Ping to 192.168.1.11 succeeded (network namespace alive), but every service port — 80 (admin UI), 53 (DNS, both TCP and UDP) — was completely closed. That pattern (ICMP up, every TCP/UDP port dead) points at the container's own service having died, not a network-layer fault. Diagnosis was briefly blocked because SSH into Pacific itself needed a password — unlike Lotus/Cooper, no key-based access existed yet.

Once key-based SSH was set up (same brabham-claude key, added via Proxmox's own web Shell — no password needed since the web session is already authenticated), pct exec 101 -- systemctl status AdGuardHome showed the service crash-looping (restart counter at 2636+, Main process exited, code=exited, status=1/FAILURE). df -h / on the container showed the root filesystem — only 3.9GB — at 100% full. AdGuardHome couldn't write its lock file/logs/query log and exited immediately on every start attempt. systemd's journal only showed start/stop events, not the actual reason — the real error only surfaced by running the binary directly (/opt/AdGuardHome/AdGuardHome -s status) and by checking df -h.

The disk fill traced to /opt/AdGuardHome/data/querylog.json (1.1GB) plus its rotated backup querylog.json.1 (1.2GB) — AdGuard Home's default querylog.interval: 720h (30 days) retention, applied to a whole household's DNS query volume, is too much for a 3.9GB LXC disk.

The Solution¶

Set up key-based SSH from Brabham to Pacific (root@192.168.1.10, same key used for Lotus/Cooper) via Proxmox's web Shell.
Stopped the crash-looping service, deleted both querylog files and truncated a stale 220MB AdGuardHome.err, freeing 2.5GB (disk went from 100% to 34%).
Restarted the service — confirmed DNS (53, TCP+UDP) and admin UI (80) both bound and responded externally, then verified with a live Resolve-DnsName lookup through it.
Reduced querylog.interval from 30 days to 7 days in AdGuardHome.yaml (backed up first as AdGuardHome.yaml.bak-20260719) to stop this recurring — left statistics.interval untouched at 30 days.

Lesson Learned¶

"Unreachable via IP" on a host that still answers ping usually means the service died, not the network — check whether ports are actually listening (ss -tulnp / Test-NetConnection) before assuming a network fault, and don't overlook a small LXC root disk silently filling from log/query-log growth.

Ping succeeding + every TCP/UDP service port closed = process/service died, network namespace still alive. Fast, reliable way to distinguish "service crashed" from "network unreachable" before touching SSH.
AdGuard Home's default 30-day query log retention can fill a small LXC disk (this one was only 3.9GB) — worth checking querylog.interval and available disk headroom on any small/legacy container running it.
A service that fails to write its own files due to a full disk typically exits immediately with a generic exit code — systemd's journal shows only start/stop events; run the binary manually or check df -h to find the real cause.
Setting up key-based SSH access mid-incident was still worth doing immediately — same pattern as the Lotus/Cooper setups from 2026-07-18/19 — and made root-cause diagnosis (crash loop, disk %, config values) possible in minutes instead of relying on the web console alone.
Edit a stopped service's config file, not a live one — AdGuardHome had already round-tripped its own 720h value to the shorthand 30d on a prior save, so a first sed targeting the old string silently matched nothing. Always re-read the file immediately after an edit to confirm it actually landed.

Documentation Updated¶

DNS Filtering — Recovery section updated with disk-fill root cause and querylog retention setting
Proxmox NUC — Pacific — Access table updated with key-based SSH; LXC 101 status corrected to reflect current active-primary-DNS role
Homelab Support project CLAUDE.md — added Pacific SSH access details

2026-07-19: Fixing `/mnt/user` doesn't fully restore Cooper's SMB access¶

What Happened¶

The day after fixing Cooper's /mnt/user FUSE crash (see 2026-07-18 entry), Lotus lost its SMB mounts to Cooper again. Cooper's own Shares page still showed shares fine locally, which made this look like a different, new problem — it wasn't.

The Problem¶

Two separate issues stacked on top of each other, both stemming from the array restart used to fix the previous day's shfs crash:

1. Cooper's Samba share export list was empty. /etc/samba/smb-shares.conf — the live config that tells smbd which shares actually exist — was a 0-byte file, timestamped to the exact second of yesterday's array restart. The share definitions were still intact on disk (shareExport="e" in each .cfg file), and Cooper's own Shares page (which reads those .cfg files, not the live Samba config) still displayed them correctly — which is why "shares are viewable locally" was misleading. But smbclient -L localhost on Cooper showed only IPC$, confirming Samba itself had nothing exported. This is why Lotus got NT_STATUS_BAD_NETWORK_NAME — a genuinely different error class from the Transport endpoint is not connected of the day before, and initially confusing because it looked like a fresh, unrelated fault.

Root cause: emhttpd generates smb-shares.conf as part of the array-start sequence, and on the previous day's restart it apparently tried to do so at the exact moment /mnt/user wasn't mounted yet (the same race that killed shfs in the first place), producing an empty file. Editing a share in the GUI and clicking through without changing any value does not force Unraid to rewrite this file — only a fresh Array Stop/Start (with /mnt/user already stable, so no race this time) regenerated it correctly.

2. Lotus had a stale, wedged CIFS mount left over from the outage. Even after Cooper's Samba config was fixed and smbclient from Lotus could read backups fine, the actual mount at /mnt/remotes/192.168.1.60_backups stayed broken: mount still listed it (with a different negotiated SMB dialect than the appdata mount — vers=3.1.1 vs vers=3.0, showing it was an independent, earlier connection attempt that got stuck), but ls/stat on it returned "No such file or directory" and the parent directory listing showed d????????? for that entry — the classic sign of a mount whose handle the kernel can no longer resolve. Clicking "Mount" in Unraid's Unassigned Devices UI did nothing, because as far as the OS was concerned the share was already mounted. It took a direct umount -l /mnt/remotes/192.168.1.60_backups on Lotus via SSH before a fresh Mount click actually worked.

The Solution¶

In order: 1. Confirmed /mnt/user was actually fine on Cooper this time (it was — the previous day's fix held). 2. Found smb-shares.conf empty via SSH; tried clicking "Done" on the backups share settings page with no value change — did not help. 3. Did another Array Stop → Start on Cooper (now safe, since /mnt/user wasn't racing this time) — smb-shares.conf regenerated correctly with all four shares. 4. Verified from Lotus with smbclient -L //192.168.1.60 directly — shares were now visible and readable over SMB, independent of the mount layer. 5. backups still failed to mount via the UI. Found the stale wedged mount via mount | grep + ls -la showing d?????????. Force-cleared it with umount -l from Lotus directly via SSH. 6. Fresh "Mount" click in the UI then worked immediately.

Lesson Learned¶

After any Unraid array restart used to fix a share-export problem, check three layers before assuming it's fixed: the FUSE mount, the server's live Samba config, and every client's existing mount state — fixing the first doesn't guarantee the other two came back clean.

/etc/samba/smb-shares.conf being empty produces NT_STATUS_BAD_NETWORK_NAME on clients — a different, more confusing error than the Transport endpoint is not connected symptom of the underlying FUSE bug, even though both trace back to the same mount race during array start
Unraid's Shares page reflects the on-disk share config, not the live Samba export list — the two can silently diverge, so "shares look fine locally" doesn't rule out a Samba-side problem
Editing a share via the GUI and clicking Done/Update without changing a value does not force Unraid to regenerate its Samba config — only an array restart (or presumably a genuine field change) does
A client's CIFS mount that broke during a server-side outage does not self-heal once the server is fixed — check for a stale wedged mount (d????????? in ls -la of the parent directory, or a different SMB dialect than a working sibling mount) before assuming a persistent remount failure means the server is still broken
Having direct SSH access to both Lotus and Cooper (set up over these two days) was what made diagnosing this quickly possible — checking only one side would have missed half the picture each time

Documentation Updated¶

Backup NAS — Cooper — added a "Full recovery checklist" covering all three layers, since the single-command shfs fix alone doesn't fully resolve this class of fault

2026-07-18: Cooper's `/mnt/user` share crashes are an shfs/FUSE bug, not an array fault¶

What Happened¶

Cooper's Shares page periodically showed "There are no exportable user shares," breaking Lotus's SMB mounts of Cooper. A previous note (2026-06-26) attributed this to a "suspected driver crash" and recorded rebooting Cooper as the fix. This recurred, so it was investigated properly this time with direct SSH access set up for the first time (previously only had a port-22 assumption; Cooper actually runs SSH on port 50123).

The Problem¶

The array and every individual disk (/mnt/disk1, /mnt/disk2, /mnt/cache) were completely healthy the whole time — zero errors, all mounted, mdState=STARTED. The actual failure was narrowly scoped to /mnt/user, the FUSE mount produced by the shfs process that pools disk1+disk2+cache into user shares. mount still listed it as mounted, but any access returned Transport endpoint is not connected — a classic dead/orphaned FUSE channel. Since every user share lives under /mnt/user, this alone was enough to make the whole Shares page look broken.

Tracing back through /var/log/syslog and its one rotation (syslog.1) pinned the exact break window to a 26-minute gap on Jul 12 (04:57–05:23), with no kernel-level event whatsoever in that window — no USB drop, no disk error, no OOM. That rules out a hardware/driver crash and points to shfs itself dying (a known class of Unraid FUSE bug), silently, with nothing else on the system noticing until the next SMB access failed. /mnt/user0 (same pool, minus the cache drive) stayed healthy both times this has now happened, making the cache drive's involvement in the pool the leading suspect for what triggers the crash — though the precise trigger is still unconfirmed.

Also discovered: a plain Array Stop/Start from the GUI does not fix it. Unraid's stop sequence doesn't force-unmount a wedged FUSE mount, so mkdir /mnt/user fails on the next start (directory already exists in the broken transport state), and the replacement shfs process dies immediately trying to chdir into it (confirmed directly in emhttpd's own startup log: shfs: error: main, 4178: Transport endpoint is not connected ... exit status: 255). This is why only a full reboot had ever appeared to fix it — reboot is the only thing that reliably wipes the stale mount state.

The Solution¶

Set up SSH access to Cooper properly first (ssh Cooper alias added to ~/.ssh/config on Brabham, port 50123, key added via Unraid's Settings → Management Access → User 'root' → Manage). Then, instead of a full reboot:

umount -l /mnt/user
/usr/libexec/unraid/shfs /mnt/user -disks 7 -o default_permissions,allow_other,noatime -o remember=330
/etc/rc.d/rc.samba restart

This lazy-unmounts the wedged mountpoint and re-runs the exact shfs invocation emhttpd uses at array start, clearing the stale state without touching the array or rebooting the box. Confirmed working — shares reappeared immediately and syslog stopped logging canonicalize_connect_path/Transport endpoint errors.

Lesson Learned¶

"The array looks fine but shares are gone" on Unraid usually means the /mnt/user FUSE layer died, not the array — check df -h /mnt/user before assuming a driver/hardware fault.

A dead FUSE mount can coexist with a perfectly healthy array; mount will still list it, df/any file access will return Transport endpoint is not connected
Array Stop/Start does not clear a wedged FUSE mount — only umount -l + re-invoking shfs (or a full reboot) does
The exact shfs mount command Unraid uses is always in the boot/array-start log (grep 'shfs /mnt/user' /var/log/syslog) — useful for reproducing it manually rather than guessing the disk bitmask
/mnt/user0 (shares pool without the cache drive) is a useful diagnostic: if it stays healthy while /mnt/user dies, the cache drive's presence in the pool is implicated
Grabbing a diagnostics zip (Tools → Diagnostics) the moment a fault is noticed, before touching anything, is the only way to preserve evidence — Unraid's logs are in RAM and get wiped by remounts/reboots
Setting up real SSH access (not just documenting a theoretical ssh root@ip command) made this whole investigation possible — previously the docs assumed port 22, which isn't what Cooper actually runs

Documentation Updated¶

Backup NAS — Cooper — corrected SSH port, added full Troubleshooting section with root cause and fix
Homelab Support project CLAUDE.md — corrected the 2026-06-26 troubleshooting note, added SSH access details, corrected the Docker container list (2 containers actually run on Cooper, not none)

What Happened¶

The Appdata Backup plugin on Lotus reported a FAILED error when copying the flash backup to /mnt/remotes/COOPERDOMAIN_backups/lotus/justtheusb/. The justtheusb folder was visible on Cooper directly but not visible via the SMB mount on Lotus. All the ab_* appdata backup folders appeared fine.

The Problem¶

The Unassigned Devices remote share was configured with ip="COOPER.LOCALDOMAIN". On Lotus, COOPER.LOCALDOMAIN resolved to 127.0.0.1 (Lotus itself) rather than Cooper at 192.168.1.60. The mount output confirmed this — addr=0.0.0.0 and disk space shown matched Lotus, not Cooper.

Lotus was effectively mounting its own backups share and writing to itself. The ab_* appdata backup folders happened to exist on both machines (from previous successful runs), making the mount appear to be working. The justtheusb folder only exists on Cooper, which is why it was the only entry missing from the listing.

Editing the samba_mount.cfg file directly was not sufficient — the change wasn't picked up cleanly. The fix required deleting the remote share entry in Unassigned Devices completely and re-adding it using the IP address directly.

Lesson Learned¶

Always use IP addresses for Unassigned Devices remote SMB shares — never hostnames.

Hostnames like COOPER.LOCALDOMAIN or COOPER.LOCAL can resolve incorrectly on Unraid, especially at boot before DNS is ready
A stale or wrong mount can look healthy if the destination share happens to contain similar-looking folders
Verify a remote share is actually pointing to the right machine by checking the disk space shown in Unassigned Devices — it should match the remote machine, not Lotus
Editing samba_mount.cfg directly doesn't reliably update a live mount entry — delete and re-add in the UI instead

Documentation Updated¶

Backup strategy — Appdata Backup plugin section expanded with flash backup detail and Cooper path info

2026-04-27: LuckyBackup was backing up the wrong Immich path for 9+ months¶

What Happened¶

Investigated why recent photos couldn't be found at the expected filesystem path. Discovered that Immich's upload storage path had changed at some point from /mnt/user/data/media/immich/photos/ to /mnt/user/immich/photos/. LuckyBackup was still configured with the old source path, which contained zero active files.

The Problem¶

Immich's Docker container had its /photos mount changed from /mnt/user/data/media/immich/photos/ to /mnt/user/immich/photos/
All phone uploads since ~July 2025 were going to the new path
LuckyBackup's backupImmichPhotos task still pointed to /mnt/user/data/media/immich/ — backing up nothing useful
51,000+ photos had no backup on Cooper for 9+ months

The Solution¶

Updated LuckyBackup source path from /mnt/user/data/media/immich/ to /mnt/user/immich/ and destination from root@192.168.1.60:/mnt/user/data/media/immich/ to root@192.168.1.60:/mnt/user/immich/.

The old path contained an orphaned snapshot of pre-July 2025 uploads. A comparison showed 429 files in the old path missing from the new path — these were confirmed to be intentionally deleted photos and will not be copied across.

Lesson Learned¶

After any Docker container reconfiguration that changes volume mount paths, immediately verify that backup tools are still pointing at the correct source paths.

Check backup source paths match actual container mounts: docker inspect <name> --format '{{json .Mounts}}'
A backup job completing with "OK" and no errors does not mean it's backing up the right data — it just means rsync ran without errors
Verify file counts at the destination periodically, not just backup job status

Documentation Updated¶

Lotus server doc — Immich storage layout section added with correct paths

2026-04-25: Unraid Community Apps port mapping cannot be safely edited — use a socat relay instead¶

What Happened¶

Setting up PostgreSQL streaming replication for Immich required Cooper to connect to Lotus's PostgreSQL on host port 5432. The connection was refused even though the port appeared to be listening.

The Problem¶

The Immich PostgreSQL container was installed via the Unraid Community Applications store as part of a pre-configured bundle. The Unraid template had a broken port mapping: host:5432 → container:5433. PostgreSQL runs on container port 5432, so the host port forwarded to nothing.

The template uses Display="always-hide" for the port field, meaning it can't be changed through the Unraid UI. Editing the XML directly and applying via the UI recreated the container from the running config rather than the updated XML. Removing and recreating the container risked breaking the CA-managed bundle.

The Solution¶

Add a lightweight socat relay container on the same Docker network as the database:

docker run -d \
  --name immich-pg-relay \
  --restart unless-stopped \
  --network arrproxy \
  -p 5452:5452 \
  alpine/socat \
  TCP-LISTEN:5452,fork,reuseaddr TCP:immich_postgreSQL:5432

This exposes the database on a new host port (5452) without touching the CA container. The replica connects to 192.168.1.80:5452.

Lesson Learned¶

When a Community Applications container has a broken or locked port mapping, don't fight the template — add a socat relay container on the same Docker network instead.

socat relays are lightweight, reliable, and completely non-invasive to the existing stack
pg_hba.conf must allow the relay's Docker network subnet (e.g. 172.18.0.0/16) as well as the external client IP, because socat forwards from within the Docker network
The relay container needs --restart unless-stopped so it survives reboots

Documentation Updated¶

Lotus server doc — Immich PostgreSQL replication section added
Cooper server doc — Immich PostgreSQL replica section added

2026-04-25: Rsyncing SWAG appdata to Cooper caused Tailscale node ID conflict¶

What Happened¶

A second SWAG instance was set up on Cooper as a standby reverse proxy, with config files rsync'd hourly from Lotus. The initial rsync job copied the entire SWAG appdata folder, including the Tailscale state directory.

The Problem¶

SWAG has Tailscale integrated and stores its own node identity (node ID, keys) inside appdata/swag/.tailscale_state/
Copying this folder to Cooper gave both SWAG instances the same Tailscale node ID
Tailscale treats duplicate node IDs as a conflict — the primary Lotus instance was kicked off the Tailnet
External access via *.djchome.uk broke because SWAG on Lotus lost Tailscale connectivity

The Solution¶

Excluded the Tailscale state directory from the rsync job:
```
/mnt/cache/appdata/swag/.tailscale_state/
```
Deleted the copied .tailscale_state folder from Cooper's SWAG appdata
Cooper's SWAG required a Tailscale auth key (TAILSCALE_AUTHKEY env var) to authenticate as a fresh node, as the browser auth flow hung without it

Lesson Learned¶

When rsyncing Docker appdata between two hosts, always exclude any folder that contains node identity data (Tailscale, WireGuard keys, machine IDs, etc.).

Each SWAG instance must have its own independent Tailscale identity
The rsync exclusion must be permanent — re-adding the Tailscale folder at any point will re-trigger the conflict
If a containerised Tailscale node hangs on first auth after clearing state, use a pre-generated auth key via TAILSCALE_AUTHKEY env var rather than the browser flow
Check for similar identity folders if replicating any other VPN-integrated containers

Documentation Updated¶

Cooper server doc — SWAG section updated with correct state directory path and failover script details

2026-07-19: Immich v3 update caused Postgres/Redis ETIMEDOUT — Tailscale's in-container DNS races Node's default lookup¶

What Happened¶

After updating the immich container (ghcr.io/imagegenius/immich) to v3.0.x, the server crash-looped on startup with ETIMEDOUT connecting to immich-postgres:5432 and the Redis client, even though immich_postgreSQL and immich-redis were both healthy and reachable.

The Problem¶

Immich's container has Unraid's native per-container Tailscale integration enabled (intentional — it Tailscale Serves the app directly at https://immich.<tailnet>.ts.net for remote/mobile access). tailscaled runs inside the container with --accept-dns (default), so it continuously manages /etc/resolv.conf, pointing it at Tailscale's MagicDNS proxy (100.100.100.100) instead of Docker's embedded DNS (127.0.0.11).

The v3 update bumped the image's underlying Node.js version (confirmed Node v24). Newer Node's default dns.lookup() (used by net.connect(hostname), i.e. every Postgres/Redis connection) fires the IPv4 (A) and IPv6 (AAAA) queries in parallel over one UDP socket — the classic glibc "happy eyeballs" pattern (see moby/moby#32106). Tailscale's local DNS proxy doesn't handle that pattern cleanly and stalls ~4 seconds before returning the correct answer. Under connection-pool load, that's long enough for every DB/Redis connection attempt to time out outright, so the app never started.

Confirmed directly inside the running container: - Connecting by raw container IP: instant - dns.lookup(hostname) (default mode): ~4000ms - dns.lookup(hostname, {verbatim: true}): ~10ms - Adding options single-request to /etc/resolv.conf: default lookup dropped to ~14ms

Editing /etc/resolv.conf directly (or via Docker's --dns-option) didn't stick, because tailscaled re-asserts its own resolv.conf continuously while running, not just at container start.

The Solution¶

Two additive changes to the Unraid container config (no other settings touched): 1. Extra Parameters: --dns-option single-request — has Docker write the option into the container's resolv.conf 2. Tailscale Parameters: --accept-dns=false — stops tailscaled from taking over /etc/resolv.conf, so Docker's own DNS (with the option above) is what's actually used for internal container-name resolution. The container keeps its Tailscale identity and Serve endpoint; only DNS management is disabled.

Lesson Learned¶

When a container has Unraid's native Tailscale integration enabled, tailscaled owns /etc/resolv.conf for the life of the container — any Docker-level DNS option (--dns-option, --dns) is silently ineffective unless --accept-dns=false is also set in the Tailscale Parameters.

This is a two-layer fix: the --dns-option single-request alone does nothing while accept-dns is on; --accept-dns=false alone leaves the default (buggy) Node DNS behavior in place. Both are needed.
Diagnose this class of bug by comparing dns.lookup(host) timing against dns.lookup(host, {verbatim:true}) and a raw-IP net.connect() — a big gap between default and verbatim timing points straight at the parallel A/AAAA race, regardless of which DNS server is upstream.
ETIMEDOUT/CONNECT_TIMEOUT between two healthy, mutually-reachable containers on the same Docker network is a strong signal to check DNS resolution timing before assuming a networking/firewall fault.
Immich's machine learning service runs separately on Brabham (not a Lotus container) — logged Machine learning server became unhealthy messages when Brabham/the ML service isn't running are expected, not a fault.

Documentation Updated¶

CLAUDE.md in the Homelab Support project — added immich-pg-relay to the Lotus container table and noted the Brabham ML service architecture

2026-08-02: A backup written 18 minutes before a power cut did not survive — page cache is not disk¶

What Happened¶

While switching Mealie's Docker tag from nightly to stable, a pg_dump of the Mealie Postgres database was written to /mnt/user/backups/mealie/ at 15:38 as a pre-change safety net. ls confirmed the file, 503K, present and correct.

Lotus lost power accidentally at roughly 15:56. After it came back and the array started, the file was gone — a find across /mnt/cache and all three data disks turned up no mealie-db-* at any path.

The Problem¶

The dump had been written through the shfs FUSE layer and was still sitting in the Linux page cache. Nothing had forced it to the physical disk in the intervening 18 minutes, so the unclean shutdown discarded it entirely.

The dangerous part is that there is no observable difference. For those 18 minutes, ls -lh /mnt/user/backups/mealie/ showed the file with the right name, size and timestamp. A backup that exists only in page cache is indistinguishable from one that is safely on disk, right up until the moment you need it.

A second, milder trap sat on top of it: immediately after the reboot, /mnt/user/backups/ returned "No such file or directory" simply because shfs had not finished starting. That looks identical to data loss but is not — the shares appeared normally a short time later. Don't diagnose a missing share in the first minute after an array start.

The Solution¶

Re-took the dump and then verified it physically, rather than trusting the share view:

docker exec mealie-postgres pg_dump -U mealie -d mealie -Fc -f /tmp/m.dump
docker cp mealie-postgres:/tmp/m.dump /mnt/user/backups/mealie/mealie-db-20260802-pre-latest.dump
sync
find /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/cache -maxdepth 4 -name 'mealie-db-*'

The find returning /mnt/disk3/backups/mealie/mealie-db-20260802-pre-latest.dump is the proof — a real path on a real disk, not the /mnt/user union view.

Lesson Learned¶

A file you can see under /mnt/user is not necessarily a file that exists on a disk. After writing anything you are relying on as a safety net, sync and then confirm it appears under /mnt/diskN or /mnt/cache.

This matters specifically for ad-hoc backups taken right before a risky change — exactly when the window between writing and needing it is shortest, and exactly when the change itself may involve a reboot.
Scheduled backups are much less exposed, because hours of subsequent writes and normal writeback flush them long before anything goes wrong.
The verification costs one sync and one find. There is no reason to skip it.
Corollary for any "did I lose data?" check after an unclean shutdown: search the physical paths, not the union mount, and wait for shfs to finish coming up before believing an empty result.

Also Confirmed The Same Day¶

Lotus DOES auto-boot after a power cut, coming back and starting the array unattended. This is the opposite of Cooper, which needs a physical power-button press (2026-07-29). Lotus's behaviour had been explicitly recorded as untested; it is now tested. Pacific remains untested — don't assume.
An unclean shutdown triggers an automatic correcting parity check (mdResyncAction=check P, mdResyncCorr=1). Expected, runs for hours, blocks nothing.
mdNumDisabled=1 / mdNumInvalid=1 on Lotus is benign, the same empty-slot artefact already documented for Cooper — slot 29 reports DISK_NP_DSBL while all four real disks report DISK_OK. Check rdevStatus.* per disk rather than trusting the summary counters.
Docker applies an edited Unraid template on boot. The my-mealie.xml repository change had been made but not yet applied via the UI; the reboot recreated the container from the template, completing the tag switch with all env vars (TOKEN_TIME=8760) intact.

Documentation Updated¶

CLAUDE.md in the Homelab Support project — corrected the Cooper power-cut note to record Lotus's confirmed auto-boot behaviour, and noted Mealie's move to the stable tag

Time	Kettle plug	House meter
23:33:27	3001 W	510 W
23:33:31	2976 W	2250 W
23:33:35	2976 W	3260 W

Lessons Learned¶

General Principles¶

Log¶

2026-08-03: Shelfmark forgot its settings on every update — the rebrand moved the config volume¶

What Happened¶

The Fix¶

Follow-up, 2026-08-04 — reverse proxy and dashboard¶

Lesson Learned¶

2026-08-03: A 60-book import took Calibre-Web down, and the sync that "succeeded" silently dropped 10 books¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-08-02: Multiroom audio — four things that each broke it silently¶

What Happened¶

The Problems¶

Lesson Learned¶

Documentation Updated¶

2026-07-22: Lotus rebooted twice with zero trace in its own logs — the smart plug between the UPS and the NAS was crashing¶

What Happened¶

What Was Ruled Out¶

Root Cause¶

Why Every Log Was Silent¶

Lesson¶

Resolution¶

Unrelated Change Made the Same Day¶

2026-07-29: Smart plug failures track load, not age — and three plugs were not what they claimed to be¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-29: Cooper does not come back on its own after a power cut¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-29: A Powercalc sensor reading unavailable usually means its source device is offline¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-29: SSH to Home Assistant failed with "Corrupted MAC on input" — one MAC algorithm was broken, not the key¶

What Happened¶

Root Cause¶

Fix¶

Lessons¶

2026-07-29: Ring alarm sensor bypass works — but ring-mqtt won't tell you what it bypassed¶

What Happened¶

What Was Already There¶

The Limitation That Shaped Everything Else¶

Lessons¶

2026-07-29: A dead smart plug jammed the living room's WiFi channel for three days — and only the AP's own radio stats could see it¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-29: Zigbee link quality was impossible to measure — 76 LQI sensors had been disabled all along¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-28: NTP was broken for every device on the NoT VLAN — a firewall rule was matching on SOURCE port¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Documentation Updated¶

2026-07-28: A smart plug reported 0 W through a 3.2 kW kettle boil — TelePeriod was hiding every short event¶

What Happened¶

The Problem¶

The Solution¶

Lesson Learned¶

Getting Firmware to an Internet-Isolated Device — Without Touching the Firewall¶

Documentation Updated¶

2026-07-29: A Powercalc sensor reading `unavailable` usually means its source device is offline¶

2026-07-28: A smart plug reported 0 W through a 3.2 kW kettle boil — `TelePeriod` was hiding every short event¶

2026-07-19: Fixing `/mnt/user` doesn't fully restore Cooper's SMB access¶

2026-07-18: Cooper's `/mnt/user` share crashes are an shfs/FUSE bug, not an array fault¶