hex

Executive summary

This is the delivery document for hex: one Minisforum MS-S1 MAX running Proxmox VE 9.2.26, one Beelink running Proxmox Backup Server 4.2.9-1, and everything that lives on them. We built it, we run it, and every number on these pages was pulled live from the machines on 2026-11-13 at 10:47 PM EST.

The week in one chart

What you are getting

• Compute: AMD Ryzen AI MAX+ 395, 16 cores / 32 threads, 123 GiB unified memory with up to 80 GiB addressable by the Radeon 8060S. See Hardware.
• Storage: two NVMe thin pools (local-lvm on the Kingston 2TB, fast-lvm on the Samsung 980 PRO 2TB) plus an 8TB ZFS FireCuda X Vault. See Storage architecture.
• Backups: nightly snapshot vzdump to hex (primary) over a dedicated hex <-> back direct link, then fanned out to vault (8TB), usb-sandisk (go disk) and B2 offsite.
• Services: the Cognition Mesh gateway (VM 104 cm), the fleet NAS (VM 301 umbrel), the NixNet GPU share (CT 310 nixnet + CT 300 ai), Pi-hole (CT 103 pihole), Flox and HyperNix labs, and Larry's agent sessions.
• Resilience: UPS with a homemade mains-loss detector (power-watch), ordered shutdown of back then hex, Gmail alerting and four staleness / coverage watchdogs.

Honest caveats

Memory is overcommitted on paper: running guests are assigned 208 GiB against 123 GiB physical, mostly because CT 300 is allowed 80 GiB for model loading. Actual use averaged 39 GiB. Swap shows 12.1 GiB in use (16 GiB zram + 8 GiB disk). The old OWC copy reports 4 ZFS data errors and is being retired into the vault. Anything we could not prove live is tagged unverified and collected in Appendix: unverified facts.

Hardware

Every disk

Network interfaces

hex ships two 10GbE ports and both negotiate at 1 Gbps: nic0 because of the LAN switch side, nic1 because back's Realtek RTL8111 tops out at gigabit. That 1 Gbps direct link is the ceiling of every backup, sync and NFS byte on this platform, and it never touches the LAN. The router is a UniFi Dream Router at x.x.x.x model from Fred; router not queried; the switch model is not captured.

The CPU package read 86 °C at capture with load around 4; that is within AMD's limits for this part but it is the warmest number on the page, so it goes on the roadmap as something to watch.

Power

hex, back and the router sit on a GoldenMate 1000VA LiFePO4 UPS model and the ~45-60 min at ~120 W runtime are from Fred. The UPS has no data port, so power-watch detects mains loss by asking two laptops that are not on the UPS whether their chargers are live.

Storage architecture

Storage inventory

Thin provisioning: what guests asked for versus what they wrote

local-lvm holds 50 guest volumes (including 4 MiB EFI and cloud-init volumes) for 32 guests. Guests were promised 1.34 TiB on a 1.76 TiB pool, and have actually written 356 GiB. fast-lvm is the fast tier: 1.73 TiB promised on 1.70 TiB, 694 GiB written. Thin metadata is at 0.86% and 1.47%. Overcommit is deliberate; the 900G rootfs of CT 300 ai is mostly model cache that is never backed up ("never back up what we can download").

ZFS pool vault

Pool vault: 7.27T, 728G allocated, 9% full, 0% fragmentation, ONLINE, "No known data errors". Single-disk by design: it is a copy target, never the only copy.

The NFS export is pinned to one client: /vault/pbs x.x.x.x/32(rw,sync,no_root_squash,no_subtree_check). back mounts it with a systemd automount (nfs4.2, hard, nofail), so a missing vault never blocks back's boot.

Backup datastores (on back)

VM 301 umbrel: the fleet NAS

umbrelOS 2.0.1. Serves NFSv4/SMB for the fleet from ZFS pool tank on the 600G scsi1 disk (Projects, Sessions, scratch, Google Drive share, /srv/git). LAN x.x.x.x + x.x.x.x, tailnet nas x.x.x.x.

Disks

The 600G tank disk (serial UMBRELTANK01) carries a native ZFS pool inside the guest holding Projects, Sessions, scratch, the Google Drive share and the git remotes. It is 33.0% written. Compression was switched to zstd on Nov 12; ZFS dedup stays off because zdb -S measured only 1.33x and the VM has 8 GiB. hex itself mounts the shares at /mnt/nas/Projects, /mnt/nas/Sessions and /mnt/nas/scratch, which is how hex.sessions containers reach project folders.

Network

DHCP on the Home LAN hands it x.x.x.x; a second address x.x.x.x carries the legacy NAS IP so old clients never noticed the swap. On the Tailscale tailnet it owns the MagicDNS name nas at x.x.x.x.

Backup coverage

Jobs: hex-pbs-nightly (03:00, snapshot mode). Both disks are included (backup=0 was removed from scsi1 on Nov 12). On the go disk filter: yes.

Restore proof: on Nov 13 at 3:42 PM EST its qemu-server.conf was restored from the go disk and matched the live config. The full rollback for the old NAS, vm/501/2026-11-12T06:07:31Z, is still on hex (primary) and verified ok.

Why it exists

The NAS used to be its own FygoOS VM whose ZFS features umbrelOS could not import read-write. Rather than gamble on an in-place upgrade, we copied 225 GB into a fresh pool in 21 minutes, cut over on the LAN, then moved the tailnet identity. Every client mounts by name, so the next migration is just as boring.

Appendix A: full inventory (generated)

Generated straight from the data model. Each row is a link target for the cross-references in this document; guests with a full chapter link onward.

Networks

Backup datastores

Appendix C: unverified facts

• unverified UPS model (GoldenMate 1000VA LiFePO4) and runtime (~45-60 min at ~120 W): from Fred, not readable (the UPS has no data port, per power-watch.sh).
• unverified UniFi Dream Router and switch models/ports: not queried (UniFi bot territory). Gateway x.x.x.x is live.
• unverified Kingston SMART power-on hours (13) contradict 21.6 TB read; counter treated as unreliable.
• unverified Larry age (10 months) from Fred.
• unverified VM 360 role: RESTORE.md says Forgejo host, Forgejo brief says Forgejo runs on VPS "for"; VM 360 host key changed, so not inspected.
• unverified VM 351 flox-mgmt: no guest agent, not inspected inside.
• unverified B2 monthly cost (~$2/month) is from the Nov 8 brief, not from a bill.