The FAI workstation: SSOT, Wake-on-LAN and the mount

Two modules: modules/home/network/fai-workstation.nix (the host's SSOT plus wake-workstation) and modules/home/services/fai-workstation-mount.nix (the rclone SFTP mount).

The SSOTCopy Anchor Link

The IP and user were literals repeated in 3 consumers (ssh.nix, the rclone mount, and the wake script). Now they come from one option. All the consumers are in modules/home/, so the option lives in modules/home/ too: rule 11 says it lives at the LOWEST level that needs it. Contrast with my.monitors, which had to move up because a system service needed it.

Wake-on-LAN, and why three pathsCopy Anchor Link

A magic packet is 6x 0xFF plus 16x the MAC. Not nixpkgs' wakeonlan, which is Perl and drags Perl-Critic and Perl-Tidy into the build. Since 22/10/2026 (rule 24) the LOCAL paths send it with socat, the packet written as printf escapes at build time; MEASURED: the same 102 bytes as the old bytes.fromhex, received on loopback with broadcast set.

THE RELAY STAYS PYTHON, as a registered exception (languages): it runs ON the fai-vm, where I cannot install anything, and the only unprivileged broadcaster there is python3 (its nc is OpenBSD's and busybox nc has no broadcast either, checked 22/10/2026).

SO_BROADCAST is mandatory: without it the kernel refuses to send to a broadcast address. Ports 9 AND 7, because an old NIC sometimes only listens on 7.

wake-workstation tries three paths, in this order:

1. A relay through the fai-vm. It broadcasts on the L2 segment, so it wakes a machine that has been off FOR A WHILE. The script goes through the remote python3's STDIN, so there is nothing to install on the other side. It needs YOUR key authorized on the fai-vm (ssh-copy-id); when the relay fails the script prints ssh's own error, because guessing the cause once sent me after the host key.
2. Unicast through the tunnel, which works while the router's ARP is still warm.
3. A directed broadcast to the /25. A router usually drops these (RFC 2644), but the FAI gateway forwards it as an L2 broadcast (measured 16/10/2026, tcpdump -e on the workstation: 2e:b8:ed:8f:67:90 > ff:ff:ff:ff:ff:ff). So this path does not need a warm cache either, and it is the fallback that still wakes the machine when the fai-vm is down. A gateway policy change can take it away without notice, which is why the relay stays first.

It checks for a ppp link first, because with no tunnel there is no route to FAI, and failing there with the right cause avoids chasing ghosts. Then it pings until a 120 s deadline.

ws: a shell that outlives the tunnelCopy Anchor Link

The SonicWall resets the VPN every 30 min to 3 h (vpn.md), and the reconnect usually brings a new IP, so no TCP session survives it. Nothing on this side prevents that, so the work lives in a tmux ON the workstation (tmux 3.6 is already installed there, and running it changes no config) and ws [session] reattaches to it in a loop: ssh's own failure (exit 255) means the tunnel dropped, so it waits 3 s and reattaches; a detach or an exit returns anything else and ends the loop. Combined with the master drop on IP rotation, it is back about as soon as the tunnel is. mosh would roam across the IP change by itself, but it needs mosh-server installed over there, and that machine is not mine to configure.

The mountCopy Anchor Link

~/FAI-workstation is the workstation's whole root over rclone SFTP with a VFS cache, so it shows up in Dolphin as a normal folder and rereads from disk instead of the network.

"/" and not just the home: the user sees what they have permission to read, and the rest shows up but stays inaccessible, which is normal for a non-root.

It is VPN-gated, not boot-mounted. autoMount = false; vpn connect fai starts the mount and a disconnect takes it down. The host 200.136.209.229 only exists through the VPN, so a boot-time mount would be a ghost/stale mount. SSHFS would FREEZE Dolphin in that scenario; rclone fails clean and reconnects.

SFTP through key_file, the SSH key that already exists, so there are ZERO secrets in Nix. The rclone.conf is generated by the module; only paths and the host are declared here.

The VPN comes up asynchronously, taking ~10 s until the tunnel and routes exist, so StartLimitIntervalSec is cleared: without it systemd would give up after 5 quick failures before the host became reachable. It retries every 10 s until it connects, and since vpn connect fai gives the start and disconnect gives the stop, there is exactly one owner.

The cache options are the same shape as ~/Drive (see restic.md): vfs-cache-mode = "writes" means READS go straight through, streaming, without piling up on disk, and only what you write is cached until it uploads. Switch it to "full" for maximum speed reopening files, at the cost of filling the cache. dir-cache-time = 5m makes browsing FAST, with F5 to reload; timeout/contimeout keep it from hanging forever when the VPN drops.

The other direction, one file at a time, does not go through the mount: clipboard-push scp's what is in the clipboard into a drop on the workstation and hands the remote path back, which is how an image reaches a TUI over ssh. See desktop-plumbing.md.

The receiving side is NOT declarable from hereCopy Anchor Link

The workstation is somebody else's Ubuntu 26.04 (superintendencia-server), out of these dotfiles' reach, and sudo over there asks for a password. To arm WoL, run ON THE WORKSTATION, once:

sudo ethtool enp7s0 | grep -i wake   # "Supports Wake-on: ...g" means it can be used
sudo ethtool -s enp7s0 wol g         # arms it now (it does NOT survive a reboot)

To persist it: that machine uses NETPLAN plus systemd-networkd, NOT NetworkManager, so there is no nmcli. In /etc/netplan/00-installer-config.yaml, under ethernets: enp7s0:, add wakeonlan: true and run sudo netplan apply. It usually also requires "Wake on LAN/PCIe" TURNED ON in the BIOS/UEFI.

The measured state (16/10/2026): ARMED and persistent. sudo ethtool enp7s0 gives Supports Wake-on: pg and Wake-on: g, and the netplan file carries wakeonlan: true (the 00-installer-config.yaml.bak-prewol next to it is the copy from before arming it, 16/06).

Do not read /sys/class/net/enp7s0/device/power/wakeup as proof. It says disabled with WoL armed, and that misread is what made the july note call it "NOT armed". Only ethtool counts, and its Wake-on fields need root ("netlink error: Operation not permitted" without it), so the read has to be done by hand on the workstation.

Checking the path while the machine is on: plain wake-workstation quits early when the host answers, so --test sends anyway and skips the wait. Watch it land on the workstation:

sudo tcpdump -ni enp7s0 'udp and (port 9 or port 7)'   # on the workstation
wake-workstation --test                                 # here

That proves the packet reaches the NIC over L2. It does NOT prove a wake from S5: with ErP/Deep Sleep on in the BIOS the NIC loses power when the machine is off, and only a real power-off test answers that.

The power-off test (16/10/2026): it WAKES. With the machine off in S5, wake-workstation booted it (up at 16:00:52, answering ping about 33 s after the send), so the BIOS keeps the NIC powered. All three paths go out together, so which one did the waking is not known.

It does NOT wake after a power loss (17-19/10/2026). The machine died abruptly on Saturday 17/10 at about 14:18 and no magic packet brought it back; FAI's IT had to press the button on Monday. The relay was fine: the packet left the fai-vm's eth0 as an L2 broadcast, but the workstation was missing from the fai-vm's ARP table (FAILED). enp7s0 is an add-in AQC113 card whose WoL is armed by the atlantic driver on a clean shutdown, so after AC loss nothing arms it and only a button press (or the BIOS) boots the machine. The fix belongs to whoever runs the machine: Restore AC Power Loss = Power On in the BIOS (an ASUS TUF Z790-PLUS: Advanced, APM).

Debugging an outage without rootCopy Anchor Link

The workstation runs a full monitoring stack (Prometheus with 1 year of retention, Loki, Grafana, blackbox, node and smartctl exporters) whose data survives a reboot, and that is what answered the Saturday question:

• Prometheus is on 127.0.0.1:9091. The 9090 there is something else and returns HTML. Query it with a script piped over ssh (ssh workstation python3 - < script.py). It also holds the fai-vm's node metrics (remote write), so it shows both machines.
• Trust Prometheus over the journal for the time of death. journald syncs every 5 min, so a hard stop loses the tail: the journal ended at 14:12:46 while every scrape target stopped at 14:18:30.
• The kernel log needs sudo (sudo journalctl -b -1 -k). On 17/10 it ended with no panic, MCE or watchdog, and the node metrics showed a healthy machine to the last sample (load 2 to 4, 50 to 73 °C, 40 GB free): a power cut, most likely. The fai-vm had rebooted at 13:06 the same afternoon.
• SMART without sudo: docker exec mon_smartctl_exporter smartctl -a /dev/nvme0 (smartctl is not installed on the host). Unsafe Shutdowns was 15 on 19/10. It counts ANY unclean stop, a hang plus a held power button too, so it tells clean from unclean, not power cut from hang.

Two side findings from that day, both reported to IT rather than touched (it is not my machine):

• The 10 GbE card logs PCIe correctable errors, about 17 per minute (RxErr 932 within 55 min of boot, /sys/bus/pci/devices/0000:07:00.0/aer_dev_correctable, readable without root). That is signal integrity on its PCIe 4.0 x4 link: reseat the card or pin that slot to Gen3. Correctable errors do not stop a machine, so this did not explain the outage.
• The FAI SSL-VPN went away on its own on 19/10 from 08:54 to 09:23: pppd saw Peer not responding mid-session, then 200.133.233.101:4433 stopped accepting connections from my network AND from the VPS, while the firewall kept answering ping and the internal blackbox probes stayed green. Nothing on my side causes that; only the SonicWall's own logs can.