Switching the Packer builder

At the end of 2023, Packer 1.10 in packer-builder-arm broke the Ansible provisioner. I pinned the last working Docker image for almost two years. For a newer RaspiOS (Trixie) I switched to packer-plugin-cross, which comes as a ready Docker image. The configs are in HCL now.

Ansible runs inside the image now: Packer installs ansible-core in QEMU, runs the playbook with ansible-local and removes Ansible again.

DietPi on a Pi Zero 2 W

The Pi Zero 2 W has only 512MB of memory, so I tried DietPi. I first used the 32-bit image, but Hermes Agent does not install on 32-bit. The Pi has a 64-bit CPU, so I switched to the current 64-bit one: DietPi_RPi234-ARMv8-Trixie.

DietPi applies most settings from /boot/dietpi.txt on the first boot, so the playbook only fills that file and /boot/dietpi-wifi.txt. The SSH key is added to root and dietpi.

Hermes Agent is a bit much for the Pi Zero 2 W. For encrypted Matrix it compiles python-olm on the Pi, which the Zero 2 W is absolutely not suited for. With its web dashboard running the Pi swaps and answers very slowly, so I turned the dashboard off. So currently Hermes is only answering via Matrix.

On my other old Pi Zeros I have not switched away from 32-bit RaspiOS yet, but I will when they need an update. Everything is in the same repository as before.

Pushing to Forgejo from DietPi

DietPi uses Dropbear instead of OpenSSH, so there is no ssh and no ssh-keygen on the Pi. The client is dbclient and keys are generated with dropbearkey:

dropbearkey -t ed25519 -f ~/.ssh/id_dropbear
dropbearkey -y -f ~/.ssh/id_dropbear | grep '^ssh-'

dropbearkey writes the private key in its own format and only prints the public key, there is no .pub file. I added the public key as a deploy key with write access to the repository in Forgejo.

dbclient does not read ~/.ssh/config and git calls ssh by default, which fails with cannot run ssh: No such file or directory. Setting core.sshCommand in the repository fixes that:

git remote set-url origin ssh://git@forgejo.example.ts.net/user/repository.git
git config core.sshCommand "dbclient -i ~/.ssh/id_dropbear"

The cause

ARTE's files start with a long series of STYLE blocks before the first cue:

STYLE
::cue(.black) { color: black; }

ffmpeg does not skip STYLE blocks. It hits one, finds no --> timing line, and stops parsing. The track exists and is empty. mpv uses the same WebVTT demuxer, so it lists a track that never shows anything.

The fix

The obvious approach is to let yt-dlp convert the subtitles on download and get SRT files that mpv has no trouble with. But --convert-subs srt does not help here, because yt-dlp hands the file to ffmpeg and ffmpeg chokes on the same STYLE blocks.

So drop everything before the first cue and put a clean header back:

{ printf 'WEBVTT\n\n'; sed -n '/-->/,$p' "$f"; } > "$f.tmp" && mv "$f.tmp" "$f"

This is done in place because mpv autoloads subtitles by filename.

Doing it automatically

A wrapper gets the final filename from yt-dlp with --print-to-file after_move:filepath and fixes every .vtt next to the video:

#!/usr/bin/env bash
p=$(mktemp)
yt-dlp -f 'bv[height<=720]+mergeall[vcodec=none][format_id!*=clear_voices]' --audio-multistreams --sub-langs all --write-subs \
       --print-to-file after_move:filepath "$p" "$@"
while read -r v; do
  for s in "${v%.*}".*.vtt; do
    { printf 'WEBVTT\n\n'; sed -n '/-->/,$p' "$s"; } > "$s.tmp" && mv "$s.tmp" "$s"
  done
done < "$p"
rm -f "$p"

format_id!*=clear_voices skips ARTE's clear voices tracks. That is an accessibility option: a separate mix of the soundtrack with the dialogue emphasized and music, ambient noise and sound effects turned down, so speech is easier to understand.

The script not only downloads all subtitles but also all audio tracks. A lot of ARTE films have the original audio and a German or French dub if available. This leaves it to me to decide if I am fine with the (dubbed) German version, or prefer to watch the original audio track and read subtitles.

A second systemd service

The second instance is a copy of tailscaled.service from the Arch Linux package, with its own state file, socket, TUN device and UDP port. I saved it as /etc/systemd/system/tailscaled-work.service:

[Unit]
Description=Tailscale node agent (work tailnet)
Wants=network-pre.target
# systemd-networkd here; with NetworkManager use NetworkManager.service instead
After=network-pre.target systemd-networkd.service systemd-resolved.service tailscaled.service

[Service]
ExecStart=/usr/sbin/tailscaled --state=/var/lib/tailscale-work/tailscaled.state --socket=/run/tailscale-work/tailscaled.sock --tun=tailscale1 --port=41642
ExecStopPost=-/usr/bin/systemctl --no-block try-restart tailscaled.service

Restart=on-failure

RuntimeDirectory=tailscale-work
RuntimeDirectoryMode=0755
StateDirectory=tailscale-work
StateDirectoryMode=0700
CacheDirectory=tailscale-work
CacheDirectoryMode=0750
Type=notify

[Install]
WantedBy=multi-user.target

The original unit has ExecStopPost=/usr/sbin/tailscaled --cleanup, which I left out. Both instances share the same policy routing rules and routing table 52, and the cleanup of one instance should not remove what the other one is using.

This does not help completely, though. Stopping tailscaled-work still deletes the shared rules -- ip rule afterwards only lists local, main and default:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Without the lookup 52 rule the private tailnet is unreachable too. Starting the work instance again adds the rules back, and restarting the private one with sudo systemctl restart tailscaled should do the same. The ExecStopPost line in the unit above restarts the private instance whenever the work instance stops, so the rules are back immediately.

Enable and start the new service:

sudo systemctl daemon-reload
sudo systemctl enable --now tailscaled-work
systemctl status tailscaled-work

Logging in

All tailscale commands for the second instance need the socket, so an alias helps:

alias tailscale-work='tailscale --socket=/run/tailscale-work/tailscaled.sock'

Logging in creates a new node in the work tailnet -- the old work profile in the main instance is not reused:

sudo tailscale --socket=/run/tailscale-work/tailscaled.sock up \
  --accept-routes --accept-dns=false --shields-up --netfilter-mode=off --operator=$USER

--accept-routes

Gets the AWS subnet routes.

--accept-dns=false

Keeps MagicDNS and the DNS settings with the private tailnet.

--shields-up

Blocks all incoming connections from the work tailnet.

--netfilter-mode=off

Leaves the iptables chains to the private instance; both would write into the same ts-input chain, and its rule drops 100.x traffic that does not come in on its own interface.

--operator

Allows my user to run tailscale-work status without sudo.

The old work node still has the hostname, so the new machine gets the old name with -1 appended. I deleted the old node in the admin console and renamed the new one to drop the -1.

Result

Both instances are logged in and each has its own interface:

tailscale status --peers=false
tailscale-work status --peers=false
ip -br addr show | grep tailscale

The subnet routes from the work tailnet land in table 52 on tailscale1, next to the peers of the private tailnet on tailscale0:

$ ip route show table 52 | grep -v '^100\.'
10.x.0.0/16 dev tailscale1
172.x.0.0/16 dev tailscale1

ip route get with an address from one of these subnets shows dev tailscale1 table 52. A host at work that is only reachable via the tailnet loads in the browser, and my private peers still answer tailscale ping. It is only this simple because the two tailnets do not share any 100.x addresses and my private tailnet has no routes into AWS.

Finally, remove the old work profile from the main instance (tailscale switch --list shows the names):

tailscale switch work
tailscale logout
tailscale switch <private-profile>

No more tailscale switch when a dev setup needs AWS access for work.