The tunnel that only came back on a new port

Problem

This is a follow-up to WireGuard site-to-site behind a FRITZ!Box. That article closes with the advice to set PersistentKeepalive on the dialling side, because an idle UDP flow through a consumer router (a FRITZ!Box 6690 Cable) is a flow that may quietly disappear. It turns out that a keepalive protects a running tunnel, not one whose peer goes away for a few minutes.

The same two sites as in the first article. This time only one thing matters: the single UDP flow that the Raspberry Pi builds through the FRITZ!Box to the gateway VM — and the state the FRITZ!Box keeps for it.

We patched the main server and rebooted it on 14 November 2026, a normal maintenance window — the latest Ubuntu and Debian kernel patches are the kind of update that needs one (I think I am spending too much time patching systems lately, crazy time). Everything came back after some time, since I patched the Proxmox host as well as the gateway VM, and a host restart takes a while with all the hardware reinitialization. Everything except the site-to-site tunnel. The Remote Location lost the networks of the Main Location: no backups to the Proxmox Backup Server, no name resolution across the link, the laptop test from the first article failing again.

The Nagios server noticed the missing tunnel quickly. That is where the search started; ten minutes later the tunnel was up again.

Analysis

Working on the Pi at all was only possible because the Remote Location has a second way in from the outside, over a port forwarded on the FRITZ!Box straight to the Raspberry Pi. Without it there would have been no way to reach the Pi while the tunnel was down, and the analysis would have started with a drive to the Remote Location.

wg on the Raspberry Pi showed the shape of the problem in one screen: no handshake, bytes going out, nothing coming in. The retry every five seconds is WireGuard’s handshake interval, not the keepalive — the keepalive only runs once a session exists.

Remote Location:

root@vpn-remote:/var/log# wg show
interface: wg0
  public key: <public key of the Raspberry Pi>
  private key: (hidden)
  listening port: 51820

peer: <public key of the gateway VM>
  endpoint: 185.6.68.20:51820
  allowed ips: 192.168.1.0/24, 192.168.253.0/24, 192.168.254.0/24, 10.254.0.1/32
  transfer: 0 B received, 19.37 KiB sent
  persistent keepalive: every 25 seconds

Main Location:

root@gateway:~# wg show
interface: wg0
  public key: <public key of the gateway VM>
  private key: (hidden)
  listening port: 51820

peer: <public key of the Raspberry Pi>
  endpoint: 82.115.124.36:51820
  allowed ips: 10.254.0.2/32, 192.168.178.0/24
  transfer: 12.57 KiB received, 15.62 KiB sent
  persistent keepalive: every 25 seconds
...

I learned that I should have checked for the “latest handshake:” line. It is missing in both outputs, and I read past it at first. Both sides show the same picture from their own end: the Pi sends and gets nothing back, the gateway receives and answers with 15.62 KiB. So the gateway was answering all along.

Then tcpdump on the Pi. The handshake initiations left the Remote Location every five seconds, and nothing came back:

root@vpn-remote:/var/log# tcpdump -ni eth0 udp port 51820
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:21:32.617964 IP 192.168.178.3.51820 > 185.6.68.20.51820: UDP, length 148
11:21:37.737952 IP 192.168.178.3.51820 > 185.6.68.20.51820: UDP, length 148
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

Connection tracking on the gateway still had the flow, with 115 seconds left on its timer and marked [ASSURED], which the kernel only sets once it has seen packets in both directions:

root@gateway:~# conntrack -L -p udp | grep 51820
udp      17 115 src=82.115.124.36 dst=185.6.68.20 sport=51820 dport=51820 src=185.6.68.20 dst=82.115.124.36 sport=51820 dport=51820 [ASSURED] mark=0 use=1
conntrack v1.4.8 (conntrack-tools): 169 flow entries have been shown.

Two different sources, one conclusion: the server side was healthy and replying — the packets died on the way back.

That narrows the path down to exactly one stateful device we do not control from the inside: the FRITZ!Box at the Remote Location, which source-NATs the Pi’s UDP flow to the internet.

The test that decided it: we changed the listening port on the Pi, the side that dials.

root@vpn-remote:/var/log# wg set wg0 listen-port 51821
root@vpn-remote:/var/log# tcpdump -ni eth0 udp port 51821
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:21:51.725672 IP 185.6.68.20.51820 > 192.168.178.3.51821: UDP, length 128
11:21:51.726179 IP 192.168.178.3.51821 > 185.6.68.20.51820: UDP, length 128
11:21:51.727640 IP 185.6.68.20.51820 > 192.168.178.3.51821: UDP, length 96
11:21:51.727683 IP 192.168.178.3.51821 > 185.6.68.20.51820: UDP, length 96
11:21:51.748370 IP 185.6.68.20.51820 > 192.168.178.3.51821: UDP, length 96
11:21:51.748547 IP 185.6.68.20.51820 > 192.168.178.3.51821: UDP, length 1452
11:21:51.748548 IP 185.6.68.20.51820 > 192.168.178.3.51821: UDP, length 368
11:21:51.748625 IP 192.168.178.3.51821 > 185.6.68.20.51820: UDP, length 96
...

The handshake completed in the first second: the last fruitless attempt is logged at 11:21:37, the working one at 11:21:51. Fourteen seconds between “nothing moves at all” and a running tunnel. Same keys, same configuration, same firewall rules, same endpoint — only a different source port. The FRITZ!Box keeps the source port unchanged (no PAT), so the only thing it rewrites is the address. What it keeps on top of that is state.

Working hypothesis

The FRITZ!Box no longer delivered the return packets of that one UDP flow. While the main server was rebooting, nothing came back for several minutes; whatever state the box kept for the flow ended up unusable, and it stayed unusable even after the server was answering again. A new source port means a new connection tracking entry, and the new entry works.

We cannot prove it. A FRITZ!Box offers no shell, no view of its connection tracking table and no counter for return packets it drops, so the hypothesis rests on the symptom and on the fact that a fresh source port cures it instantly.

There is no second candidate. The FRITZ!Box at the Remote Location holds a real public IPv4 address, so there is no carrier-grade NAT behind it that could have lost the flow instead. Everything between the two WireGuard interfaces is routing; the one place that keeps per-flow state is that box.

One observation supports the theory from the other side. The second way in that we used throughout is a static port forward in that same FRITZ!Box — it has nothing to do with the tunnel, which the box only ever sees as a flow built from the inside out. That static rule kept working through the reboot and through the entire outage. It survived what the dynamically created entry did not, which is at least an argument for putting the tunnel on a static rule as well.

Fix

There is no fix yet, only a workaround that works reliably: give the tunnel a new source port, with the wg set from the analysis above.

wg set changes the running interface only. wg-quick reads ListenPort from /etc/wireguard/wg0.conf, so the config file needs the same value if the change is supposed to survive a restart of the interface or a reboot of the Pi.

Two things we did not do. Restarting the Pi is the obvious reflex, and it has cured this once before without any analysis at all — that reboot doubled as a good moment for the patches that were waiting anyway. Today the same restart did not help. The likely reason is timing: the entry in the FRITZ!Box only disappears when nothing is sent for long enough, and a Raspberry Pi on an NVMe is back up in well under a minute. The earlier outage apparently kept the tunnel down longer than whatever timer the box uses. Rebooting the FRITZ!Box would almost certainly clear the state, but it is also our only way in. If it did not come back, the next step would be the car.

A word on that reflex. As a UNIX and Linux person I normally do not like “just reboot it first” — it throws away the very state you need to understand the problem, and here it would have thrown away the hanging flow together with the evidence. But it has worked for me a few times lately, without any analysis at all. So apparently one can learn something from Windows ;-).

The plan for the next maintenance window is therefore to avoid the stale connection tracking entry instead of repairing it: stop the tunnel on the Pi before the main server goes down, and bring it up afterwards — with a changed port if it still refuses.

A self-healing timer on the Pi would be the only candidate that works without me: check wg show wg0 latest-handshakes, rotate the listening port when the last handshake is too old. I have thought about it and do not trust it yet. A script that changes the port of a working tunnel because of a badly chosen threshold is a new source of outages, and this is the link the backups run over.

Turning the direction around would be the other way to reach a static rule. The Remote Location has a MyFRITZ name, so the gateway could be the one that dials, and the incoming flow would meet a port forward instead of dynamic state. I do not want a DNS name in a VPN config, though. WireGuard resolves Endpoint once, when the interface comes up — a name that fails to resolve at boot, or changes later, needs yet another timer to re-resolve it, which is the same kind of moving part I just turned down. And there is a reason “it was DNS” is a T-shirt — Jeff Geerling sells a good one, the story behind it and the shirt. DNS debugging is wild.

So for now the manual fix stays. The tunnel has been up since the beginning of the year, and this is the first time it needed more than a restart. The outage cost ten minutes, the repair is one command, and every permanent candidate either adds a moving part or a dependency I do not want on this link. A permanent fix becomes worth it when the problem shows up more often than this.

If it does, the first thing to try is a fixed listening port on the Pi plus a UDP port forward for it in the FRITZ!Box. That is also the experiment that answers the open question: does FRITZ!OS use a static rule for the return path of a flow that was built from the inside out? The tcpdump on the gateway VM shows it, in the source port of the arriving packets.

Takeaway

A keepalive keeps a living flow alive. It does nothing for a flow whose far end disappears for the length of a reboot, and that is exactly when a consumer router is most likely to lose track of it.

When the far end answers and the near end never hears the answer, stop debugging the endpoints. Both of ours were fine the whole time; everything we needed to see was in one tcpdump on the near end and in the connection tracking table on the far end.

Give every remote device a way in that does not depend on the thing you may have to debug. Ours is a forwarded port on the FRITZ!Box, and it is the only reason I could run this analysis from a keyboard instead of hopping into my car.

Changing the source port is a diagnosis and a cure in one move. If a new port connects instantly, the old flow was the problem — not the keys, not the firewall, not WireGuard.

And a box you cannot query turns an analysis into a hypothesis. That is the real price of a consumer router in an otherwise observable path.

Next time the order is clear. Read the latest handshake line first — if it is missing, there is no session and everything else is noise. Then run tcpdump on the Pi, because the Pi is the side that has to build the connection, and it is therefore the side that shows whether anything is coming back.

One thing is still missing, and this outage made it obvious: a way to force a hard reboot, a power cycle, on a device that no longer answers at all. At the Main Location I am building that right now with Home Assistant and switchable Zigbee sockets, for exactly this reason. At the Remote Location the first step is the devices behind the FRITZ!Box — the VPN Raspberry Pi, the printer, whatever else sits there — and the way in we already have covers those. The FRITZ!Box itself is the harder case, because a socket that hangs off the very router you want to power-cycle is no help; that one would need an independent path, LoRaWAN perhaps. Not decided yet, and it will be its own article.