scout: stop the dhclient/tailscaled resolv.conf fight and give k3s a stable resolv.conf #16
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up from #15 /
docs/forgejo-runner-dns.md. Not urgent; the runner-side fix in #15 already covers the job failures.Problem
On scout, two
dhclients (enp88s0, enp2s0f0) rewrite/etc/resolv.confon every DHCP renewal (~25 min) and tailscaled rewrites it back: 423 "trample" cycles and 395health(warnable=resolv-conf-overwritten)warnings in 7 days (journalctl -u tailscaled | grep -i trampled). At the moment of each cycle the file is briefly empty (current contents: "").Consequence for the cluster: whatever the file says when a pod sandbox is created is what kubelet bakes in. Today k3s's CoreDNS has exactly one
.upstream — tailscaled's MagicDNS forwarder100.100.100.100, which resolves over DoH to NextDNS — so every pod's external DNS has no redundancy. CoreDNS metrics since 2026-03-29: 4,443 failed health checks against that upstream, 16,266 queries served while it was marked down, 15,597 SERVFAILs to clients.Proposed
supersede domain-name-servers/make_resolv_conf(){ :; }hook, or let tailscaled own it via--accept-dns+ a resolvconf-aware setup). Pick whichever keeps the host's MagicDNS working./etc/rancher/k3s/config.yaml→resolv-conf: /etc/rancher/k3s/resolv.conf(k3s v1.33.6 has the flag), containing e.g. the NextDNS anycast addresses DHCP already hands out (45.90.28.145,45.90.30.145) plussearch tuatara-puffin.ts.netso pod behaviour is unchanged. Restart k3s, thenkubectl -n kube-system rollout restart deploy/coredns./etc/rancher/k3s/config.yamlis hand-managed today; consider bringing it (or at least the resolv.conf) underansible/.Caveat
CoreDNS does not fail over on an upstream SERVFAIL answer (only on timeouts/unreachable), so this helps outages of the tailscaled path but is not a substitute for the runner's own fallback nameservers from #15.
Verify
kubectl -n kube-system port-forward deploy/coredns 9153thencurl -s localhost:9153/metrics | grep -E 'proxy_healthcheck|SERVFAIL'— theto=label should show the new upstreams and the health-check failure counter should stop growing.