forgejo-runner: add fallback nameservers so a SERVFAIL doesn't kill a job #15

Merged

sophie merged 1 commit from help-me-figure-out-why-the-forge into main

2026-10-17 01:09:26 +11:00

myagent commented

2026-10-17 01:09:02 +11:00

Collaborator

Why

Actions jobs on the scout runner intermittently die at their first step with
Could not resolve host: data.forgejo.org (Timeout while contacting DNS servers) — 4 of ~50 recent jobs (simpleplayer-svc runs 13/25/30, simpleplayer-helper run 1), and this Forgejo has no re-run via API.

Diagnosis

The message is not a timeout: the lookup took 1.4s and the retry 0.5s. It is curl's text for EAI_AGAIN, which the Alpine runner's musl returns the instant any nameserver answers SERVFAIL.
The pod has exactly one nameserver (CoreDNS), whose only . upstream is tailscaled's MagicDNS forwarder (DoH → NextDNS), because the node's resolv.conf is tailscaled's. No redundancy anywhere in the chain.
CoreDNS metrics show 23 SERVFAIL answers relayed from that upstream (and 4,443 failed health checks against it); CoreDNS passes those through silently and caches them for 5s, so the runner's own retry fails too.
Ruled out with evidence: CoreDNS errors (zero [ERROR] lines across 3 of the 4 failures), dind network churn in the shared netns (770/770 lookups clean under load), the ts.net search-domain candidate, the IPv6 upstream, and any other hostname.

Full write-up in docs/forgejo-runner-dns.md, including follow-ups (node resolv.conf fight, local actions mirror).

Change

Set the chart's dnsConfig to append 1.1.1.1 and 9.9.9.9 after CoreDNS, with ndots:1. musl queries all three in parallel and ignores a SERVFAIL while another server answers.

Verified on the cluster with an Alpine pod against a nameserver that always answers SERVFAIL: alone → 0/5 lookups succeed; with a second server → 20/20.

The pod template changes, so the Deployment rolls (Recreate). Side effect: NextDNS filtering no longer applies to the runner's lookups.

Verify after sync:

kubectl -n forgejo-runner exec deploy/forgejo-runner -c runner -- cat /etc/resolv.conf

🤖 Generated with Claude Code

## Why Actions jobs on the scout runner intermittently die at their first step with `Could not resolve host: data.forgejo.org (Timeout while contacting DNS servers)` — 4 of ~50 recent jobs (simpleplayer-svc runs 13/25/30, simpleplayer-helper run 1), and this Forgejo has no re-run via API. ## Diagnosis - The message is **not a timeout**: the lookup took 1.4s and the retry 0.5s. It is curl's text for `EAI_AGAIN`, which the Alpine runner's musl returns the instant any nameserver answers **SERVFAIL**. - The pod has exactly one nameserver (CoreDNS), whose only `.` upstream is tailscaled's MagicDNS forwarder (DoH → NextDNS), because the node's resolv.conf is tailscaled's. No redundancy anywhere in the chain. - CoreDNS metrics show 23 SERVFAIL answers relayed from that upstream (and 4,443 failed health checks against it); CoreDNS passes those through silently and caches them for 5s, so the runner's own retry fails too. - Ruled out with evidence: CoreDNS errors (zero `[ERROR]` lines across 3 of the 4 failures), dind network churn in the shared netns (770/770 lookups clean under load), the ts.net search-domain candidate, the IPv6 upstream, and any other hostname. Full write-up in `docs/forgejo-runner-dns.md`, including follow-ups (node resolv.conf fight, local actions mirror). ## Change Set the chart's `dnsConfig` to append `1.1.1.1` and `9.9.9.9` after CoreDNS, with `ndots:1`. musl queries all three in parallel and ignores a SERVFAIL while another server answers. Verified on the cluster with an Alpine pod against a nameserver that always answers SERVFAIL: alone → 0/5 lookups succeed; with a second server → 20/20. The pod template changes, so the Deployment rolls (Recreate). Side effect: NextDNS filtering no longer applies to the runner's lookups. Verify after sync: ``` kubectl -n forgejo-runner exec deploy/forgejo-runner -c runner -- cat /etc/resolv.conf ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code)

myagent added 1 commit

2026-10-17 01:09:02 +11:00

forgejo-runner: add fallback nameservers so a SERVFAIL doesn't kill a job 4c5ca736fa

Jobs intermittently fail at their first step with "Could not resolve host:
data.forgejo.org (Timeout while contacting DNS servers)" — 4 of ~50 recent
jobs. The message is musl's EAI_AGAIN, returned the instant any nameserver
answers SERVFAIL. The pod has one nameserver (CoreDNS) whose only upstream
is tailscaled's DoH forwarder to NextDNS; CoreDNS relays its occasional
SERVFAIL answers silently and caches them for 5s, so the runner's retry
fails too. Nothing else was at fault: CoreDNS logged no errors, dind's
network churn reproduced nothing, and no other name ever failed.

Append 1.1.1.1 and 9.9.9.9 via the chart's dnsConfig (CoreDNS stays first).
musl queries all three in parallel and ignores a SERVFAIL while another
server answers — verified against a SERVFAIL-only nameserver (0/5 alone,
20/20 with a second). ndots:1 keeps the public resolvers from racing
CoreDNS on search-suffixed cluster names; the runner only resolves public
names. Full diagnosis and follow-ups in docs/forgejo-runner-dns.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>