Handoff: UrlResolver reconnect-gossip flake fixed (non-blocking dispatcher, #763 GREEN) — merge chain #763 → #757 → #700 ready to execute

Written: 2026-07-12 17:10 UTC. Thread origin: 2026-07-06 (this continues 2026-07-06-flake-elimination-buildtest-oom-relay-bytearray-and-700-netlab-block.md; the surviving goal from that handoff is merging UrlResolver #700).

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at 2026-07-12 17:10 UTC. PRs merge, merge queues shuffle, CI re-runs, buildtest restarts. A fresh query always wins over this document. Re-verify:

• PRs: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
• #757 merge-queue slot: gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){pullRequest(number:757){mergeQueueEntry{position state}}}}'
• buildtest health: curl -m20 https://buildtest.kotlin.build/api/runs?limit=15 (was mid-recovery from an outage at write time — see §4).

1. Mission

Original standing request (this session): "drive [the 2026-07-06 flake-elimination handoff] to completion" — whose one surviving concrete goal is getting UrlResolver #700 merged (a trivial PR that only ports a regression test for the already-fixed gossip bug #656). The user has approved merging #757 and #700 ("approved to merge, proceed").

The chain to get there expanded as blockers surfaced:

• #700 is gated behind UrlResolver #757 (a resurrection-race fix) landing first.
• #757 kept getting evicted from the merge queue by a distinct flaky test (not its own code), which required a new fix: UrlResolver #763.

2. Current state in one line

#763 is GREEN and merge-ready (both checks pass, CLEAN). It is a NEW PR, so per the standing rule it needs explicit user merge approval (the prior "approved to merge" covered #757/#700, not #763). Merge order intended: #763 → #757 → #700 → close stale issues.

3. Relevant PRs / refs (write-time state)

| PR | Node id | State @ write time | What it is | |----|---------|--------------------|------------| | UrlResolver #763 | PR_kwDOQjgD7s7wutXy | OPEN, CLEAN, both checks ✅ GREEN, not merged. Branch fix/reconnect-gossip-flake, head a80732e0 ("Publish initial gossip on dedicated dispatcher") | The durable flake fix. Needs user merge approval. | | UrlResolver #757 | PR_kwDOQjgD7s7wnopz | OPEN, CLEAN, head checks green, NOT in queue (evicted during the buildtest outage), not merged | Resurrection-race fix (announce/withdraw bookkeeping lock). User already approved merging. | | UrlResolver #700 | PR_kwDOQjgD7s7uQ1KQ | OPEN, CLEAN, not merged | The goal — ports a regression test. User already approved merging. | | DigitalOceanDropletServiceServer #85 | — | MERGED + DEPLOYED earlier this session | Fixed a DO rate-limit outage (renewDroplet's unnecessary per-renewal GET). See [[do_ratelimit_renewdroplet_unnecessary_get_provisioning_outage]]. |

Working copy: /code/UrlResolverFlake (branch fix/reconnect-gossip-flake @ a80732e0, tree clean, pushed — no unsaved work).

4. What was found and done (the chain, with evidence)

1. #757 kept getting evicted from the merge queue (4–5 times). Verified each eviction: #757's OWN head checks (bld-build, kotlin.build (remote)) stayed green every time — so it was the merge-queue combined build (PR squashed onto main) failing, not #757's code.

2. Diagnosed the evicting test. buildtest truncates /api/test-results?id=<run> at 100 rows/page but supports &page=N. Paging the #757 combined-build run (f3372166) found exactly one failing test of 1087: testReconnectFactoryWaitsForGossipWhenNoAlternativePeersAvailable (file tests/testReconnectFactoryWaitsForGossipWhenNoAlternativePeersAvailable.kts). A second UrlResolver run failed the same test. (Other evictions hit sibling gossip-timing flakes — stressTestRegistrationAfterWithdrawalRaceUnderContention — and a known unrelated relay flake testBytecodeFetchAggregatesRelayFailureDetails.)

3. Root-caused the flake (verified via a deterministic reproducer). When a provider calls registerGlobalService, its initial "I exist" gossip announcement was launched via launchSurfaceErrors { registration.broadcastSafely(...) } onto the resolver's saturable general background coroutine pool. The reproducer deliberately occupies that pool (16 held tasks); the announcement is then starved and never published, so a reconnecting client can't discover the just-registered replacement provider. codex gpt-5.6-sol built an amplified deterministic reproducer that failed 10/10 before fix, passed 10/10 after on a 16-core host.

4. First fix attempt (REJECTED by user). codex's initial fix made the initial broadcastSafely(...) run synchronously on the caller's thread (commit 66205fbf). registerServiceInternal is a non-suspend fun, so this genuinely blocks a thread on gossip/relay network I/O during registration. An adversarial review (codex) found two MAJOR issues (orphaned local registration on broadcast failure; registration could block ~12s on a synchronous relay reconnect). The user reviewed it and rejected the synchronous approach outright — "doing anything synchronously which requires network io sounds very suspicious." Confirmed in code that it blocks a real thread.

5. Redesign (APPROVED by user, now landed as a80732e0). The user chose the non-blocking design. The flake's root cause is executor starvation, not the async-ness. The fix: publish the initial announcement asynchronously on a DEDICATED single-thread dispatcher (initialPublicationExecutor / initialPublicationDispatcher, a private CoroutineScope(initialPublicationDispatcher + SupervisorJob() + handler)) that unrelated saturated work cannot starve. Registration performs no synchronous network I/O on its caller's thread. Verified in the diff: the dedicated executor is shut down on close (initialPublicationScope.cancel(), initialPublicationExecutor.shutdown() / shutdownNow() — no thread leak), and the #757 lock-ordered announcement-vs-withdrawal timestamp capture is preserved (so the async publish can't reopen the resurrection race). The old synchronous ThreadLocal initialServicePublication machinery was fully removed (grep count 0).

6. #763 went GREEN. After a ContainerNursery route outage (see §7) delayed it, bld-build ✅ and kotlin.build (remote) ✅ both COMPLETED SUCCESS on a80732e0; mergeStateStatus CLEAN. The full remote suite — which runs the reconnect test on the standard kompile environment — passed, confirming the non-blocking fix on the authoritative environment.

5. Next steps (ordered — re-verify §RE-VERIFY first)

1. Re-verify all three PRs (commands in the banner) and buildtest health.
2. Get explicit user approval to merge #763 (it's a new PR; prior approval didn't cover it). If green + approved, enqueue it (PlanRepo-style GraphQL enqueuePullRequest on PR_kwDOQjgD7s7wutXy, or the repo's merge-queue procedure). Do NOT merge without that approval.
3. Once #763 is MERGED: re-enqueue #757 (node PR_kwDOQjgD7s7wnopz) if OPEN/CLEAN/not-in-queue (warm the webhook: curl github-webhooks.wasmserver.com 3×). With #763's gossip fix on main, #757's combined build should stop hitting the reconnect flake — but it may still hit the known unrelated relay flake testBytecodeFetchAggregatesRelayFailureDetails; if so, re-request that check once (do not treat as a real failure). Check whether #757 needs a rebase on #763 — both edit UrlResolver.kt (near the registration/broadcast path); a merge-queue combined build reconciles them, but a textual conflict would need a rebase.
4. Once #757 is MERGED: re-enqueue #700 (node PR_kwDOQjgD7s7uQ1KQ) if OPEN/CLEAN/not-in-queue.
5. Once #700 is MERGED — cleanup:
   • gh pr close 629 --repo CodexCoder21Organization/UrlResolver --comment 'Superseded: fix landed in #656/#690; regression test ported by #700 (merged).'
   • Close issues #621, #622, #623: gh issue close <n> --repo CodexCoder21Organization/UrlResolver --comment 'Fixed by #656 (gossip retry) with the resurrection race additionally closed by #757; regression test ported by #700.'
6. Deferred (do-last, surface to user separately): the fleet CN protocol deploy (~90 container restarts) from the original thread — NOT started; high blast radius.

6. Session automation still live (session-scoped — will die with this Claude session)

• A CHAIN-DRIVER cron (fires :06/:26/:46) that drives #757 → #700 → cleanup, and a /status-report cron (:17/:37/:57). These are session-only (in-memory) and will not survive this session — a new session/agent picking this up must re-establish its own driving loop or drive manually per §5. Note: the chain-driver script does NOT know about #763 — it only drives #757/#700; #763's merge is a manual/user-approved step in front of it.

7. Reusable / operational knowledge (so you don't rediscover it)

• The buildtest/kotlin.build route outage (2026-07-12 ~15:20–16:05 UTC). Symptom: buildtest.kotlin.build and kotlin.build (both DNS → 198.199.106.165) returned HTTP 000 / 15–20s connect timeouts, stalling every kotlin.build (remote) check and the merge queue. Triage that isolated it: the CN control plane https://api.nursery.wasmserver.com/ (200 in 0.35s) and https://kotlin.directory/ (200 in 0.77s) stayed healthy, and GitHub was fine (0.5s) — so it was a scoped wedged/dead CN route/container, not a host outage. It self-recovered (~16:05, 200 but slow 5–8s). If it recurs: the CN control plane is reachable, so use the ContainerNursery CLI to inspect/recover the buildtest+kotlin.build container (diagnose root cause before any restart) — see the CN outage memories and the investigate-broken-service skill. This sandbox has no prod SSH / no hardware-fabric config — only the CN control-plane API.
• Degraded local build cache = pathologically slow kompile verifies. Killing a mid-flight cold compile (done once here) can leave ~/.aibuildcaches / bldbinary in a state where every subsequent local scripts/test.bash verify takes 50–90 min. Do NOT rm the caches to "fix" it — a full cold recompile of UrlResolver is also ~1 h+. Pattern that worked: let codex implement + commit, then push and rely on #763's own kotlin.build (remote) CI (runs the reconnect test on the standard env) for the authoritative verification rather than waiting on the slow local build.
• codex gpt-5.6-sol delegation pattern used all session: codex exec --dangerously-bypass-approvals-and-sandbox -m gpt-5.6-sol --skip-git-repo-check -C <repo> "$(cat brief.md)" < /dev/null > log 2>&1 run in background. ALWAYS < /dev/null (else it hangs on stdin) and -C <repo>. codex reliably does the engineering + commit but often exits without pushing/opening the PR — relaunch a narrow mechanical codex to push + gh pr create/gh pr edit (see [[codex_owns_build_test_merge_even_after_stall]]).
• gh pr edit --body fails on this org (projectCards permission); the mechanical codex used the REST fallback gh api -X PATCH repos/<owner>/<repo>/pulls/<n> -f body='...'.
• The flake test itself (testReconnectFactoryWaitsForGossipWhenNoAlternativePeersAvailable): occupies provider2's bounded worker pool with held tasks to force the starvation window; asserts a reconnecting client discovers a late-registered replacement provider. It is the regression guard — keep it.
• buildtest per-test failure retrieval: /api/test-results?id=<runId>&page=N (100/page, totalPages in the JSON). The <runId> is in the kotlin.build (remote) check's detailsUrl (https://buildtest.kotlin.build/run?id=<runId>).