← ArchiveArea

UrlResolver merge campaign: three race fixes code-complete (873/882/866); landing paused pending netlab-manager redeploy from merged main

PRs 873, 882, and 866 are code-complete, pushed, and verified against fail-first reproducers, but six queue/validation attempts were consumed by estate failures (netlab-hosted drops persistent RPC connections under a single full suite's load) with zero fix-attributable failures. Gating step: redeploy netlab-manager from merged main (user-authorized), then land 873 -> 882 -> 866 solo via GraphQL enqueue, then closures (737/870/804), held PRs (823/838/848/872/850), and the orchestrator final reviews.

hf-2026-07-27-urlresolver-merge-campaign-leak-fix-deployed-seven-prs-escortable-stopped-by-user
Completed Jul 29, 2026 · 14:34 UTC This handoff is read-only in ArchiveArea.

Handoff document

Markdown

UrlResolver merge campaign: three race fixes code-complete and verified; landing paused because the shared netlab-hosted service drops connections under a single full suite's load

Written 2026-07-28 ~22:45 UTC. RE-VERIFY EVERYTHING BELOW BEFORE ACTING — PR states, merge-queue occupancy, and netlab health drift by the hour. Authoritative re-checks: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup, the merge-queue GraphQL query, and netlab-cli health (a 30 s timeout or closed while requests were still pending errors mean the estate is degraded again).

Mission summary

Land the remaining UrlResolver merge-campaign PRs. Three production races were root-caused and fixed this session (all codex-authored, statistically verified against fail-first reproducers):

  • https://github.com/CodexCoder21Organization/UrlResolver/pull/873 — every close-boundary decision plus UrlProtocol2.close() now executes atomically under one admissionAccountingLock, so a dial admitted during shutdown can no longer leak past the close. Reproducers went 8/10 and 10/10 failing → 0/20 and 0/20 after the fix.
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/882 — tier probes must complete before that tier's registration-attempt budget opens, and same-tier relay ties break deterministically (fewest attempts, then stable candidate order). Fixes the stressTestHealthySameTierRelayWinsUnderContention flake and is expected to resolve the lower-tier-selection challenge (see Next steps item 5).
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/866 — onPeerAnnouncement dedups semantic services once per announcement (was ~2,048 locked transactions each), diagnostics removal is atomic with creation, and the retained-heap churn test now completes within budget (one-time setup moved out of the measured window).

Verified current state (write-time snapshot)

  • All three PR heads are pushed and current: 873 at ea34056614a9581fefcb0f6bd11f5a0d61bd2573 (OPEN, CLEAN, head checks green), 882 at 793fa6d871749b5127dd0175d057be69d9b286ec (OPEN, head suite re-dispatched), 866 at 3dbabc47deff4fcc65c8bc237dafda591e15d188 (OPEN, freshly pushed — head CI newly triggered). No unpushed commits, stashes, or local-only work remain anywhere; the lane workspaces are clean.
  • Why nothing has landed: six queue/validation attempts across 873/882 were consumed by estate failures with zero fix-attributable failures: two netlab connection-drop kills, one netlab meltdown window (nine failures in one run; even netlab-cli health timed out), two distinct pre-existing test flakes (challenges filed, links below), and one speculative queue-stack dissolution. The decisive observation: the sixth queue attempt was enqueued only after a verified-healthy netlab-cli health round-trip (212 ms) with an otherwise empty queue, and still lost testNetlabNatToNatRelayOutageRecovery to Persistent RPC connection to service 'netlab-hosted' was closed while requests were still pending at 451/1,214 tests. A single full resolver suite's own load is enough to make the currently-deployed netlab manager drop connections; there is no quiet-window workaround.
  • Manager forensics: the netlab-manager JVM on 198.199.106.165 stays alive (no OOM, stable PID, modest heap) while the host sits at load ~36–45 on 8 cores during suites — connection handling starves while the process looks healthy. The connection-handling fixes are already merged on UrlResolver/netlab main; the running binary predates them.
  • Full escort/diagnosis records (per-run failure attributions, measurements): https://github.com/CodexCoder21Organization/PlanRepository/tree/handoff-artifacts/urlresolver-campaign-2026-07-28 (remote head 29fe3e9c29c2).

Relevant PRs / refs

  • https://github.com/CodexCoder21Organization/UrlResolver/pull/873 — branch fix/shutdown-admission-race, head ea34056614a9581fefcb0f6bd11f5a0d61bd2573, builds, focused tests 0/20 failing.
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/882 — branch fix/relay-same-tier-contention, head 793fa6d871749b5127dd0175d057be69d9b286ec, builds, 6/6 focused tests green.
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/866 — branch fix/gossip-forwarding-state-retention, head 3dbabc47deff4fcc65c8bc237dafda591e15d188, builds, 6/6 focused tests green incl. heap-churn test in 9.7 s.
  • Merged this session: https://github.com/CodexCoder21Organization/UrlResolver/pull/852, https://github.com/CodexCoder21Organization/UrlProtocol/pull/416, /435, /436, /437, /438, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/386, https://github.com/CodexCoder21Organization/kompile-buildscript/pull/68; artifact foundation.url:protocol:0.0.416 published.
  • Held campaign PRs (untouched, land after the estate recovers): https://github.com/CodexCoder21Organization/UrlResolver/pull/823 (CLEAN), /838, /848, /872 (BEHIND — rebase first), /850 (re-request suites).
  • Post-merge closures owed once 866 lands: https://github.com/CodexCoder21Organization/UrlResolver/pull/737, /870 (refresh containment evidence — it gained commit a427b7c5), /804; also raise the small warmup follow-up carrying commit 74a8efc901a4e6babcff58a0f98d9273ed9d21df if still absent from main.
  • Flake challenges filed this session: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-2107-the-sandboxed-cold-start-self-heal-test-intermittently.md, .../2026-07-28-1308-relay-contention-check-intermittently-chooses-the-lower.md, .../2026-07-28-1941-a-fixed-2-second-pex-deadline-intermittently-fails-on-a.md, .../2026-07-28-0242-urlresolver-healthy-same-tier-relay-stress-test-fails-on.md.

Next steps (in order)

  1. Redeploy netlab-manager-server from current merged main (contains the connection fixes) and apply the ContainerNursery HTTPS-facade keepalive deploy — both awaiting explicit user authorization; do not deploy without it. This is the gating step: until the deployed manager matches main, full resolver suites fail on infrastructure at a rate that makes queue validation a coin-flip.
  2. After deploy, verify with netlab-cli health plus one full head suite on https://github.com/CodexCoder21Organization/UrlResolver/pull/873 (its head is green from two prior full passes, so a fresh green re-confirms the estate rather than the code).
  3. Enqueue 873 solo via GraphQL enqueuePullRequest (org rejects gh pr merge), escort with build-watchman 0.0.13 --to-merged; then 882, then 866, one at a time — never stacked (speculative queue stacks produced phantom merge_conflict evictions twice).
  4. Post-merge: run the documented containment comparison, then close 737/870/804 with linked evidence; land the held PRs (823, then 838/848/872 after rebase, then 850).
  5. Orchestrator final reviews are still owed on 873/882/866 before or immediately after landing: specifically (a) triage 866's removal of one diagnostic-history assertion (the delegate justified it; needs independent judgment), (b) confirm 882's fix actually resolves the lower-tier-selection challenge 2026-07-28-1308, and (c) verify no test anywhere reduced its caller count (an unpushed 500→100 caller reduction was once drafted in a lane workspace — any pushed reduction is a defect).
  6. Netlab manager redeploy is user-gated; record stands: no production action without explicit authorization.

Reusable / operational knowledge

  • Gate queue retries on the dependency's health, not on PR cleanliness — re-enqueue only after netlab-cli health succeeds; a run enqueued during degradation is a wasted validation.
  • The merge queue's speculative stacking dissolves position-2 entries when position 1 fails; the reported merge_conflict on the dissolved entry is phantom — verify against a fresh fetch + git merge-base --is-ancestor origin/main <head> before believing any conflict.
  • Executor ground truth for any red: /root/buildtest-data/runs/<id>/ (build.log, test-events.jsonl) on 198.199.106.165 (ssh port 23, last resort) — grep for re_lease_exhausted and closed while requests were still pending.
  • Build-watchman's coarse sharded counter lags the detailed event stream; a "stall" warning with a moving test-events.jsonl is false.