HHandoff
← Priority list

Handoff: UrlResolver PR batch landing + `testBytecodeFetch` de-flake (#767) + #747↔#749 collision

Autonomous-loop shepherding of a batch of **UrlResolver** PRs to green/merge-ready. Mid-session the user asked, verbatim: **"Did you create a PR which fixes the testBytecodeFetch flake?"** — expanding the mission to **creating and landing a de-flake PR** for `testBytecodeFetchAggregatesRelayFailureDetails` (now **[#767](https://github.com/CodexCoder21Organization/UrlResolver/pull/767)**, green), which was the highest-frequency flake blocking the whole `foundation.url:resolver` pipeline. The user is now actively merging the batch. **Nothing is merged by the agent without explicit user approval** (handoffs excepted); this doc is NOT authorization to merge code PRs.

hf-2026-07-12-urlresolver-pr-batch-green-and-testbytecodefetch-deflake

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver PR batch landing + testBytecodeFetch de-flake (#767) + #747↔#749 collision

Written: 2026-07-12 ~21:50 UTC. (Updated snapshot — supersedes the earlier version of this file.)

RE-VERIFY BEFORE ACTING. Every state claim below was true only at write time. PRs merge, CI re-runs, merge queues shuffle, branches get force-pushed, main advances (bumping the resolver coordinate and re-conflicting open PRs), and buildtest.kotlin.build goes in and out of reachability. A fresh query always wins over this document. Authoritative re-checks:

  • PRs: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,headRefOid,mergedAt
  • buildtest run: https://buildtest.kotlin.build/run?id=<id> (HTML only, no JSON API); reachability: curl -sS --max-time 15 -o /dev/null -w '%{http_code}' https://buildtest.kotlin.build/ (000 = down)
  • check-runs for a head: gh api repos/CodexCoder21Organization/UrlResolver/commits/<sha>/check-runs

Mission

Autonomous-loop shepherding of a batch of UrlResolver PRs to green/merge-ready. Mid-session the user asked, verbatim: "Did you create a PR which fixes the testBytecodeFetch flake?" — expanding the mission to creating and landing a de-flake PR for testBytecodeFetchAggregatesRelayFailureDetails (now #767, green), which was the highest-frequency flake blocking the whole foundation.url:resolver pipeline. The user is now actively merging the batch. Nothing is merged by the agent without explicit user approval (handoffs excepted); this doc is NOT authorization to merge code PRs.

Current disposition (write-time, 21:50 UTC — RE-VERIFY)

| PR | Title | State @ write | Head | Note | |----|-------|--------------|------|------| | #767 | de-flake testBytecodeFetch | OPEN / CLEAN (green) | ebd92bbf | The de-flake. Fully validated + green. Merge-ready. Branch fix/deflake-bytecode-fetch-aggregates-relay-failure | | #616 | ByteArray RPC-param marshaling coverage | OPEN / CLEAN (green) | d8e50c10 | Merge-ready. No coordinate bump → doesn't re-conflict | | #605 | openConnection JSON encoding fix | OPEN / BLOCKED (CI re-running) | 2de3db00 | Re-rebased to pin 0.0.633; CI in progress. Pin treadmill (see below) | | #747 | retry withdrawals while gossip dial pending | OPEN / DIRTY | b664cf46 | Needs a decision — collides with #749 (see below). Branch fix-registration-withdrawal-resurrection-race | | #632 | netlab tests for degraded networks | MERGED (17:38 UTC) | d24887ba | Done | | #595 | netlab relay-reconnection test | OPEN / DRAFT / BEHIND | cf55687e | Deferred; rebase + un-draft after the batch lands |

main @ write time: b3760739 (#653), resolver coordinate 0.0.632.

What was found and done (the chain — verified unless flagged)

  1. testBytecodeFetchAggregatesRelayFailureDetails was a suite-wide flake (VERIFIED). Failed today on ≥4 distinct commits across sessions, ~40–50%/run. From merged #733 (1df283ec). Root cause: the test rebuilt expected after the fetch by re-reading the live client.getKnownPeers() set and embedding each relay peer's ${it.multiaddresses}, then exact-assertEquals'd against the exception.message production built during the fetch. That registry/address set mutates asynchronously (peer exchange / p2p-circuit reservations), so the two snapshots diverge nondeterministically. Production is correct.
  2. Fixed via #767 (codex gpt-5.6-sol per the global bug-fix workflow). Branch head ebd92bbf, 2 commits: 082221f0 core fix (deterministic reproducer: add a late relay peer → old assertion fails 10/10; new assertion asserts the stable header + full direct-candidate failure + a regex for the full relay reason, wildcarding only the volatile address list; 20/20 pass); ebd92bbf tightening from an adversarial review finding (require endsWith("]"), split the relay block into ; -delimited entries, assert the known relay as one anchored complete entry AND every sibling well-formed; 21/21 + a negative test). Reviews (codex) approved on substance.
  3. #767 fix is VALIDATED and now GREEN. testBytecodeFetch passes on both CI paths. Getting the whole run green took several rerequests because the suite has other independent flakes (see reusable knowledge) + fleet degradation (a wedged run stuck at 3/1094 for 31 min, and a buildtest outage http_code=000). It is CLEAN now.
  4. #605 rebased repeatedly onto an advancing main. The resolver's own published coordinate lives in build.kts and every source PR bumps it, so #605 (a source change) re-conflicts on that one line each time any bumping PR merges: rebased 0.0.626 → 0.0.628 → 0.0.633. Current head 2de3db00, CI re-running. Each conflict is ONLY the coordinates = "…" line; resolve by keeping main's changelog and setting the coordinate to main+1. This is a treadmill — on the slow fleet #605's CI takes ~1–2h, during which main often advances again → DIRTY again. The clean escape is to merge it while green.
  5. #747 now collides substantively with the just-merged #749 (NEW — the key open item). Main's 0.0.627 is #749 ("registration-after- withdrawal ordering regression", commit 95e53898), which fixes the same withdrawal-resurrection race (issue #746) that #747 targets, editing the same src/foundation/url/resolver/UrlResolver.kt. #747 is a large change there (282 lines, 202+/80−), so rebasing it hits a 6-block source conflict (plus the build.kts pin). This is a genuine collision between two fixes for the same concurrency race — NOT a mechanical rebase, and force-resolving it risks reintroducing the race. Left untouched (rebase aborted; remote branch unchanged at b664cf46). #747's own 4 commits: b664cf46 close-ordering guard + idle drain, 0d88b23f harden queue admission, d2a821d7 keep withdrawals queued beyond retry budget, 583deb83 fix withdrawal loss during pending gossip dial.

Next steps (ordered — re-verify current state first)

  1. Re-verify all PRs with the banner command; do not trust the table above.
  2. #747 — needs a human/owner decision (the single most important open item): Determine how #747 relates to the now-merged #749. Either (a) reconcile — rebase #747 onto post-#749 main, carefully merging its additional mechanisms (withdrawal loss during a pending gossip dial, queued-beyond-retry-budget, queue-admission hardening, close-ordering guard + idle drain) on top of #749's fix, then verify the race is still fixed (full TDD, delegate to codex gpt-5.6-sol); or (b) reassess scope — check whether #749 already covers enough that #747 should be rescoped/closed. Do the 6-block UrlResolver.kt conflict resolution with tests, not by hand-picking hunks.
  3. Merge decisions (need the user): #767, #616 are green/merge-ready now; #605 when its re-run is green. Independent PRs (not competing). Do not merge without explicit approval; do not bypass CI / merge queue (--admin/--no-verify forbidden). Recommend merging the green ones promptly to end #605's pin treadmill.
  4. #605: if it goes DIRTY again before merging, re-rebase (pin → current main coordinate +1) and push --force-with-lease. If its kotlin.build run wedges (stuck count for >~15 min) or fails on infra (0 tests / "Waiting for droplet capacity" / shard "BuildTestRunner … is gone"), rerequest.
  5. #595: rebase onto main + un-draft only after the batch lands.

Reusable / operational knowledge

  • Known suite flakes (rerequest, don't "fix" unless doing a real de-flake): testBytecodeFetch (~45%, FIXED by #767 — not yet on main until #767 merges, so other open PRs still trip it), stressTestRegistrationAfterWithdrawalRaceUnderContention (~1–2/32), netlab-hosted tests (testNetlabPartitionAndHealAcrossNats, testNetlabDynamicFailureRecovery, testNetlabResolutionUnderLatencyAndPacketLoss — external netlab infra), shard-death crashes, stressTestConcurrentRegistrationsReliablyPropagateToRelay (exit code 1).
  • Re-run a red check w/o a code change: kotlin.build (remote): gh api --method POST repos/CodexCoder21Organization/UrlResolver/check-runs/<id>/rerequest (id from commits/<sha>/check-runs). bld-build: gh run rerun <run_id> --failed. Confirm both flip to in_progress on the head.
  • Infra-failure signatures: shard death = "The BuildTestRunner process on the droplet is gone …"; DO provisioning = "Failed to create droplet in all regions" + "Waiting for droplet capacity" (0 tests run); wedged run = completed-count frozen while status=TESTING for >~15 min (rerequest it); http_code=000 = CN-relay TLS starvation (wait for recovery, don't churn). Runs can also crawl (coursier resolve on the CPU-starved CN box) — advancing slowly ≠ wedged.
  • build.kts pin-conflict resolution (the #605 treadmill): the conflict is the single coordinates = "foundation.url:resolver:0.0.NNN" line; keep main's changelog block, append the PR's own changelog entry renumbered to main+1, set the coordinate to main+1. kompile builds each tests/test*.kts independently by its own @file:WithArtifact pins, so a test-only PR (#616, #767) does NOT bump the coordinate and does NOT re-conflict — only source PRs (#605, #747) treadmill.
  • codex delegation: codex exec -C <repo> -m gpt-5.6-sol --dangerously-bypass-approvals-and-sandbox -o <last-msg> "$(cat brief.md)" </dev/null > run.log 2>&1 & — </dev/null is mandatory (stalls otherwise); poll the last-msg file + git log on the branch; nohup/& background job (no auto-notify).
  • Local clones: ~/workspace writable (/code/workspace not); gh repo clone (git https has no creds); set git identity on fresh clones.
  • Prior handoff PR (merged): https://github.com/CodexCoder21Organization/PlanRepository/pull/1167 (this file supersedes that snapshot).