Handoff: UrlResolver #752 (relay-restart recovery) is green + CLEAN but stuck behind an ~18h merge-queue jam; the underlying resolver-suite starvation was root-caused and durably fixed (CN → jvm-libp2p snapshot-12), and that fix is merged to CN main
Written: 2026-07-12 ~17:10 UTC.
⚠️ RE-VERIFY BEFORE ACTING
Every state claim below is a write-time snapshot (2026-07-12 ~17:07 UTC) and will drift. Merge queues shuffle, CI re-runs, entries get flake-dropped, and the fleet's health changes hour to hour. Re-verify before you act:
- PRs:
gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
- Merge queue:
gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){pullRequest(number:752){isInMergeQueue mergeQueueEntry{position state}} mergeQueue{entries(first:8){nodes{position state pullRequest{number}}}}}}'
- Whether the queue is draining at all:
gh api "repos/CodexCoder21Organization/UrlResolver/commits?per_page=1" --jq '.[0].sha[0:10]+" "+.[0].commit.committer.date' — if the top commit date is still 2026-07-11T22:36:46Z, the queue is still jammed.
A fresh query always wins over this document.
Mission summary
Original user request (verbatim intent): "https://github.com/CodexCoder21Organization/UrlResolver/pull/752 is red — ensure it goes green and gets merged." The merge was explicitly user-authorized.
The task expanded once the "red" was diagnosed. #752 was not failing on its own code — it was a victim of a fleet-wide UrlResolver-test-suite CPU starvation on buildtest.kotlin.build. The user was presented the options and chose the durable root-cause path ("Fix-A"): land the yamux fix in jvm-libp2p → publish a snapshot → get it into the ContainerNursery libp2p relay, rather than mitigate. So this workstream became two tracks:
- Primary: get #752 green and merged.
- Durable: fix the relay-CPU starvation at its source so it stops blocking all resolver PRs.
What was found and done (the chain — verified unless flagged)
- "#752 is red" was a dead zombie CI run, not a code failure. The
kotlin.build (remote) check on #752 (head efaa494a) had been in_progress for 4.5 hours (run id d2421ad6) showing 203 passed / 886 failed, where all 886 failures had the identical 0ms message "Test did not complete during build execution" — i.e. an infrastructure mass-abort (droplets reaped), zero real assertion failures. Verified by pulling the run page https://buildtest.kotlin.build/run?id=d2421ad6 and parsing the badges.
- The starvation was fleet-wide and resolver-specific. Three unrelated resolver commits (
#700, #762, #752) all wedged identically (hundreds of tests pending, 1–3 running = near-zero concurrency), while non-resolver suites (buildtest-wui 180/180, droplet-service-server 114/114, buildtest-embedded 518/519) completed normally in 9–15 min. So this was shared-infra starvation, not a #752 defect.
- Root cause (verified via jvm-libp2p PR#23 evidence + prior memory): ContainerNursery's libp2p relay (
RelayService.sendViaRelay, which base64s + 128KiB-chunks payloads) drains those chunks through jvm-libp2p's yamux ByteBufQueue, whose take did removeAt(0) in a loop — O(n²) — pinning relay CPU under JAR-bootstrap relay load. jvm-libp2p PR#23 replaced it with an ArrayDeque drain (42.4s → 0.166s on the drain benchmark).
- The fix was already partly landed. jvm-libp2p PR#23 was already MERGED (08:17:08Z) and
jvm-libp2p:1.3.0-codexcoder21-snapshot-12 was already published to kotlin.directory (POM HTTP 200). But ContainerNursery and UrlProtocol both still pinned snapshot-9, so the production relay lacked the fix.
- CN-only bump is binary-compatible — no UrlProtocol republish needed (verified by build). A codex gpt-5.6-sol run bumped CN
build.kts snapshot-9 → snapshot-12, built the fat jar GREEN (214s, sha256 8b5c7ca4afddac03550114df95c801d4e6f1a438303aa7167367b473bd1ef4e3) with no API incompatibility. Two constrained-heap stress tests (stressTestServerSurvivesMidNegotiationChurnDropWithConstrainedHeap, …SustainedSubstreamChurnOnLiveConnectionsWithConstrainedHeap) initially failed because they hard-assert the libp2p jar version via require(libp2pJar.name.contains("…snapshot-9")); a second codex run bumped those guards + ~15 test-file @file:WithArtifact pins to snapshot-12 in lockstep (historical narrative comments about snapshot-5/8/9 left intact). Full local kompile suite then 331/331 green.
- CN #522 created, CI green, MERGED to main. The one Gradle-CI failure was the known pre-existing
HttpsAcceptLoopSurvivesHandlerFatalErrorTest.testAcceptLoopKeepsServingAfterHandlerFatalOom OOM flake (CN PR #424 exists to delete it; branch fix/https-facade-test-flakes addresses it) — mechanistically unrelated to yamux; it cleared on re-run. User chose "merge to main, hold deploy." CN #522 MERGED at 14:24:41Z → CN main HEAD eecb5d4a "Bump jvm-libp2p to snapshot-12 for relay CPU fix (#522)"; build.kts now pins snapshot-12. The durable fix is now in CN's source of truth, so any future CN deploy carries it (this also dissolves the earlier "competing staged container-nursery-tailfix-staged.jar" collision worry).
- The fleet self-recovered (~09:30Z) and #752 went green. Resolver runs started completing again (several 1088/1088), prod load fell 38→16 on 8 cores. I cleared the dead zombie by re-requesting #752's
kotlin.build check (gh api -X POST repos/.../check-runs/<id>/rerequest); the fresh run 7a8cfa73 came back SUCCESS. #752 flipped MERGEABLE / CLEAN (both head checks green) and I enqueued it.
- BLOCKER (current): the UrlResolver ALLGREEN merge queue is jammed.
main has merged nothing since 2026-07-11 22:36:46Z (~18.6h). #752's own head is green, but the combined queue-branch CI (gh-readonly-queue/main/pr-752-…) keeps failing — both bld-build and kotlin.build (remote) flake intermittently on the batch branches. This is a separate 1–2-test gossip/peer-exchange flake, independent of the CPU-starvation that snapshot-12 fixes — so the CN deploy will not fix it. #752 was flake-dropped repeatedly; a self-healing watcher re-enqueued it each time, but re-enqueue puts it at the tail, and the queue is also clogged with UNMERGEABLE entries. At write time #752 is OUT of the queue entirely and must be re-enqueued.
Relevant PRs / refs (write-time state — RE-VERIFY)
| Item | State @ 17:07Z | Link |
| --- | --- | --- |
| UrlResolver #752 (relay-restart reservation restoration; the goal) | OPEN, mergeStateStatus CLEAN, both head checks SUCCESS, NOT in queue (needs re-enqueue). Branch fix-relay-restart-recovery-742, head efaa494a75ae78fdb06319915b26c99f5d1fbf2d, node id PR_kwDOQjgD7s7wjobH | https://github.com/CodexCoder21Organization/UrlResolver/pull/752 |
| UrlResolver #608 (NAT↔NAT relay; closes when #752 merges) | OPEN, DIRTY (needs rebase, or auto-closes on #752 merge) | https://github.com/CodexCoder21Organization/UrlResolver/pull/608 |
| ContainerNursery #522 (durable fix: jvm-libp2p snapshot-9→12) | MERGED 2026-07-12 14:24:41Z (CN main eecb5d4a) | https://github.com/CodexCoder21Organization/ContainerNursery/pull/522 |
| jvm-libp2p #23 (yamux ByteBufQueue O(n²) → ArrayDeque) | MERGED 2026-07-12 08:17:08Z; shipped as snapshot-12 | https://github.com/CodexCoder21Organization/jvm-libp2p/pull/23 |
| UrlResolver #759 (de-flake synchronous PeerExchange latency guard) | OPEN, BLOCKED — in-flight de-flake work by others; may overlap if you fix the combined-CI flake | https://github.com/CodexCoder21Organization/UrlResolver/pull/759 |
Artifact (not needed unless deploying): CN fat jar with snapshot-12 built at ~/workspace/ContainerNursery/container-nursery.jar, sha256 8b5c7ca4afddac03550114df95c801d4e6f1a438303aa7167367b473bd1ef4e3 — rebuildable from CN main via ./scripts/build.bash containernursery.buildFatJar container-nursery.jar.
Next steps (ordered — start by re-verifying)
- Re-verify current state first (see the RE-VERIFY banner). Especially: is #752 merged? is the queue draining (top-of-main commit date moved past 22:36Z 07-11)? is #752 in the queue?
- Re-enqueue #752 — at write time it is OUT of the queue. It merges only from inside the queue:
gh api graphql -f query='mutation{ enqueuePullRequest(input:{pullRequestId:"PR_kwDOQjgD7s7wjobH"}){ mergeQueueEntry{ position state } } }'
(gh pr merge is rejected — the repo requires the merge queue; ALLGREEN + SQUASH.)
- Keep it enqueued and wait for one clean combined batch run. #752 will merge the instant the queue produces a run where every batched entry passes both
bld-build and kotlin.build (remote). Re-enqueue on each flake-drop. Do not bypass the queue (no --admin) — a passing branch does not justify skipping the merge-queue integration test, and it is policy-forbidden.
- PENDING USER DECISION (unanswered — the user pivoted to "create a handoff" before choosing): how to get past the queue jam —
- (a) Wait it out — lowest risk; convergence depends on the shared flake clearing (de-flake PRs like #759) or the queue draining; could be minutes to many hours.
- (b) Actively fix the combined-CI flake — identify the specific test(s) failing on
gh-readonly-queue/main/pr-* branches (both harnesses flake), reproduce, and fix via codex gpt-5.6-sol. Check #759 and the UrlResolver open-PR list first to avoid duplicating/colliding with in-flight de-flake work.
- (c) Stand down — leave #752 enqueued to merge whenever the jam clears.
- DO NOT deploy the CN relay to prod without explicit user go-ahead. The user chose "merge to main, hold deploy" (13:20Z). The snapshot-12 fix is in CN main; a redeploy of
bin/container-nursery.jar on 198.199.106.165 (currently running an older jar; process PID observed 2089839) would make it live, but the fleet self-recovered so it is not urgent, and the deploy is delicate (CN hosts everything). If asked to deploy: prefer HardwareControlFabric/ContainerNursery skills over SSH; rebuild the jar from CN main so it carries any other merged work; verify CN health after.
- #608 is DIRTY — it auto-closes when #752 merges; only rebase it if #752's approach changes.
Reusable / operational knowledge
- Diagnosing a "red" resolver PR: the
kotlin.build (remote) check is a buildtest run — open https://buildtest.kotlin.build/run?id=<id> (id is in the check-run details_url). Per-test rows carry badge badge-<status>; "Test did not complete during build execution" at 0ms = infra mass-abort (droplets reaped), not a real failure. Contrast with genuine pending/running (a slow-but-live run) and a real assertion failure (nonzero duration + specific message).
- Fleet-health tell: on
https://buildtest.kotlin.build/, if non-resolver suites complete in ~10 min but every foundation.url:resolver run wedges, it's resolver-suite starvation, not a global outage. Prod load: ssh -p 23 -i ~/.ssh/id_ed25519 root@198.199.106.165 uptime (box is 8 real cores, nproc; the "1vcpu" slug is a lie).
- Merge-queue mechanics: UrlResolver and ContainerNursery both use GitHub merge queues (ALLGREEN, SQUASH, repo auto-merge OFF). Enqueue via GraphQL
enqueuePullRequest with the PR node id (gh pr view <n> --json id). The queue tests a combined gh-readonly-queue/main/pr-<n>-<sha> branch; a PR green on its own head can still be dropped if the combined batch flakes. Re-enqueue puts you at the tail.
- CN jvm-libp2p pin:
ContainerNursery/build.kts line ~279: MavenPrebuilt2("community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-12"). The snapshot-5/8/9 mentions elsewhere in that file are historical changelog comments — do not blind-bump them. Test scripts also pin libp2p via @file:WithArtifact, and the two …ConstrainedHeap stress tests hard-assert the version via require(libp2pJar.name.contains("…snapshot-N")) — bump those in lockstep with the module or they fail with "pins snapshot-N … but the runtime jar is snapshot-M".
- The snapshot-12 deploy does NOT fix #752's merge flake. The yamux fix addresses CPU-starvation wedges (mass "did not complete"); #752's queue-jam is a separate intermittent 1–2-test gossip/peer-exchange flake. Don't conflate them.
- codex gpt-5.6-sol (used for the CN build/test/PR work):
codex exec -m gpt-5.6-sol --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check -C <dir> "<prompt>" </dev/null > out.log 2>&1 & — must redirect stdin from /dev/null and output to a file, or it stalls.
- Related prior handoffs: 608 NAT↔NAT relay land-and-merge, the jvm-libp2p/UrlProtocol/UrlResolver/CN entropy fix chain.