Standing request (verbatim intent): *"keep working (flakey test skill) and merging until UrlResolver goes consistently green. Ensure codex does an adversarial review of each change before we merge."* Then, mid-effort: *"let's split it now. Create a PR and link it to me. Create a handoff for the remaining work/issues."*
Moves it out of every priority list and into ArchiveArea.
Written: 2026-07-16 (UTC). Effort began 2026-07-15.
⚠️ RE-VERIFY BEFORE ACTING. Every state claim below is a write-time snapshot and goes stale. Re-check before doing anything:
- PRs:
gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt- Published artifacts:
curl -s -o /dev/null -w "%{http_code}" https://kotlin.directory/<group-path>/<artifact>/<ver>/<artifact>-<ver>.pom- UrlResolver main:
gh api repos/CodexCoder21Organization/UrlResolver/commits/main --jq '.sha[0:9]'A fresh query always wins over this document.
Standing request (verbatim intent): "keep working (flakey test skill) and merging until UrlResolver goes consistently green. Ensure codex does an adversarial review of each change before we merge." Then, mid-effort: "let's split it now. Create a PR and link it to me. Create a handoff for the remaining work/issues."
This handoff covers the one flake that is fixed-but-not-fully-landed: stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation in UrlResolver. It was root-caused to two distinct upstream libp2p defects. The first (multistream Negotiator close-race) has been split into its own PR. The second (a rarer Noise-handshake stall under extreme dial saturation), the downstream protocol/version reconciliation, and the resolver-side pin/recovery PR all remain.
Reliable local reproduction (verified). At the fixed pin, stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation failed ~40–60% of runs locally (operator loop 3/8; codex baselines 3/8 and 2-CPU-amplified up to 3/5). Every failure had one deterministic signature: assertion at tests/stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation.kt:209 ("PERMANENTLY failed to learn 1 service"), accompanied in node logs by [PEX-EMPTY] re-pull of configured bootstrap … could not complete and a libp2p multistream handler-removal exception during stream negotiation. No OOMs, no env-acquisition errors.
Root cause #1 — multistream Negotiator close-race (verified, fixed, PR open). Under many concurrent stream negotiations to one peer (52 providers → one shared relay), a selected protocol's setup can synchronously destroy a closing child stream's pipeline while the Negotiator's terminal-close callback is still in flight; the second cleanup threw NoSuchElementException removing an already-removed handler, aborting negotiation and silently dropping the provider's register-time peer exchange (relay never learns the service). Fix = make Negotiator handler removal idempotent. Split into jvm-libp2p PR #26 (base develop), with a new MultistreamTest case. This fix alone took the flake from ~40–60% → ~8% (11/12 clean).
Root cause #2 — Noise-handshake stall under saturation (hypothesis, fix in progress, NOT landed). The residual ~1/12 shows dial failures stuck still-handshaking>Xms. Codex's working hypothesis: the Noise read timeout is being measured across time when no outbound handshake frame is actually awaiting a remote response, so under extreme concurrency it trips spuriously. The direction: measure the Noise read timeout only while an outbound handshake message is genuinely awaiting its remote response. This is not yet verified to a clean 12/12 — it hit 11/12 and occasionally 12/12 but never a statistically-confident clean pass before the run was stopped to split out fix #1. Treat as hypothesis, not proven.
Resolver-side recovery (verified as part of the 11/12). On branch fix/saturation-resync-confirmed-pex (commit a7311b12d, pushed): src/foundation/url/resolver/UrlResolver.kt gains PeerExchange-driven relay-reservation recovery (wakes relay-reservation recovery when PeerExchange proves a configured relay candidate's transport is healthy) plus a formatThrowableChain dial-failure cause-chain diagnostic. It also bumps protocol pins to 0.0.362. Not PR'd yet because of version skew (below).
Protocol version skew (verified, blocks the resolver PR). While this work was in flight, a parallel lane published protocol 0.0.363/0.0.364 (adding a "bounded acknowledgement response projection"). The committed WIP pins 0.0.362, which is behind what UrlResolver main now expects — pinning it would regress the parallel lane. Codex was reconciling to 0.0.365 (0.0.364 base + negotiator fix + Noise fix). Both 0.0.362 and 0.0.365 are already published to kotlin.directory — but ⚠️ a higher version number is NOT automatically a content superset; verify 0.0.365's actual contents before trusting it (see the parallel-lane-version-content-skew gotcha below).
A codex run died on the model-capacity ceiling (~1.3M tokens / ~2h). Its work was preserved via WIP commits and the run relaunched. Mentioned so the next driver expects it and preserves-then-relaunches rather than losing work.
| Item | State @ write | Link |
|---|---|---|
| jvm-libp2p negotiator fix (the split PR) | OPEN, mergeStateStatus CLEAN | jvm-libp2p PR #26 |
| UrlResolver resolver-side recovery + 0.0.362 pins (WIP, pushed, no PR) | branch pushed, head a7311b12d | fix/saturation-resync-confirmed-pex on UrlResolver |
| UrlProtocol snapshot-15 + protocol 0.0.362 (WIP, pushed, no PR) | branch pushed, head 0717f12 | fix/multistream-negotiator-close-race on UrlProtocol |
| Noise-handshake residual (in-progress fix) | local git stash only — exported as patch artifact | artifacts/2026-07-15-noise-handshake-residual.patch |
| Codex residual notes | ephemeral capture | artifacts/2026-07-15-codex-noise-residual-notes.txt |
Already merged this session (context, not remaining work): UrlResolver PR #769, UrlResolver PR #776, UrlProtocol PR #360, UrlProtocol PR #362, jvm-libp2p PR #25. UrlResolver main was at b4eeb7c7b.
⚠️ The 0.0.365 pin-bumps in UrlProtocol and UrlResolver are in local git stashes only (residual-0.0.365-pins in each of /code/workspace/UrlProtocol and /code/workspace/UrlResolver). They are regenerable mechanical pin bumps, but they live only on the /code box — do not rely on them surviving.
Re-verify everything in the table above (PR #26 state, whether main advanced past b4eeb7c7b, whether a newer protocol version than 0.0.365 now exists, whether the parallel lane's 0.0.364 work has merged/moved).
Shepherd jvm-libp2p PR #26 to green + merge (needs its own review; adversarial-review it per the standing instruction before merging). It is self-contained (Negotiator + MultistreamTest) and independently valuable.
Finish root cause #2 (Noise-handshake residual) under fix-flaky discipline. Restore the stashed WIP from artifacts/2026-07-15-noise-handshake-residual.patch (touches libp2p/src/main/kotlin/io/libp2p/security/noise/NoiseXXSecureChannel.kt, .../security/noise/NoiseHandshakeOffEventLoopTest.kt, and a NoiseHandshakePathHarness "saturated" mode: taskset -c 0,1, SATURATED_DIALERS=120, SATURATION_SPINNERS=32). First reproduce the residual reliably (≥50%) with an amplified harness, then confirm/refute the "measure the read timeout only while genuinely awaiting a remote handshake frame" hypothesis, fix it upstream in jvm-libp2p, and verify the reproducer flips to a clean 12/12 (statistical confidence, not one lucky pass). Have codex gpt-5.6-sol do the implementation; do an adversarial review before merge.
Publish the reconciled artifacts. After #26 (and the Noise fix) merge: publish a new jvm-libp2p snapshot from merged develop; then rebuild+publish a protocol version that is a true content superset of current UrlResolver-main's protocol (0.0.364's bounded-ack projection + the negotiator fix + the Noise fix). Verify contents, not just the version number.
Land the resolver-side PR. Rebase fix/saturation-resync-confirmed-pex on latest UrlResolver main, bump the protocol pins from 0.0.362 to the reconciled version, keep the UrlResolver.kt relay-reservation recovery + formatThrowableChain diagnostic, re-verify stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation 12/12 locally, then open the UrlResolver PR and shepherd it green through the merge queue.
Re-measure UrlResolver main's flake rate after the chain lands, to confirm the saturation flake is actually gone.
Separate still-open flakes (task-tracker #12, not part of the above):
testRelayRestartSameIdentityReattach — failed on the b4eeb7c post-merge main run: after a relay restarts with the same identity, a fresh client cannot resolve the NAT-ed provider within the 30s budget (Both HTTP relay forwarding and direct libp2p connection failed … TimeoutException, ~54.4s). Not yet reproduced locally or fixed.testHealthySameTierRelayWinsBeforeLowerTier (relay tier-selection) — documented, not reproduced.ERROR: Selected model is at capacity. Preserve work with WIP commits before it hits that, and relaunch from the committed state with a "do not redo finished work, continue from here" brief.coursier launch community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.5 -r https://kotlin.directory -- --workspace /code/workspace/UrlProtocol --buildrule foundation.url.protocol.buildMaven --api-url https://api.kotlin.directory/upload --force. Check publish with the .pom HTTP code (200 = present).cd /code/workspace/UrlResolver && ./scripts/test.bash --test stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation (compile-heavy, ~3 min/run).pkill/pgrep -f patterns in codex prompts or in your own kill loops — pgrep -f "codex exec" self-matches your own shell (the footgun bit this session). Kill codex by explicit PID, or match the actual node+codex process.enqueuePullRequest, poll state until MERGED; never --delete-branch.