HHandoff
← Priority list

Handoff: URL-facade libp2p entropy flake — upstream crypto-fix chain (jvm-libp2p ✅, UrlProtocol ✅, UrlResolver in-flight, ContainerNursery not started)

This continues the handoff [`handoffs/2026-06-30-cn-477-httpclient-reuse-flake-fixed-and-urlfacade-p2p-entropy-flake.md`](2026-06-30-cn-477-httpclient-reuse-flake-fixed-and-urlfacade-p2p-entropy-flake.md), whose §4 left an open decision. User request this session (verbatim): *"Pull any latest changes into your skills repository (if any upstream) and then let's drive [that handoff] to completion."* When asked which fix approach to take, the user chose **approach B: fix the owning crypto libraries upstream** (not a downstream mitigation).

hf-2026-06-30-urlfacade-entropy-flake-upstream-fix-chain-jvmlibp2p-urlprotocol-urlresolver-cn

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: URL-facade libp2p entropy flake — upstream crypto-fix chain (jvm-libp2p ✅, UrlProtocol ✅, UrlResolver in-flight, ContainerNursery not started)

Written: 2026-07-12 (authored/paused). Workstream continues the effort dated 2026-06-30.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time (2026-07-12 ~16:50 UTC). PRs merge, branches get force-pushed, CI re-runs, the buildtest droplet restarts, external automation snipes Maven version numbers, and flakes fire intermittently. Re-verify before you act. Authoritative checks:

  • PRs: gh pr view <num> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid
  • Published artifact exists? curl -s -o /dev/null -w "%{http_code}\n" "https://kotlin.directory/foundation/url/protocol/<v>/protocol-<v>.pom" (200 = taken, 401 = free). Same shape for resolver and for community/kotlin/libp2p/jvm-libp2p/<v>/jvm-libp2p-<v>.pom.
  • A published artifact's VERSION NUMBER does NOT imply it carries this chain's fixes — parallel branch lines published overlapping numbers. Always re-verify the JAR contents (see Reusable knowledge).
  • kotlin.build run: gh api repos/CodexCoder21Organization/<repo>/commits/<sha>/check-runs --jq '.check_runs[] | select(.name=="kotlin.build (remote)") | {status,conclusion,output:.output.summary}'

A fresh query always wins over this document.

1. Mission summary

This continues the handoff handoffs/2026-06-30-cn-477-httpclient-reuse-flake-fixed-and-urlfacade-p2p-entropy-flake.md, whose §4 left an open decision. User request this session (verbatim): "Pull any latest changes into your skills repository (if any upstream) and then let's drive [that handoff] to completion." When asked which fix approach to take, the user chose approach B: fix the owning crypto libraries upstream (not a downstream mitigation).

The flake being fixed (root cause proven in the source handoff, not re-derived here): ContainerNursery's testUrlFacade_autoStartPersistentRouteAndRestartResilience intermittently times out (30 s budget) on kotlin.build (remote) because a fresh test JVM blocks ~10 s seeding the JVM-default SecureRandom (NativePRNG) from the blocking /dev/random on the entropy-starved kernel-5.4 buildtest droplet, during libp2p host startup. The fix makes every host-startup / key-generation entropy source use NativePRNGNonBlocking (reads /dev/urandom only; cryptographically equivalent on any booted host, not a security downgrade).

How the effort expanded (in order): fixing the entropy at the owning layer required touching jvm-libp2p (Noise + all key-gen) and UrlProtocol (Libp2pHostFactory). Publishing the new protocol then surfaced a second, independent upstream defect (relay fallback serialized behind a blocking direct dial — see §2.3) that had to be fixed in UrlProtocol too. And because every published UrlResolver was compiled against an old protocol constructor surface, the resolver had to be recompiled before ContainerNursery (the original flake target) could bump. So the dependency chain to green the original test is: jvm-libp2p → UrlProtocol → UrlResolver → ContainerNursery.

Bottom line at write time: the two upstream libraries are DONE and published (jvm-libp2p snapshot-11, foundation.url:protocol:0.0.348), both their PRs green on kotlin.build. UrlResolver is in-flight (code + reviews done; blocked on a version-number collision + buildtest infrastructure flakiness). ContainerNursery is not yet started as a PR. Nothing is merged — all PRs deliberately left for user approval per the global "never merge without approval" rule.

2. What was found and done (the chain — evidence, not speculation)

2.1 jvm-libp2p — DONE, published, green

  • PR: https://github.com/CodexCoder21Organization/jvm-libp2p/pull/22 — branch nonblocking-entropy-sources, head f318ef25. kotlin.build (remote) GREEN (verified). OPEN (awaiting user merge).
  • What it does: adds internal fun nonBlockingSecureRandom() in libp2p/src/main/kotlin/io/libp2p/crypto/SecureRandom.kt (prefers SecureRandom.getInstance("NativePRNGNonBlocking"), falls back to SecureRandom() on NoSuchAlgorithmException). Applied to: the Noise static X25519 key (NoiseXXSecureChannel.localStaticPrivateKey25519), the Noise per-handshake ephemeral key (via the getFixedEphemeralKey().setPrivateKey(...) injection hook — a review found the ephemeral path still blocked through noise-java's process-static SecureRandom, this closes it), and all key-gen defaults (Key.kt generateKeyPair, Ed25519, Secp256k1, Ecdsa, Rsa, TLSSecureChannel, SecIoNegotiator), and BouncyCastle ECDSA/secp256k1 normalization randomness.
  • Regression test: PoisonedDefaultSecureRandomTest — installs a JCA provider whose default SecureRandom throws from every entropy method in a child JVM, then exercises key-gen + Noise + secio; demonstrated to FAIL against the pre-fix code. Deterministic, no mocks, no reflection.
  • Published + content-verified: community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-11 (this develop-fork also rides snapshot-10's Yamux parent outbound-buffer backpressure — relevant to §2.3 ride-along). Build/release per the fork's FORK.md (Gradle → publishToMavenLocal -PmavenArtifactId=jvm-libp2p → cs publish).

2.2 UrlProtocol — entropy fix — DONE, green (part of the published 0.0.348)

  • PR: https://github.com/CodexCoder21Organization/UrlProtocol/pull/354 — branch nonblocking-identity-key-entropy, head 51bc1309 (rebased onto current main). bld-build + bld-all-tests GREEN; kotlin.build (remote) was FAILURE at write time but only on buildtest infrastructure (see §2.5), not code. OPEN.
  • What it does: Libp2pHostFactory.sharedKeyGenSecureRandom → NativePRNGNonBlocking (with fallback). Its regression test (tests/testHostIdentityKeyGenerationAvoidsBlockingDefaultSecureRandom.kts) is a poisoned-provider child-JVM test whose Java source is embedded as a local val inside the .kts. Why embedded, verified the hard way: the remote kotlin.build runner materializes only the tests/*.kts scripts themselves, NOT a sibling tests/<name>/ support directory — a sibling .java file caused a real remote-only failure (Child-JVM support source does not exist at /root/tests/...). kompile script validation also rejects top-level properties and return-typed helper functions, so the source lives as a local variable and is written to a temp dir + compiled with ToolProvider.getSystemJavaCompiler() at test runtime.

2.3 UrlProtocol — relay direct/relay race — DONE, green (also in 0.0.348); a SECOND, independent upstream defect

  • PR: https://github.com/CodexCoder21Organization/UrlProtocol/pull/356 — branch relay-direct-race, head f5a36c84, STACKED on #354 (its diff includes #354's commits until #354 merges). bld-build + bld-all-tests GREEN; kotlin.build FAILURE = same infra (§2.5). OPEN.
  • The defect (proven): UrlProtocol PR #348 ("Prefer direct mesh paths before relay fallback", shipped in protocol 0.0.341) made RelayService.sendViaRelay call sendDirectlyToRoutableTarget() sequentially-blocking before consulting the explicitly-selected relay. A caller that orchestrates its own direct-vs-relay race (UrlResolver does) then gets no latency benefit — the "relay" arm silently waits out the full direct-connect TCP reachability timeout to an unreachable address. Downstream evidence: UrlResolver's stressTestRelayFallbackRacesDirectConnectionForNonNatPeer failed (iteration 1 took ~4064 ms remote / ~2028 ms local against its 2000 ms contract).
  • The fix: sendViaRelay now races the routable-direct attempt against the selected relay on a bounded, cancellable, tracked executor (inFlightRaceDials, cap 8); commits to one path before constructing/transmitting the request (exactly-once, no double-send); closed-state guards so a post-close() winner fails descriptively instead of transmitting; close() cancels all in-flight races and drains bounded. The dial-reachability budget is a class-body property RelayConfig.reachabilityTimeoutMs (default 2000). Critical compat lesson: an earlier revision made it a primary-constructor parameter, which regenerated the RelayConfig data class's synthetic default-args constructor and broke binary compatibility with resolver ≤ 0.0.628 — caught by testUrlResolver0600CanConstructAgainstCurrentProtocol on bld-all-tests. Moving it to the class body (does not participate in constructor signature / equals / hashCode / copy) restored compat.
  • Tests: deterministic tests/testSendViaRelayDoesNotBlockOnUnreachableDirectAddress.kts (hangs the direct dial with a loopback backlog-exhausted ServerSocket, not a TEST-NET IP that a different CI network might fast-reject), testSendViaRelayDirectRaceWinnerDeliversExactlyOnce.kts, testSendViaRelayBothPathsFailCleanly.kts. The three #348 tests (testPublicNatPairUsesDirectRegisteredConnectionWithZeroRelayHops, testBothNatPairFallsBackToRelay, testRelayPathMigratesOffRelayWhenDirectRegistrationAppears) still pass unchanged.

2.4 The published protocol coordinate — USE 0.0.348 (version saga)

foundation.url:protocol:0.0.348 is published + content-verified and is the reunification coordinate carrying, on one line: the entropy fix (§2.2), the relay race (§2.3), AND main's 0.0.342 inbound-JSON parse-footprint bound + 0.0.343 legacy-constructor bridge. Do not pin 0.0.344/0.0.345/0.0.347:

  • 0.0.344 / 0.0.345 — published from a parallel branch line cut before 0.0.342/0.0.343 merged, so they lack the parse bound + bridge.
  • 0.0.347 — published, but its RelayConfig used the constructor-parameter form → broke binary compat with resolver ≤ 0.0.628 (§2.3). Superseded.
  • 0.0.348 — same content as 0.0.347 with reachabilityTimeoutMs in the class body, compat restored. This is the one. Verify contents (Reusable knowledge) before relying on it.

2.5 The buildtest infrastructure flakes (NOT code failures)

At write time all three green-code PRs (#354, #356, #755) showed kotlin.build (remote) = FAILURE, but every failure was buildtest droplet infrastructure, confirmed from the check-run summaries:

  • Build failed: 0 passed, 0 failed, N total — the DigitalOcean create-429 window-drain signature (no droplet ever provisioned; the retry budget drained). Self-heals when the DO rate-limit window rolls (~25 min); re-fire the check-suite then.
  • Build infrastructure error … SJVM virtual method invocation timed out after 120s: BuildTestServiceClientImpl.getTestResults and … error during resume — orchestrator/SJVM RPC interruption mid-build.
  • One resolver run failed 1069/1070 on testNetlabDynamicFailureRecovery — a known-flaky real-droplet test ([PEX-EMPTY] … connect: dial-failed = netlab droplet-peer transient), NOT our code. Do not treat these as code failures. Re-fire and persist.

3. Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | Branch / head | State @ write | Link | |---|---|---|---| | jvm-libp2p #22 | nonblocking-entropy-sources / f318ef25 | OPEN, kotlin.build GREEN. Artifact snapshot-11 published+verified | https://github.com/CodexCoder21Organization/jvm-libp2p/pull/22 | | UrlProtocol #354 (entropy) | nonblocking-identity-key-entropy / 51bc1309 | OPEN, GHA green, kotlin.build infra-FAILURE | https://github.com/CodexCoder21Organization/UrlProtocol/pull/354 | | UrlProtocol #356 (relay race, stacked on #354) | relay-direct-race / f5a36c84 | OPEN, GHA green, kotlin.build infra-FAILURE. Artifact 0.0.348 published+verified | https://github.com/CodexCoder21Organization/UrlProtocol/pull/356 | | UrlResolver #755 | bump-protocol-0.0.344-nonblocking-entropy / d629df13 | OPEN, mergeStateStatus=DIRTY (main moved). Version collision + infra flake (§4) | https://github.com/CodexCoder21Organization/UrlResolver/pull/755 | | ContainerNursery | (branch exists locally, no PR) | NOT STARTED as a PR | — |

Published/verified artifacts: community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-11; foundation.url:protocol:0.0.348.

Unsaved work: all three fix branches (jvm-libp2p, UrlProtocol ×2, UrlResolver) are fully pushed, 0 unpushed commits. The ContainerNursery checkout at /code/workspace/ContainerNursery (branch bump-protocol-0.0.344-nonblocking-entropy) has an uncommitted, superseded local diff from the first codex attempt (build.kts pins protocol 0.0.344 + resolver still 0.0.600; codex correctly stopped on the resolver-incompatibility condition and never committed). It was intentionally NOT pushed — the pins are wrong/superseded; it must be redone against the final versions (§4.5), not resumed. Nothing else is unsaved.

4. Next steps (for the next person)

Re-verify current state first (§ banner). The two upstream libs are done; the remaining work is UrlResolver → ContainerNursery, then user-merge all five.

4.1 — UrlResolver PR #755: resolve the version collision, then green + publish.

  • Blocker A (version collision): foundation.url:resolver:0.0.629 is now published on kotlin.directory but it pins protocol 0.0.337 + jvm-libp2p snapshot-10 — i.e. STALE content from another automation/main, NOT this PR's fix (verified: its .pom shows those versions). The PR's own coordinate is 0.0.629, now collided. Bump the resolver coordinate in build.kts to 0.0.630 (first curl its .pom → expect 401/free; if taken, take the next free N and note it).
  • Blocker B (DIRTY): PR is mergeStateStatus=DIRTY — git fetch origin && git rebase origin/main on branch bump-protocol-0.0.344-nonblocking-entropy, re-resolve any build.kts version-history conflicts, git push --force-with-lease.
  • Confirm the pins are protocol 0.0.348 + jvm-libp2p snapshot-11 (they were, pre-rebase). The code adaptation (named-arg UrlProtocol2 constructor call at UrlResolver.kt ~975, LanDiscoveryConfig(enabled=false) on the internal delegate to preserve 0.0.337 behavior) is already done and reviewed.
  • Get kotlin.build (remote) green — persist through the infra flakes (§2.5): re-fire the check-suite (gh api --method POST repos/CodexCoder21Organization/UrlResolver/check-suites/<suite_id>/rerequest, suite via gh api repos/.../commits/<sha>/check-suites --jq '.check_suites[]|select(.app.slug=="kotlin-build-ci-test")') after the DO window rolls; if testNetlabDynamicFailureRecovery alone fails, that's the known netlab-droplet flake — re-fire.
  • Publish foundation.url:resolver:0.0.630 via the publish-maven-artifact flow using /home/christian/bin/cs (see Reusable knowledge). Verify the published JAR resolves and pins protocol:0.0.348.

4.2 — ContainerNursery PR (the original flake target). Fresh checkout / discard the stale local diff. In build.kts: foundation.url:protocol (line ~175, currently 0.0.344) → 0.0.348; foundation.url:resolver (line ~252, currently 0.0.600) → 0.0.630 (from 4.1); community.kotlin.libp2p:jvm-libp2p (line ~288) is already snapshot-11. Bump matching tests/*.kts @file:WithArtifact pins for those exact artifacts only (never blanket-bump — a prior incident broke ~77 tests). Run scripts/test.bash --test testUrlFacade_autoStartPersistentRouteAndRestartResilience locally (the flake target — should now pass reliably; on a modern-kernel dev box it can't reproduce the block, so also rely on the remote run). Open the PR; drive kotlin.build (remote) green.

4.3 — Merge (needs the user). All five PRs are OPEN and were deliberately not merged per the global rule. Once each is green, the user merges: jvm-libp2p #22, UrlProtocol #354 then #356 (stacked order), UrlResolver #755, ContainerNursery (new). Note #356 is stacked on #354.

4.4 — Close out the source handoff. Once the ContainerNursery flake target is verified green end-to-end and the chain is merged, delete handoffs/2026-06-30-cn-477-httpclient-reuse-flake-fixed-and-urlfacade-p2p-entropy-flake.md and this handoff (per the create-handoff completion procedure).

5. Reusable / operational knowledge

  • Working coursier is /home/christian/bin/cs. /usr/local/bin/coursier is aarch64 (wrong arch on this x86_64 host) — Exec format error. The watch-build/build-watchman launcher resolves the broken one; a challenge was logged. Publish artifacts with cs launch community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.5 -r https://kotlin.directory -- --workspace <repo> --buildrule <pkg>.buildMaven --api-url https://api.kotlin.directory/upload --force.
  • A higher Maven version number is NOT a superset of fixes here — parallel branch lines (main vs this chain) published overlapping/interleaved numbers, and external automation snipes numbers mid-work. ALWAYS (a) curl the target .pom returns 401 before claiming a version, and (b) re-verify the published JAR contents, e.g. for protocol: unzip and check foundation/url/protocol/Libp2pHostFactoryKt$sharedKeyGenSecureRandom$2.class contains NativePRNGNonBlocking, and RelayService.class contains inFlightRaceDials; check RelayConfig.class has getReachabilityTimeoutMs/setReachabilityTimeoutMs (body-property form = compat-safe).
  • kompile stale-cache gotchas before re-running tests: rm -rf ~/.aibuildcaches/*<Repo>* ~/.cache/coursier/v1/bldbinary/repository/foundation.url*. A "succeeded in 3ms" build is a stale ~/.aibuildcaches hit.
  • Re-firing CI: GHA workflows bld-build / bld-all-tests fire only on push / pull_request / merge_group; a PR in CONFLICTING/DIRTY state blocks them — rebase to re-fire. kotlin.build (remote) is a separate GitHub App (kotlin-build-ci-test); re-fire via check-suite rerequest (POST repos/.../check-suites/<id>/rerequest), NOT check-run rerequest (which no-ops). A close/reopen of the PR also re-fires the GHA pull_request workflows without disturbing the sha-keyed kotlin.build run.
  • kompile test-script constraints: a tests/*.kts may only contain top-level functions with block bodies and no declared return types; no top-level val/property, no return-typed helper. The remote runner materializes only the .kts files, not sibling tests/<name>/ dirs — embed any support source inside the script (write to a temp dir at runtime) rather than shipping a sibling file.
  • /code/workspace can become root-owned/unwritable — fix with rmdir /code/workspace && mkdir /code/workspace (do not fall back to scratchpad).
  • Implementation was delegated to codex gpt-5.6-sol per the global bug-fixing workflow (define problem → codex writes failing test → codex fixes → codex reviews). One codex run stalled ~2h48m (drifted into read-only analysis of kompile internals); it was hard-killed and its verified on-disk work taken over directly. Don't put pkill -f <pattern> in codex prompts (self-kill footgun).
  • Adversarial-review discipline paid off repeatedly here: the Noise ephemeral key path (missed by the first jvm-libp2p pass), the RelayConfig binary-compat break, and the sibling-.java-file remote materialization failure were all caught by review/CI, not by the initial implementation. Keep the two-reviewer (test-adequacy + adversarial) gate on each PR in this chain.