Handoff: ContainerNursery #477 kotlin.build red — HttpFacade per-request-HttpClient flake fixed (merged); a separate pre-existing URL-facade libp2p entropy flake root-caused but not yet fixed

Written: 2026-06-30 (authored/paused). Workstream dated 2026-06-30.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below was true only at write time. PRs merge, branches get force-pushed, CI re-runs, the buildtest droplet restarts, and flakes fire intermittently. Re-verify before you act on anything here. Authoritative checks:

• PRs: gh pr view <num> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid
• kotlin.build run: gh api repos/CodexCoder21Organization/ContainerNursery/commits/<sha>/check-runs --jq '.check_runs[] | select(.name=="kotlin.build (remote)") | {status,conclusion,output:.output.summary,details_url}'
• buildtest run page: curl -s "https://buildtest.kotlin.build/run?id=<id>"

A fresh query always wins over this document.

---

1. Mission summary

Original user request (verbatim):

"Use your flakey test skills to resolve https://github.com/CodexCoder21Organization/ContainerNursery/pull/477/checks?check_run_id=84176919609"

That check was kotlin.build (remote) on ContainerNursery PR #477 (the "Round 2 extreme-condition bug fixes" PR: child-PID-reuse kill, TCP lingering-client leak, metrics CME). The failing run (c6dca45b) reported 287 passed, 1 failed — the one failure was testHttpFacade_xForwardedHeadersReachBackendOverWire with SocketTimeoutException: Read timed out.

The effort then expanded in two ways:

1. Fixing that flake required landing a sibling PR (#474) and rebasing #477 onto it.
2. Verifying the fix surfaced a cascade of kotlin.build infrastructure flakes and then a second, unrelated pre-existing test flake (testUrlFacade_autoStartPersistentRouteAndRestartResilience), which the user explicitly asked to fix ("Also fix the P2P flake now"). That second fix is root-caused but not implemented — its approach was awaiting a user decision when this handoff was written.

Bottom line at write time: the original task is DONE — #477 is MERGED with kotlin.build (remote) GREEN. The URL-facade libp2p entropy flake is still live (pre-existing, unrelated to #477, unfixed) and will keep firing intermittently on the contended droplet.

---

2. What was found and done (the chain — evidence, not speculation)

2a. Flake #1 (the one asked about): testHttpFacade_xForwardedHeadersReachBackendOverWire — FIXED & MERGED

• Symptom: client opens a raw socket to the HTTP facade, sends GET … Connection: close, then readBytes() (read-to-EOF) at tests/testHttpFacade_xForwardedHeadersReachBackendOverWire.kts:160 — timed out after the 10 s soTimeout.
• Root cause (proven): the request path constructs a fresh HttpClient(CIO) per request — real production behaviour, DockerBackedContainer.handle (src/main/kotlin/org/example/docker/DockerBackedContainer.kt:74), and mirrored by the test's fake container. On the contended 2-core droplet the one-time CIO-engine cold-start (~7 s in the failed run's thread timeline) ate most of the 10 s read budget → timeout. Locally in a quiet env it passes (I ran it: scripts/test.bash --test testHttpFacade_xForwardedHeadersReachBackendOverWire → 1/1 PASSED). It is the only test doing a real per-request CIO forward, which is why it was the only one flaking.
• The fix already existed: PR #474 "Reuse one HttpClient per container…" — its commit message documents this exact failure and fixes it (one lazily-created reused HttpClient per container, closed at shutdown/kill; the test's fake rebuilt to construct its client once at setup, outside the read window; adds testHttpFacade_concurrentForwardsDoNotStallSharingOneClient). #474 and #477 touched disjoint files, so the fix comes in cleanly by rebasing.
• Actions taken:
  1. Waited for #474 to leave the merge queue → it MERGED to main at 2026-06-30T01:02:10Z (merge commit fe1061246).
  2. Rebased #477 (extreme-condition-bugfixes-2) onto the new main — conflict-free — bringing fe1061246 in as an ancestor. Verified the flaky test still passes locally against the rebased tree.
  3. Force-pushed (2091bfb → 12d97e6, later amended to 1a79fd5).

2b. kotlin.build infrastructure flakes hit during verification (NOT test failures, NOT #477's code)

These are the recurring buildtest/kotlin-build-ci infra problems (documented across prior handoffs). Each cost a ~20 min cycle:

1. Run on 12d97e6 (6612c3f7) — Build infrastructure error: SandboxException: SJVM host-side RPC interrupted during method=getBuildRun … InterruptedException at BuildRunner.pollUntilComplete(BuildRunner.kt:1367). The buildMaven() build rule showed FAILED at 0 ms and no tests ran. Orchestrator interrupted/restarted mid-build.
2. Amend-push (1a79fd5) — the kotlin-build-ci-test app created an (empty) check-suite but never a check-run → the push webhook was dropped (the receiver races GitHub's 10 s delivery limit). GHA checks ran fine; kotlin.build simply never started.
3. bld-all-tests (GitHub Actions) flaked once with a kompile test-runner internal NPE Cannot invoke "Object.toString()" because "this.context" is null — cleared by re-running (gh run rerun <id> --failed).

2c. Flake #2 (surfaced during verification): testUrlFacade_autoStartPersistentRouteAndRestartResilience — ROOT-CAUSED, NOT FIXED

• Once the build actually ran (b83f5536): 287 passed, 1 failed, and the HttpFacade flake (2a) PASSED — confirming the #474 fix works. The new failure was testUrlFacade_autoStartPersistentRouteAndRestartResilience: timed out after 30000 ms (actual 38127 ms).
• This is pre-existing and unrelated to #477. The test is on main (added before #472); #477 does not touch the URL facade or the P2P stack. Verified: git log origin/main --oneline -1 -- tests/testUrlFacade_autoStartPersistentRouteAndRestartResilience.kts; and #477's diff touches only EventStore/NurseryPidFile/SystemMetricsProjection/TcpFacade + 3 new tests.
• Root cause (proven from code + the failed run's console): /dev/random entropy starvation in libp2p host crypto. In the failed console there is a clean ~10 s gap between [UrlFacade] Saved persistent config (…33.416) and [UrlFacade] Local peer ID: (…43.466) — no interleaved log lines, the signature of a single blocking syscall. The chain:
  1. Test does UrlProtocol2(bootstrapPeers = emptyList()) with no key seed → persistentKey = null (UrlResolver.kt:817).
  2. Libp2pHostFactory.createHost(keyPair = null) (UrlResolver.kt:3571) → identity { keyPair ?: generateRandomEd25519Key() } (UrlProtocol Libp2pHostFactory.kt:619).
  3. generateRandomEd25519Key() → generateKeyPair(ED25519, 0, sharedKeyGenSecureRandom) (Libp2pHostFactory.kt:568).
  4. sharedKeyGenSecureRandom is a default new SecureRandom() (Libp2pHostFactory.kt:470) = NativePRNG on Linux. Its own code comment (lines 460-464) admits it "can block on /dev/random for seconds" on entropy-starved hosts. The shared/eager-seed optimisation only avoids repeated blocking; the first seeding in a fresh test JVM still blocks.
  5. A second JVM-wide entropy consumer on the host-build path: the Noise static X25519 key NoiseXXSecureChannel.localStaticPrivateKey25519 = ByteArray(32).also { Noise.random(it) } (jvm-libp2p libp2p/src/main/kotlin/io/libp2p/security/noise/NoiseXXSecureChannel.kt:60); Noise.random is the external noise-java lib's internal SecureRandom.
  6. The build droplet is kernel 5.4 with a tiny entropy pool — documented verbatim in ContainerNursery/scripts/test.bash:34-42, which applies -Djava.security.egd=file:/dev/./urandom only to the kompile JVM. That flag does not reach the separate test JVM kompile spawns via executeTestInSeparateJvm, so the test JVM seeds from blocking /dev/random. The first seeding blocks ~9-15 s (= the observed 10 s gap). The test starts two hosts → past the 30 s budget under contention.
• Could NOT reproduce locally, and this is understood, not a gap in understanding: this dev box is kernel 6.12, where /dev/random is non-blocking, so the block cannot fire. I verified that pointing securerandom.source at a blocking FIFO does not cleanly discriminate NativePRNG from NativePRNGNonBlocking (both stall Sun-provider SeedGenerator init). The discriminating behaviour only exists against a real blocking /dev/random. NativePRNGNonBlocking is available on this JDK (java 21.0.11, SUN provider) and reads /dev/urandom only.

2d. How #477 actually went green and merged

After I paused (pending the flake-approach decision), kotlin.build (remote) was re-run and PASSED (6d57d979, 11m37s) — the intermittent URL flake simply did not fire that time — and #477 MERGED at 2026-06-30T06:10:34Z (merge commit 4ae4899f). So the original task completed. The URL entropy flake remains latent.

---

3. Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | State @ write | Link | |---|---|---| | ContainerNursery #477 (extreme-condition bugfixes 2) | MERGED 2026-06-30T06:10:34Z, merge commit 4ae4899f; all checks incl. kotlin.build (remote) GREEN (run 6d57d979) | https://github.com/CodexCoder21Organization/ContainerNursery/pull/477 | | ContainerNursery #474 (reuse one HttpClient per container) | MERGED 2026-06-30T01:02:10Z, merge commit fe1061246 | https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 | | Original failing run (HttpFacade flake) | 287 pass / 1 fail | https://buildtest.kotlin.build/run?id=c6dca45b | | Infra-error run (getBuildRun InterruptedException) | build rule FAILED 0ms, 0 tests ran | https://buildtest.kotlin.build/run?id=6612c3f7 | | URL-facade flake run | 287 pass / 1 fail (testUrlFacade_autoStartPersistentRouteAndRestartResilience 30 s timeout) | https://buildtest.kotlin.build/run?id=b83f5536 | | Final green run that let #477 merge | PASS | https://buildtest.kotlin.build/run?id=6d57d979 |

No uncommitted code changes exist in any repo. The only code change made (the #474 rebase into #477) is merged. The entropy fix (§4) was intentionally not written as code — its approach was pending a user decision. Clones inspected during investigation (/code/UrlProtocol @ 4d1c220 main, /code/jvm-libp2p @ dc4a571 develop, /code/UrlResolver @ 783a93af main) are unmodified.

Dependency versions relevant to the fix (verify current before acting):

• ContainerNursery build.kts: foundation.url:protocol:0.0.305 (line ~142), foundation.url:resolver:0.0.557 (line ~219). Tests pin community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-8 (some files still pin -snapshot-5 → the harmless-but-noisy WithArtifactVersionMismatchWarningEffect seen in local builds).
• UrlResolver main build.kts: foundation.url:protocol:0.0.320, jvm-libp2p:1.3.0-codexcoder21-snapshot-6.

---

4. Next steps (for the next person)

Re-verify current state first (the URL flake is intermittent — confirm it is still firing before investing). Search recent kotlin.build failures for testUrlFacade_autoStartPersistentRouteAndRestartResilience timing out, or any test whose console shows a multi-second gap before [UrlFacade] Local peer ID:.

The original ask (#477 kotlin.build green) is complete/merged — no action needed there. The remaining work is the URL-facade libp2p entropy flake, whose fix approach needs a user decision (I asked; it was not yet answered). The three candidate approaches, with tradeoffs:

A. Systemic entropy fix in the build harness (recommended — highest leverage, true root cause). Make the spawned test JVMs on the remote builder use /dev/urandom. This fixes this flake and every other entropy-sensitive test (TLS handshakes, all libp2p/Noise tests) org-wide, in one place.

• First determine where the gap is: does the remote kotlin.build builder set -Djava.security.egd=file:/dev/./urandom on its kompile JVM, and does kompile forward it (or system properties) to the per-test JVM in executeTestInSeparateJvm? Local scripts/test.bash:42 sets it on the kompile JVM only.
• If kompile does not forward it: the fix is in kompile-cli (default spawned test JVMs to -Djava.security.egd=file:/dev/./urandom). If it does forward it but the remote builder doesn't set it: the fix is in the kotlin-build-ci / buildtest launcher config. Either is shared CI infra — treat with care; use investigate-broken-service / the relevant infra skill.
• This is arguably the most-root fix (the defect is "test JVMs seed from blocking /dev/random"), but it touches shared infrastructure, so confirm scope/ownership before changing.

B. Upstream crypto fix ("fix at the owning layer" per global rules — but whack-a-mole + multi-repo). Make the key-gen entropy sources non-blocking:

• UrlProtocol Libp2pHostFactory.kt:470 — change sharedKeyGenSecureRandom from java.security.SecureRandom() to SecureRandom.getInstance("NativePRNGNonBlocking") (with a try/catch NoSuchAlgorithmException fallback to SecureRandom() for non-Unix). This covers the identity key.
• jvm-libp2p NoiseXXSecureChannel.kt:60 — replace Noise.random(it) with a non-blocking SecureRandom.getInstance("NativePRNGNonBlocking").nextBytes(it) (same fallback). This covers the Noise static key. (Consider also Key.kt:127, Ed25519.kt:67, TLSSecureChannel.kt:123 defaults for full hardening.)
• Caveat proven during investigation: because the JVM's first SecureRandom seeding is the expensive one, fixing only one site may just move the block to the next site — you likely must fix both (identity + Noise) to eliminate the host-start block. This is why (A) is cleaner.
• Publish chain: fix jvm-libp2p → publish 1.3.0-codexcoder21-snapshot-9 (Gradle/Cloudsmith flow); fix + bump UrlProtocol → publish next foundation.url:protocol (0.0.321) with publish-maven-artifact; then bump ContainerNursery's build.kts protocol pin (and test jvm-libp2p pins). NOTE: the protocol change is internal/binary-compatible, so you can likely bump only CN's foundation.url:protocol pin to the new version and keep resolver:0.0.557 (Coursier mediates one protocol version; resolver picks up the fixed Libp2pHostFactory at runtime) — verify this on kotlin.build.
• Validation is remote-only (can't reproduce locally on kernel 6.12): each iteration is a ~20 min kotlin.build cycle on a builder that is itself flaky. Budget for that.
• Regression test idea (deterministic, environment-independent): assert the key-gen SecureRandom's algorithm is a non-/dev/random source (e.g. NativePRNGNonBlocking) rather than the default NativePRNG. Fails on current code, passes after. (Testing .getAlgorithm() is a reasonable proxy since the fix's entire purpose is entropy-source selection; keep it package-level, don't add public API.)

C. Do nothing structural / accept intermittency. Not acceptable under the "NEVER accept kotlin.build (remote) red" rule — but note the flake is low-frequency (it did not fire on the passing run that merged #477). If deprioritised, at minimum leave the root cause documented (this handoff / a memory entry) so the next person who sees it doesn't re-derive it.

Decision needed from the user: which of A / B to pursue (I recommended A, with B as the by-the-book upstream alternative). Confirm before committing to the multi-repo publish chain in B.

---

5. Reusable / operational knowledge (so you don't rediscover it)

• Re-triggering kotlin.build (remote) when it's dropped/failed on infra:
  • The check is owned by GitHub App kotlin-build-ci-test (not a GHA workflow), fired on push / check_suite webhooks. A check_run rerequest does NOT re-trigger it (I tried — returns {} but nothing starts).
  • A check_suite rerequest DOES work when an (empty) suite exists: gh api --method POST repos/CodexCoder21Organization/ContainerNursery/check-suites/<suite_id>/rerequest (get the suite via gh api repos/.../commits/<sha>/check-suites --jq '.check_suites[] | select(.app.slug=="kotlin-build-ci-test")'). This started run b83f5536.
  • If the suite/run is entirely missing, a new push (git commit --amend --no-edit && git push --force-with-lease) reliably re-fires the webhook (but may itself be dropped — the receiver races GitHub's 10 s limit).
• A queued kotlin-build-ci-test check-suite with no check-run for many minutes = dropped/slow webhook, not a wedged orchestrator. Confirm the orchestrator is alive by checking https://buildtest.kotlin.build/ for other repos currently TESTING/COMPLETED.
• Build infrastructure error … getBuildRun … InterruptedException at BuildRunner.pollUntilComplete = the orchestrator/SJVM was interrupted/restarted mid-build; buildMaven() shows FAILED at 0 ms and no tests run. It is an infra flake, not a code/test failure — re-trigger.
• Reading a buildtest failure without the WUI: gh api repos/CodexCoder21Organization/ContainerNursery/commits/<sha>/check-runs --jq '.check_runs[] | select(.name=="kotlin.build (remote)") | .output.summary' gives the failed-test list / infra error inline. The per-test console (with the [UrlFacade] timestamps that reveal the 10 s gap) is in the run HTML: curl -s "https://buildtest.kotlin.build/run?id=<id>" then grep around the test's data-test-name.
• git in this workspace: the origin HTTPS remote needs gh auth setup-git (raw git fetch/push otherwise errors could not read Username). Commit identity used here: codexcoder21 <laserjim@gmail.com> (matches the branch's existing commits). Rebase needs the identity set before git rebase or it aborts mid-replay.
• Local single-test run: cd ContainerNursery && scripts/test.bash --test <name>. First run compiles everything (slow). The WithArtifactVersionMismatchWarningEffect (jvm-libp2p snapshot-5 vs snapshot-8 across test files) is non-fatal noise locally and does not fail the remote builder — separate pre-existing cleanup, out of scope here.
• Entropy repro caveat: you can only reproduce the /dev/random block on a kernel where /dev/random actually blocks (e.g. the droplet's kernel 5.4). On modern kernels (≥5.6) it's non-blocking. A blocking-FIFO securerandom.source does not cleanly separate NativePRNG from NativePRNGNonBlocking (both stall Sun SeedGenerator init) — don't waste time on it.
• Why the fix must not be a downstream mitigation: per global rules ("Fix Upstream Library Bugs Upstream") and this repo's ethos, don't paper over the entropy block inside ContainerNursery (e.g. pre-seeding in the test, or a try/catch). Fix it in the harness (A) or the owning crypto libs (B).