Handoff: buildtest 0–1-concurrency — relay CPU-starvation (partially fixed, box RE-SATURATED) + kompile per-file resolution serialization (fix green, unmerged)

Written 2026-07-12 ~17:10 UTC

⚠️ RE-VERIFY BANNER — everything below is a write-time snapshot and WILL be stale. Merge queues shuffle, CI re-runs, branches get force-pushed, and — critically here — the production box's load re-climbed after an apparent relief, so host state changes hour to hour. Before acting, re-verify:

• PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Box (198.199.106.165): ssh -p 23 root@198.199.106.165 'cat /proc/loadavg; ss -tn state syn-sent | wc -l' and a jstack of the bin/container-nursery.jar PID (grep ByteBufQueue.take and io.libp2p...Yamux).
• A fresh query always wins over this document.

Mission summary

Original user request (near-verbatim): "tests are progressing at an alarmingly slow rate. There should be 4 concurrent tests per droplet and ~a droplet per 25 tests up to 10 droplets. Instead we see builds like https://buildtest.kotlin.build/run?id=c5bf3f99 with only zero or one tests in progress at a time... understand and fix."

It expanded, at user direction, to: fix both identified root causes (A and B); run a recurring status loop (now canceled); land the yamux fix cascade to production, confirming before the prod redeploy; ensure all builds were on latest upstream before deploying; and fix the cascade PRs' CI → green → merge. Then: create this handoff.

What was found and done (the chain)

1. Diagnosis of run c5bf3f99 (LambdaServer PR#62), forensically proven — two compounding defects:

• (A) Relay CPU-starvation of the shared box. ContainerNursery (198.199.106.165) hosts BOTH the libp2p url:// relay AND kotlin.directory (the Maven repo buildtest workers resolve from). The relay's yamux send-buffer drain was O(n²) — io/libp2p/etc/util/netty/ByteBufQueue.kt used an ArrayList with removeAt(0) inside the drain loop. Relaying ~140 worker droplets' traffic pinned the box: load ~58 on 8 real cores (nproc=8; the "1vcpu" slug is a lie), 2752 ESTABLISHED + 1127 SYN-SENT sockets, TLS handshakes 11–19s, TLS_QUEUE_WAIT in CN logs. Evidence: ss counts, jstack, CN logs.
• (B) kompile per-file classpath resolution serialized. Live jstack of BOTH buildtest shard runners (buildtest-runner-0.0.53) showed the sole main thread PARKED (identical stack + identical cpu across snapshots) in kompile.WorkspaceKt.runTests → processOneFile → unifiedResolve → coursierapi.Fetch.fetch → Await.result, executed inside ConcurrentHashMap.computeIfAbsent. Workers were ~90% idle (c5bf3f99: 36m wall / 43s CPU); each test computes ~2s but the gap between tests was 20–260s. So kompile's intended 4-way parallel compile collapsed to one serialized, network-bound resolve at a time. Not droplet count, not worker capacity, not test logic.

2. Fix A (yamux) — built, cascaded, DEPLOYED — but NOT sufficient on its own (see step 6). The O(n²) fix already existed as jvm-libp2p PR #23 (swap to ArrayDeque, O(1) head removal; TDD repro 42.4s→0.166s). Merged it, published community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-12, then cascaded the pin: UrlProtocol 0.0.349 (#358) → UrlResolver 0.0.635 (#764) → ContainerNursery 0.0.143 (#523) → rebuilt the CN fatjar (all on latest upstream main, verified unchanged; ByteBufQueue.class in the jar confirmed to reference ArrayDeque) → redeployed to prod with user approval.

3. Deploy footgun (self-corrected). ContainerNurseryProductionRemoteRestartCli 0.0.5 forces -Xmx128m, which OOM'd the real CN workload within minutes (heap dump preserved), a ~1-minute blip (kotlin.directory :443 connection-refused). A per-minute watchdog cron (* * * * * flock -n /tmp/cn-watchdog.lock /root/cn-watchdog.sh) restarted CN at -Xmx1024m using the same uploaded jar (/root/ContainerNursery/bin/container-nursery.jar, sha256 05d533184c05810d049b7101dcebe55b682376c183ea5f5e03a1a81b40070ea2). Lesson: do NOT deploy prod CN via RestartCli 0.0.5 (its 128m is unrealistic; the box needs ~1024m). The watchdog is the authoritative restart mechanism.

4. Immediately post-deploy the box looked relieved — load fell to ~3–11, SYN-SENT ~90, TLS_QUEUE_WAIT 0, ByteBufQueue.take no longer a hotspot. I reported "relief holding." This was premature / a post-restart lull.

5. Fix B (kompile per-file resolution) — green, unmerged. kompile-core PR #214 "Share test classpath resolution across compile workers" (branch fix/shared-test-classpath-resolution, head 5082571). A first codex pass introduced an effect-leak regression (NotificationEffect emitted on a handler-less worker thread → stdout leak, caught by the existing testNoLeakWhenWorkerThreadProcessesFileWithLocalBuildRule); I fixed it: process the FIRST test file on the caller/main thread (which holds the Effective{} handler, so its shared-resolution + local-build-rule effects are handled, warming the memo), then release a CountDownLatch-gated 4-worker pool to compile the rest in parallel reusing the memo. Verified leak test green locally; PR CI all-green.

6. ⚠️ CRITICAL, re-verified at 17:07 UTC: the box RE-SATURATED. load 123 / 102 / 71 (climbing, higher than the original 58), SYN-SENT 633, ESTAB 2212. jstack of CN (pid was 3680054, -Xmx1024m, up ~50min, NOT crash-looping) shows ByteBufQueue.take is NO LONGER a hotspot (1 frame, O(1) — the yamux fix works), but CPU is now spread across io.libp2p.mux.yamux.YamuxHandler frame processing (channelRead/handleFrameRead/handleDataRead, ~10 threads), io.libp2p.security.noise.NoiseXXCodec.decode (crypto), and reflection (~6 frames — the deferred UrlProtocol #300 per-send Class.getMethods() on the relay forward hot path). Conclusion (verified): the yamux O(n²) fix was NECESSARY but NOT SUFFICIENT. The relay is still saturated by sheer relayed-traffic volume (2200+ conns) plus the un-landed companion CPU sinks. Durable relief requires the deferred work below.

Relevant PRs / refs (write-time state — RE-VERIFY)

| PR / ref | State @ write time | Notes | |---|---|---| | jvm-libp2p #23 | MERGED | yamux O(n²)→ArrayDeque | | jvm-libp2p #24 | MERGED | version bump → snapshot-12 published | | UrlProtocol #358 | MERGED | pin snapshot-12; published foundation.url:protocol:0.0.349 | | UrlResolver #764 | OPEN, ms=BLOCKED | branch bump/protocol-0.0.349; build.kts libp2p pin snapshot-10→12; published foundation.url:resolver:0.0.635; CI settling | | ContainerNursery #523 | OPEN, ms=BLOCKED | branch bump/url-yamux-snapshot-12 (head 0f5f9a0, rebased on main); CN 0.0.143 fatjar built + DEPLOYED; test-file version pins bumped to 0.0.349/0.0.635/snapshot-12; CI settling | | kompile-core #214 (Fix B) | OPEN, ms=CLEAN, all-green | branch fix/shared-test-classpath-resolution head 5082571; awaits user merge | | UrlResolver #762 | OPEN, ms=CLEAN (companion) | "Reduce gossip control-plane churn" — deferred, now mergeable | | UrlProtocol #300 | OPEN (companion) | removes per-send Class.getMethods() reflection on relay forward hot path — this is one of the current CPU sinks | | UrlProtocol #356, UrlResolver #758, UrlResolver #761 | OPEN (companions) | deferred CPU-reduction PRs |

Deployed artifact: CN bin/container-nursery.jar sha256 05d533184c05810d049b7101dcebe55b682376c183ea5f5e03a1a81b40070ea2 (contains ByteBufQueue=ArrayDeque). Local clones on branch: ~/workspace/{ContainerNursery,UrlResolver} (may not persist across sessions).

Next steps (ordered — re-verify current state FIRST)

0. Re-verify: box load + jstack CN for the current CPU sinks; the 6 PRs above; whether CN #523/#764 merged on their own. A background watcher I launched (cascade-finisher.sh) FAILED (exit 1) and is NOT running — do not rely on it.
1. The effort is NOT done: the box is saturated again (load ~121). Treat "the yamux fix relieved the box" as FALSE for durable purposes. The next real work is landing the deferred relay-CPU reductions so relief holds:
   • Land UrlProtocol #300 (reflection on relay hot path — confirmed in the 17:07 jstack) and UrlResolver #762 (gossip churn), then cascade-pin + rebuild + redeploy CN the same way (they were deferred only to avoid a ~100-min merge-queue stall; #762 is now CLEAN).
   • Do the genuinely-new "stop relaying worker JAR bootstrap through CN" work (the deepest fix; NO PR exists). NAT-only workers relay their 1.7–2.4 MB SJVM JAR through CN as base64'd 128 KiB chunks. Cleanest fix: pre-stage the SJVM bootstrap JAR in the buildtest/netlab droplet image so it's never fetched over url://. Reproduce first (two-NAT NetLab topology; assert a large __bytecode_request for a routable target does not hit the relay branch; counter: RelayService.relayForwardingCountsByNextHop()). See the memory note fixA-cn-relay-yamux-oom-mostly-already-in-pr23-2026-07-12 and the scout findings.
2. Merge Fix B kompile-core #214 (needs user approval per global rule). After merge, a kompile-cli publish is required for buildtest workers to actually pick up the shared-resolution fix — Fix B is inert in production until then.
3. Land the cascade reconciliation PRs CN #523 + UrlResolver #764. Their fixes are CORRECT (test-file version-pin bumps to match the cascaded deps; CN also needed a build.kts libp2p pin fix). They only reconcile main's pins (a future rebuild would otherwise revert the deployed pins). Both use merge queues that require green checks before enqueue; CI is slow BECAUSE the box is re-saturated (chicken-and-egg). Enqueue via GraphQL enqueuePullRequest once CLEAN; rebase-on-BEHIND if needed (git rebase origin/main && git push --force). Watch for the stale-archive dispatch race: a buildtest run started 4s after a push tested the pre-push code (reported an already-fixed version-assertion failure) — a fresh run (empty commit re-trigger) cleared it.
4. Verify durable relief the honest way: after (1), watch load over ≥30 min under real buildtest activity (not just the post-restart lull), and jstack CN to confirm no relay hotspot returns. Do NOT declare victory on a single low reading.

Reusable / operational knowledge

• Fabric daemon (8443) was DOWN this whole session (no auto-restart) → used SSH -p 23 root@198.199.106.165 directly for all box forensics (this is the sanctioned fallback when the daemon is down).
• Verify a CN deploy by jar marker, not intent: on-host sha256sum /root/ContainerNursery/bin/container-nursery.jar must equal the local build's sha, AND python3 -c 'import zipfile; print(b"ArrayDeque" in zipfile.ZipFile(jar).read("io/libp2p/etc/util/netty/ByteBufQueue.class"))' (xxd/unzip not installed; use python3 zipfile). The running CN can differ from the deployed jar if it crashed and the watchdog restarted it — check the actual process's loaded jar + -Xmx.
• Prod CN deploy: the per-minute cn-watchdog.sh cron restarts CN at -Xmx1024m. RestartCli 0.0.5 forces -Xmx128m which OOMs → do not use it for prod; rely on/adjust the watchdog, or fix the launch heap.
• codex gpt-5.6-sol over-runs: it repeatedly over-verified locally (leak fix ~2h) and once got stuck waiting on a slow merge queue. Give TIGHT briefs ("do X, then STOP"), and monitor; be ready to TaskStop and take over the finish. Launch with codex exec --dangerously-bypass-approvals-and-sandbox -m gpt-5.6-sol --skip-git-repo-check -C <dir> -o <last.txt> "$(cat brief.md)" </dev/null > log 2>&1 (the </dev/null is mandatory or it stalls).
• Merge queues: UrlProtocol/UrlResolver/CN(#523) and PlanRepository all use merge queues with auto-merge OFF; gh pr merge --merge/--auto is rejected. Enqueue via gh api graphql -f query='mutation($pr:ID!){ enqueuePullRequest(input:{pullRequestId:$pr}){ mergeQueueEntry{position state} } }' -f pr=<nodeId>; the queue only accepts a PR whose required checks already pass.
• jvm-libp2p publish (FORK.md "Cut a release"): bump version in build.gradle.kts to 1.3.0-codexcoder21-snapshot-N (confirm N free via curl the .pom → non-200), land on develop, ./gradlew :libp2p:publishToMavenLocal -PmavenArtifactId=jvm-libp2p, then publish jar+pom via community.kotlin.maven.artifact.publishing CLI to https://api.kotlin.directory/upload (409 = already published). quic/tls SSL_CTX + parallel BindException test failures are known-environmental.
• Related memory: buildtest-slow-concurrency-coursier-resolve-on-cn-relay-starvation-2026-07-12, fixA-cn-relay-yamux-oom-mostly-already-in-pr23-2026-07-12, cn-tls-slowness-relay-cpu-starvation-2026-07-12.