Handoff: ContainerNursery slowness — root cause found, (c) reaper-idempotency deployed to prod, (b) super-linear fixes merged/queued but fleet-bump BLOCKED on a red protocol release train

Written: 2026-07-12 ~17:10 UTC (effort began 2026-07-12).

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below (PR states, merge-queue positions, CI results, deployed jar SHAs, running JVM pids) was a write-time snapshot and goes stale fast. Re-check before you act:

• PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Merge-queue entry: gh api graphql -f query='{ repository(owner:"CodexCoder21Organization",name:"<repo>"){ pullRequest(number:<n>){ mergeQueueEntry{ position state } } } }'
• Prod jars/JVMs on the CN host: use the HardwareControlFabric CLI (see Reusable knowledge) to run ls -la/sha256sum on the jar and pgrep -f <jar> for the JVM. A fresh query always wins over this document.

---

Mission summary

Original user request (verbatim intent): "The container nursery server, including even https://kotlin.directory/ - appears to be super slow right now. Something is clearly wrong. Perform a comprehensive forensic analysis. What is the root cause of the performance issues?"

It expanded, in order:

1. Forensic root-cause of CN / kotlin.directory slowness.
2. A second/third look at the HardwareControlFabric monitoring daemon (daemon.jar), which the user reported was restarted by its supervisor and "already GC-thrashing again" — suspected a pernicious leak.
3. Fix directives from the user: (b) "fix the O(N²) gossip fan-out and O(n²) ByteBufQueue/Base58 upstream in UrlProtocol2/jvm-libp2p"; (c) "add backoff/idempotency to the renew loop." Plus for the daemon: "do A and B" — A = make the relay stream payloads instead of materializing byte[]/JSON; B = analyze the OOM heap dump (leak vs burst).
4. The user set a HARD INVARIANT (save this): "we MUST NEVER decouple droplet management from webcron, since webcron callbacks is the only safety check we have to prevent runaway droplets. A droplet must never be started without a webcron that will remind us to kill it." (This is why the (c) fix is an atomic upsert, never a decouple — DODroplet PR 88 "and all ideas like it are absolutely wrong.")
5. The user authorized merge+deploy ("approved to merge and deploy as needed") and, when asked, chose "Full auto — bump + restart" for the (b) fleet bump. That answer assumed a clean, ready cascade — which it turned out NOT to be (see the blocker). The full-auto premise is now invalid; a decision is pending (see Next steps).

What was found and done (the chain)

1. Root cause (verified via host forensics, memory cn-tls-slowness-relay-cpu-starvation-2026-07-12): host CPU saturation, not memory/GC/quota. The CN box (slug says "1vcpu" but has 8 real cores — check nproc) was at load ~100, its cores burned by CN's libp2p relay relaying ~146 buildtest/netlab worker droplets' JAR-upload bootstrap over url://, amplified by super-linear code: yamux ByteBufQueue.take O(n²) and gossip service-announcement fan-out O(N²). Symptom was TLS appconnect 11–19s only (app + :80 fine) — i.e. the shared :443 handshake path was starved. The incident SELF-RESOLVED when the droplet fleet drained (kotlin.directory TLS recovered to 0.08–0.15s during the session). The (b) fixes are therefore preventive (stop the next storm from re-degrading CN), not fighting an active fire.

2. Daemon (A)+(B): heap-dump analysis concluded the fabric daemon.jar OOM was burst-materialization, not a leak (it recovers after the burst; supervisor restarts it). The (A) relay streaming/bounding fix is already merged in UrlProtocol main — RelayMessages.kt does chunked request/response (chunkSeq/chunkAck, streams next chunk on ack), BytecodeResponseStreamer.kt streams the base64 response "instead of a fully-materialized JSON String", and merged commits #298/#355/#352/#294 bound the footprint. It reaches prod only via the fleet bump (below).

3. (c) droplet reaper idempotency — MERGED + DEPLOYED to prod + VERIFIED. Replaces the old blind delete-then-create reaper timer with an atomic createOrUpdateTimerByName on the WebCron server, keyed by an account-scoped name, so every droplet always has exactly one enforcing kill-timer across retries, client-side DO timeouts, and overlapping generations. Three-PR chain (all MERGED): droplet #87, WebCronApi #18 (adds createOrUpdateTimerByName), WebCronApiServer #50 (implements POST /timers/by-name). Deployed server-first, then droplet (so the droplet's by-name calls never 404):

   • webcron jar webcron-url-server.jar sha256 397c1701… (76,101,718 bytes) → /root/ContainerNursery/apps/webcron-url-server.jar
   • droplet jar droplet-service-server.jar sha256 23f56b41… (57,693,896 bytes) → /root/ContainerNursery/apps/droplet-service-server.jar
   • Verified live at deploy (15:05Z) and re-verified 17:07Z: both jars in place with matching SHAs, both JVMs running. At relaunch the droplet log showed 71 [AutoTermination] Reconciled timer … via url://webcron/ for real live buildtest droplets, 0 real renew failures, 0 404s — the invariant holds and the new by-name endpoint is serving.
   • Fail-SAFE caveat (by design, reviewed): the NEW global recurring sweep job (a secondary net) does NOT register in prod because webcronserver/WebCronServiceClientImpl.listJobs():List SJVM-times-out after 30s under the many-timers load; the code bounds it at 20s and skips fail-closed ("SKIPPING sweep job registration to avoid creating a duplicate"). Per-droplet timers cover the invariant regardless, so this is reduced redundancy, not a hole. Fix direction: register the sweep via a by-name upsert (like the per-droplet timers) rather than listJobs-check-then-create, OR paginate/bound listJobs server-side.

4. (b) super-linear fixes — merged/queued, NOT yet in prod:

   • yamux O(n²): jvm-libp2p #23 MERGED (ArrayDeque instead of removeAt(0)-in-loop drain). Published as community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-12 (verified: resolves from https://kotlin.directory; jvm-libp2p default branch is develop, not main). UrlProtocol main already pins snapshot-12 (#358); published foundation.url:protocol:0.0.349 carries snapshot-12 (verified via coursier resolve).
   • gossip O(N²): UrlResolver #762 — coalesces duplicate announcements, commits "forwarded" state only after a successful send (fixed a real discovery-loss bug), applies withdrawal-filtering to forwarding. Its own CI is green on its base (protocol 0.0.337). At write time it is OPEN, CLEAN, re-enqueued (pos 5, QUEUED) — it will merge to main autonomously (no prod impact from the merge itself).

5. THE BLOCKER (verified from CI logs). Getting the yamux fix into prod needs UrlResolver to pin a protocol version carrying snapshot-12. The candidate bump UrlResolver #764 (0.0.337 → 0.0.349, pin-only diff) is RED — bld-build + kotlin.build (remote) both FAILURE, 3 of 1087 stress tests fail because protocol 0.0.349 lacks other in-flight UrlProtocol fixes:

   • stressTestRelayFallbackRacesDirectConnectionForNonNatPeer — "callService() took 4110ms, exceeding the 2000ms per-iteration budget … direct connection is not being raced with relay." Needs UrlProtocol #356 "Race direct and relay paths in sendViaRelay" (OPEN, DIRTY — needs rebase).
   • stressTestOnPeerRegisteredWithRelayObservationBeforeRegistry — "4373 race-window violations … a concurrent reader observed a peer added by onPeerRegisteredWithRelay in the registry BEFORE the corresponding observationLog.recordSuccess() landed." A registry-vs-observationLog ordering race not fixed in 0.0.349.
   • a testNetlabDynamicFailureRecovery failure: "Inbound JSON frame cannot be parsed within the memory budget: frameBytes=2706967, worstCaseParseBytes=18948769, parseAmplification=7x, totalBudget=8388608."

   So a clean full fleet bump cannot happen until the UrlProtocol release train reaches a green, publishable protocol version. Force-shipping red 0.0.349 to prod would be worse than not deploying. This train is being worked by parallel sessions (UrlResolver has ~8 open PRs #760–#767; UrlProtocol #351–#359) — coordinate, don't collide.

Relevant PRs / refs (write-time state — RE-VERIFY)

| Item | State @ write | Link | |---|---|---| | droplet (c) #87 | MERGED 13:55Z · deployed | https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/87 | | WebCronApi (c) #18 | MERGED 13:52Z | https://github.com/CodexCoder21Organization/WebCronApi/pull/18 | | WebCronApiServer (c) #50 | MERGED 14:33Z · deployed | https://github.com/CodexCoder21Organization/WebCronApiServer/pull/50 | | jvm-libp2p (b) yamux #23 | MERGED 08:17Z · published snapshot-12 | https://github.com/CodexCoder21Organization/jvm-libp2p/pull/23 | | UrlResolver (b) gossip #762 | OPEN, CLEAN, queued pos 5; CI green | https://github.com/CodexCoder21Organization/UrlResolver/pull/762 | | UrlResolver protocol-pin #764 (→0.0.349) | OPEN, CI RED (3 stress-test fails) — the blocker | https://github.com/CodexCoder21Organization/UrlResolver/pull/764 | | UrlProtocol #356 (relay-direct racing) | OPEN, DIRTY (needs rebase); required for #764 green | https://github.com/CodexCoder21Organization/UrlProtocol/pull/356 | | DODroplet #88 (decouple) | must stay CLOSED — violates the hard invariant | https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/88 |

Deployed artifacts (prod CN host 198.199.106.165): /root/ContainerNursery/apps/webcron-url-server.jar (sha 397c1701…), /root/ContainerNursery/apps/droplet-service-server.jar (sha 23f56b41…). Old jars backed up on-host as *.jar.bak-20260712-145320 (webcron) and *.jar.bak-20260712-150448 (droplet).

Next steps (ordered — re-verify current state first)

0. Re-verify everything with the commands in the RE-VERIFY banner. In particular confirm whether #762 has merged, whether #764/#356 are still red/dirty, and whether the deployed jars/JVMs still match.
1. Confirm #762 lands on main (re-enqueue if it fell out again — it dropped out of the queue once this session; re-enqueue via GraphQL enqueuePullRequest). This is orthogonal to the deploy decision and just lands the gossip code.
2. Resolve the pending user decision on the (b) fleet bump (the full-auto premise is void because the cascade is red). Options presented to the user:
   • A — ship the gossip fix now (clean partial): once #762 merges, republish UrlResolver on protocol 0.0.337 + redeploy CN. Green; gets the O(N²) gossip win live. Yamux waits.
   • B — wait for the protocol train to green, then one full bump (RECOMMENDED, since the incident already self-recovered): hold prod until UrlProtocol's in-flight fixes (#356 + the observation-ordering fix) land and a green protocol publishes and #764-equivalent goes green; then a single clean fleet bump ships yamux + gossip together.
   • C — defer entirely: leave everything in main, deploy later.
3. When the protocol train is green (path B) or for path A: republish UrlResolver from main, then fleet-bump CN + url:// services + the fabric daemon and restart CN (drops warm containers ~30–60s). Verify with the deploy recipe below. Preserve the hard invariant on every path.
4. Follow-up (independent of the above): fix the WebCron listJobs() 30s SJVM timeout so the global sweep-net job can register — prefer a by-name upsert for the sweep job instead of listJobs-check-then-create.

Reusable / operational knowledge

• HardwareControlFabric CLI (preferred over SSH for the CN host): coursier launch community.kotlin.hardwarecontrolfabric:cli:0.2.6 -r https://kotlin.directory -- -H 198.199.106.165 -p 8443 -c ~/.config/hardware-control-fabric/fabric-client-cert.pem -k ~/.config/hardware-control-fabric/fabric-client-key.pem --ca-cert ~/.config/hardware-control-fabric/fabric-server-cert.pem <subcommand>. ping to check liveness.
• Binary-safe jar deploy: write-file --local-input-path <local.jar> --remote-output-path /tmp/new.jar (chunked upload; NOT --content, which base64-bloats). Then a host-side script via launch --cmd bash --args /tmp/deploy.sh --wait (60s cap; detach for longer). The deploy script must: verify sha256sum == expected → cp -p the old jar to .bak-<ts> → mv new jar into place → pgrep -f <jar> and kill the warm JVM so CN relaunches the new jar lazily on next request. read-file mangles binary — never pull jars back; verify by on-host sha256sum.
• processes --top N does NOT show low-CPU JVMs — the droplet service was warm but absent from top-40. Always confirm a JVM via pgrep -f <jar>, not the top-N list.
• CN CLI upload-jar does NOT work for url:// facade routes ("Route not found: 'webcron'") — that's why the deploy goes through fabric write-file + swap + kill, not the container-nursery-deploy skill's upload-jar.
• Merge queue: gh pr merge is rejected org-wide; enqueue via GraphQL enqueuePullRequest (get node id from gh pr view <n> --json id). PRs can silently drop out of the queue — re-check mergeQueueEntry and re-enqueue.
• jvm-libp2p default branch is develop (not main); group is community.kotlin.libp2p, artifact jvm-libp2p (no -jvm suffix). kotlin.directory returns HTTP 401 for non-existent artifact paths (not 404) — a 401 on curl doesn't prove "exists but unauthorized"; confirm publish with coursier resolve <coord> -r https://kotlin.directory.
• "kotlin.build (remote)" is a check-run (poll statusCheckRollup), not a legacy commit status. A 5–15s fast-fail there is a shared-infra provisioning flake; a 45–55min run that then fails is a real test failure.
• codex delegation (codex exec --dangerously-bypass-approvals-and-sandbox -m gpt-5.6-sol): always </dev/null and write output to a FILE; a resumed codex can stall at 0 CPU after pushing its work (reap it — check parent+child utime deltas in /proc/<pid>/stat; verify its work actually landed on the branch first).
• Related memory keys (author's private store): cn-tls-slowness-relay-cpu-starvation-2026-07-12, droplet-service-cchain-deployed-listjobs-sweep-skip, webcron-coupling-invariant-runaway-droplet-safety, fixA-cn-relay-yamux-oom-mostly-already-in-pr23-2026-07-12.