Handoff — Maven-server-unresponsive → netlab-manager __bytecode_request OOM → "my lane" (#657 + #229 deploy)

Date: 2026-06-29 (date of the first message in this thread) Author: Claude (Opus 4.8, 1M context) session Status: My PR #73 (netlab OOM fix) was superseded by the team's streaming approach (NetLabManagerServer #76, MERGED). My scoped-down "my lane" is essentially done: BuildTestEmbedded #229 is deployed + verified working; UrlResolver #657 is green + queued but blocked by a stalled org merge queue.

---

1. Original request (evolved across the conversation)

1. Start: "Rebase — the world has changed; verify whether the two UrlResolver flakes are still a problem" (flake #1 testRelayRegistrationCompletesQuickly / joinNetwork <2000ms; flake #2 stressTestLateJoinerServiceDiscoveryUnderLoad), after main bumped foundation.url:protocol to 0.0.320. Standing directive: if you find flakes, fix them and open PRs (don't merge, await review), get them green.
2. Detour — buildtest outage: buildtest.kotlin.build appeared stuck → "fix it 100%."
3. Detour — maven server: kotlin.directory maven server went unresponsive, blocking a buildtest deploy → "investigate exactly what went wrong and create a PR to fix it, make the PR go green, do not merge until I approve."
4. Scope narrowed (user chose "Just my lane"): leave the team's netlab/SJVM PRs alone; take UrlResolver #657 toward merge and deploy BuildTestEmbedded #229 to url://buildtest:.

---

2. What is DONE

2a. Root-caused the maven-server unresponsiveness — it was a VICTIM, not the culprit

• kotlin.directory maven server going unresponsive (000 timeouts, later connection-refused) was not its own bug: its live heap was healthy (~10–20 MB after GC), thread pool idle (35/39 Jetty threads parked), but ~70 MB of it was swapped out (VmSwap in /proc/<pid>/status). It was starved by host memory pressure (swap) — the same class as the prior cn_no_xmx_host_swap_starves_buildtest incident.
• Driver of the host swap = netlab-manager-server OOM-crash-looping every few minutes (heap dumps in /root/ContainerNursery/heapdumps dominated by foundation/url + netlab/api; pid churning with uptime=0s). Crash-loop persisted even after host RAM freed to ~2.9 GB → confirms it's the manager's own 128 MB-heap code, not host pressure.

2b. Root-caused the netlab-manager OOM crash-loop

NetLabManagerRpcHandler.dispatch's "__bytecode_request" branch re-encoded BOTH served jars (the ~2 MB client-impl jar and the ~2 MB SJVM stdlib jar) to fresh Base64 strings on EVERY call:

"__bytecode_request" -> mapOf(
    "jar"       to Base64.getEncoder().encodeToString(jarBytes),        // fresh ~2.7 MB String each call
    "stdlibJar" to Base64.getEncoder().encodeToString(stdlibJarBytes))  // another fresh String each call

Every sandboxed url:// client fetches __bytecode_request when it opens its connection. When the manager restarts, every client reconnects at once → a storm of re-encodings, each allocating multiple MB of transient Base64 → exhausts the -Xmx128m heap (OutOfMemoryError: Java heap space) → JVM exits → cold-start → reconnect storm again = self-sustaining crash-loop. (OOM is HEAP, not Metaspace — jstat M=98% was a symptom; jmap -histo:live post-GC heap was tiny = transient burst, not retention leak.)

2c. My fix (NetLabManagerServer #73) — CLOSED / SUPERSEDED

• Encoded each immutable jar to Base64 once at handler construction, served cached strings (jarBase64, stdlibJarBase64).
• Failing-first test bytecodeRequestEncodesJarsOncePerHandlerNotPerCall: 200 __bytecode_request fetches retaining each response. Red — re-encoding produced 200 distinct multi-MB strings and OOM'd the test JVM with the exact Java heap space symptom (0.2 s). Green after the fix. All 3 CI checks went green (incl. kotlin.build (remote)).
• Branch (pushed, still on origin): fix-manager-bytecode-request-oom-crash-loop @ commit 1aaebc8.
• The user closed #73, superseding it with the more robust streaming approach: NetLabManagerServer #76 (MERGED 2026-07-01) streams the bytecode via the framework (no full in-heap encoding at all), on top of upstream UrlProtocol #328. That is the upstream-correct fix and I agree it's better than my downstream base64-cache. The reusable failing-first test in #73's branch is still worth cherry-picking as a regression guard for the reconnect-storm OOM.

2d. "My lane" item 1 — BuildTestEmbedded #229 deploy: DONE + verified working

• #229 (recovery no longer re-projects already-recovered runs on every restart — fixes the buildtest recovery-OOM crash-loop) is MERGED and long since rolled into current main (buildtest-embedded is now 0.0.207; BuildTestServerService main already references it).
• The running buildtest jar already contains the fix: /root/ContainerNursery-uploads/jars/buildtest-server-service.jar (deployed 2026-07-01 05:26) — unzip -p … BuildTestEmbeddedService*.class | grep recovery-complete-v1 → present. NOTE: the /root/ContainerNursery/apps/… copy is a stale (March) decoy; the real deployed jar is the one in the running process's /proc/<pid>/cmdline (under ContainerNursery-uploads/jars/).
• Verified active + healthy: 363 .recovery-complete-v1 markers across 403 run dirs in /root/buildtest-data/runs; buildtest stable (pid uptime growing, not OOM-looping); buildtest.kotlin.build serving 200 (/health fast). No deploy action needed — it's already live.

2e. "My lane" item 2 — UrlResolver #657 toward merge: green + queued, BLOCKED by stalled merge queue

• #657 ("Guard: SJVM ServiceBridge.rpc carries numeric map params as native host types") is a small additive guard test. OPEN, MERGEABLE, both checks green (bld-build + kotlin.build (remote) SUCCESS). It is already in the merge queue.
• It will not merge because the UrlResolver merge queue is stalled — no PRs merged to main for an extended window (last merges were #664 @ 2026-06-30 07:27 / #666 @ 2026-06-29). A CLEAN, green, queued PR sitting unmerged = the queue's combined-CI isn't advancing, which is the same broad org-wide red CI condition (netlab/SJVM chain + kotlin.build flakiness) that the user separately flagged. Not #657's fault.

---

3. Context: the netlab OOM is now the team's workstream (I was scoped out)

The netlab/SJVM-OOM effort spans several repos. Current verified states (2026-07-05):

• NetLabManagerServer #76 — stream manager bytecode — MERGED (2026-07-01) ← the real fix that superseded my #73.
• NetLabManagerServer #75 — stream uploadFile end-to-end — MERGED.
• NetLabManagerServer #77 — resolver 0.0.582 / SJVM connection-leak — CLOSED (not merged).
• UrlProtocol #328 — stream proxy bytecode path (upstream) — OPEN, DIRTY (conflicts with main).
• The team's handoffs in this folder cover this fully: 2026-07-05-netlab-manager-oom-*.md, 2026-07-05-buildtest-server-service-heap-leak-oom-crashloop-and-netlab-sharing-verified.md, 2026-07-05-ci-stabilization-and-fleet-redeploy.md. Read those before touching shared netlab/CN infra.

---

4. NEXT STEPS (in priority order, enough detail to drive to completion)

1. Land UrlResolver #657 (the only open "my-lane" item).

   • It is green + mergeable + already queued. It's blocked solely by the stalled UrlResolver merge queue.
   • Option A (preferred): wait for the merge queue to recover once the org-wide CI health is restored (that recovery is the CI-stabilization workstream — see the 2026-07-05-ci-stabilization-and-fleet-redeploy.md handoff). Then confirm #657 auto-merges.
   • Option B (if authorized): direct-merge bypassing the stuck queue — gh pr merge 657 --repo CodexCoder21Organization/UrlResolver --merge (no --auto, no --delete-branch). This sidesteps branch protection / the queue, so it needs explicit maintainer approval. #657 is green + trivial (additive test), so it's low-risk to land directly.
   • Definition of done: #657 state == MERGED.

2. Optionally preserve the reconnect-storm regression test. My superseded #73 branch (fix-manager-bytecode-request-oom-crash-loop, commit 1aaebc8) contains tests/bytecodeRequestEncodesJarsOncePerHandlerNotPerCall.kts, which reproduces the __bytecode_request OOM deterministically (200 fetches → Java heap space). Since #76 changed the code path to streaming, this test would need adaptation, but it's a valuable guard against a re-introduction of in-heap bytecode buffering. Cherry-pick + adapt if the team wants belt-and-suspenders coverage.

3. Nothing else in my lane. #229 is deployed + verified; the netlab OOM is the team's (#76 merged, #328/#77 their call). The broader "lot of red" is (a) the team's active netlab/SJVM chain and (b) stale/deliberately-red experiments (e.g. UrlProtocol #236/#237 kompile-cli:0.0.33/0.0.34 pin-tests) — I was explicitly scoped OUT of both.

---

5. All relevant PRs + current status (verified 2026-07-05)

| PR | Purpose | Status | |----|---------|--------| | UrlResolver #657 | SJVM numeric-param guard test | OPEN, green, MERGEABLE, queued — blocked by stalled merge queue | | BuildTestEmbedded #229 | buildtest recovery-OOM (stop re-projecting recovered runs) | MERGED + DEPLOYED + verified live (recovery-complete-v1, 363 markers) | | NetLabManagerServer #73 (mine) | __bytecode_request re-encode OOM — cache Base64 | CLOSED / superseded by #76 (branch fix-manager-bytecode-request-oom-crash-loop @ 1aaebc8 still on origin) | | NetLabManagerServer #76 | stream manager bytecode (replaces #73) | MERGED (2026-07-01) | | NetLabManagerServer #75 | stream uploadFile end-to-end | MERGED | | NetLabManagerServer #77 | resolver 0.0.582 SJVM connection-leak | CLOSED | | UrlProtocol #328 | stream proxy bytecode path (upstream) | OPEN, DIRTY |

---

6. Uncommitted / unsaved work

None. Every local working clone (/code/NetLabManagerServer, /code/BuildTestServerService, /code/BuildTestEmbedded) has 0 dirty files and 0 unpushed commits. My only code contribution is fully committed + pushed to origin as branch fix-manager-bytecode-request-oom-crash-loop (NetLabManagerServer PR #73, now closed/superseded but the branch — with the fix + the failing-first test + the 0.0.559→0.0.562 test-pin alignment + the 0.0.48 build.kts bump — is preserved on origin and linked above). Nothing is stranded locally.

---

7. Key gotchas for whoever picks this up

• A maven/build-server going unresponsive is usually a VICTIM of host swap, not its own bug. Diagnose with VmRSS/VmSwap in /proc/<pid>/status + jmap -histo:live (a tiny post-GC heap ⇒ transient burst / external starvation, not a leak). Find the memory hog on the host (ps -eo pid,rss,comm --sort=-rss; check each JVM's -Xmx and swap).
• Heap dumps in /root/ContainerNursery/heapdumps are SHARED across all CN containers. Attribute a dump by strings <dump> | grep -oE '<pkg>/<sub>' package counts + the pid sequence — never assume which service produced it.
• kompile-cli 0.0.65 treats a cross-.kts WithArtifact version mismatch as a FATAL unhandled effect (WithArtifactVersionMismatchWarningEffect at LocalBuildWorkspace.execute) → the whole test workspace fails to build and NO test runs. Align the pins (here: three worker-client test files pinned foundation.url:resolver at 0.0.559 ×2 vs 0.0.562 ×1 — I aligned to 0.0.562). (My older memory that "WithArtifactVersionMismatch = noise" is WRONG for 0.0.65.)
• Stale-build trap (kompile_buildmaven_stale_jar_coursier_bldbinary_cache): after editing module source, scripts/test.bash can reuse the cached <module>.buildMaven() jar and NOT recompile (0 compile lines; a "fixed" test still fails). Clear BOTH ~/.aibuildcaches/_code_<Project>/{buildRuleResultIndex,artifactBinaryCache,persistent-scratchspace,buildRuleLocalArtifactDirIndex} AND the coordinate-keyed ~/.cache/coursier/v1/bldbinary/repository/<group>/<artifact>/<version>/…jar.
• The real deployed jar is the one in the running process's /proc/<pid>/cmdline (ContainerNursery-uploads/jars/…), not ContainerNursery/apps/… (which can be a months-old decoy). Verify a fix is live by unzip -p <running-jar> <class> | grep <marker>.
• A green+mergeable PR that won't merge ⇒ check whether the merge queue is advancing at all (last-merge-to-main timestamp). A stalled queue is an org-CI-health problem, not the PR's.