Handoff: NetLab NAT-traversal e2e → netlab-manager __bytecode_request OOM → fix by STREAMING the proxy bytecode path (protocol 0.0.321)

Written: 2026-07-05 (thread began 2026-06-21). Every state snapshot below was verified at write time — RE-VERIFY before acting. PRs merge, queues shuffle, prod services restart, the host's memory picture changes hour to hour, and concurrent agents bump foundation.url:protocol/resolver versions fast. gh pr view <num> --repo <r> --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative; never trust this doc over a fresh query.

Overlap with other handoffs — read them too. This is the __bytecode_request facet of the netlab-manager -Xmx128m OOM. Sibling 2026-07-05 handoffs diagnose the SAME manager OOM from other angles: 2026-07-05-netlab-manager-oom-uploadfile-materialization-root-cause.md (the uploadFile payload materialization), 2026-07-05-netlab-manager-oom-sjvm-close-leak-and-cn-outage-handoff.md / -sjvm-sandbox-leak.md (SJVM close leak), NetLab-workstream-handoff-2026-07-05.md, and 2026-07-05-ci-stabilization-and-fleet-redeploy.md (the 2-core/3.9 GB host swapping). The manager OOMs from several independent transient-materialization sources; this handoff removes one of them (bytecode serving) and is a genuine software fix, NOT "raise the heap" (forbidden by the owner and every sibling handoff).

---

1. Mission / original request

Two phases in one thread:

• Phase 1 (done, merged earlier): make NetLab run a custom-Docker topology e2e — a browser + W3Wallet plugin + daemon behind a NAT on one node ↔ webserver + permissions server on another — verified by Playwright, in JS and JVM variants. It was already built; I wired the last gap (Playwright trace capture + netlab download-file retrieval) and corrected stale docs.
• Phase 2 (this handoff): "The NetLab NAT-traversal checks in W3WalletTests are still red. Ensure it goes green, then enable it as a required check for merging." Root-caused to url://netlab-hosted/ (the CN-hosted netlab manager) OOM-crash-looping. This handoff is the bytecode-serving contributor to that OOM.

Global CLAUDE.md workflow applies (root-cause first; failing-first e2e test; fix at the owning layer / upstream, never mitigate downstream; never bypass CI / raise heap).

2. Bottom line (status at handoff — RE-VERIFY)

• Root cause (bytecode facet) found + fixed + verified. The -Xmx128m manager Base64-materialized its whole SJVM impl jar (~1.7 MB) plus a deprecated stdlib jar (~0.67 MB) into a Map on every __bytecode_request; every sandboxed client re-fetches on (re)connect, so a reconnect storm OOM-crash-loops it.
• Fix = "stream the proxy path" upstream (the user explicitly chose this over "remove the handler" and over #73's "cache the base64"). Two pieces, both in foundation.url:protocol 0.0.321 (published to kotlin.directory): a new bytecode-aware bind overload + a proxy-fallback passthrough. NetLab binds via the overload and deletes its hand-rolled handler.
• foundation.url:protocol:0.0.321 is PUBLISHED and live (https://kotlin.directory/foundation/url/protocol/0.0.321/ → 200).
• NetLabManagerServer #76 is MERGED (manager 0.0.50).
• UrlProtocol #328 is OPEN — the upstream fix. Its required kotlin.build (remote) lane finally passed (27m53s); bld-all-tests then showed a fast 4m flake (it passed cleanly earlier at 16m55s — almost certainly a runner/infra flake, since only empty re-trigger commits were pushed). Needs a re-trigger to go green, then merge.
• NetLabManagerServer #73 is CLOSED (superseded — it only cached the base64; the upstream fix eliminates the materialization).
• No uncommitted/unsaved changes anywhere. Both work branches are fully pushed; the prod-host investigation was read-only.
• DEPLOYMENT IS NOT DONE — it is the owner's step ("merge only, I deploy"). See §6. There is a hard deploy-order coupling: prod ContainerNursery must run protocol ≥ 0.0.321 before the manager (0.0.50) is deployed, or bytecode serving breaks.

3. PRs / artifacts (state verified at write time — RE-VERIFY)

| PR / artifact | What | State | |---|---|---| | UrlProtocol #328 | THE upstream fix. (1) New UrlProtocol.bind(url, handler, implementationJar, implementationClassName) overload — the framework answers __bytecode_request for a TcpServer-bound service with a StreamingBytecodeResponse (peak heap ≈ jar size; TcpServer.kt:224 already streams that marker), so the bound handler never materializes. (2) ServiceRegistration.kt translateBackendBytecodeResponse passthrough: forward an already-final BytecodeProtocol response (jarBytes/implClassName, or notModified) unchanged instead of demanding legacy {jar, className} backend keys. Additive + binary-compatible with 0.0.314. foundation.url:protocol → 0.0.321. Branch fix-stream-proxy-bytecode-path. | OPEN — remote lane green, bld-all-tests flaked; re-trigger + merge | | NetLabManagerServer #76 | Manager binds via the new 0.0.321 overload; deleted the hand-rolled __bytecode_request dispatch case + the dead jarBytes/implClassName/stdlibJarBytes handler params + the loadStdlibJar plumbing. getImplementationJar/getImplementationClassName kept for P2P mode. foundation.url:protocol 0.0.314→0.0.321; netlab-manager-server → 0.0.50. Test testManagerDoesNotServeBytecodeRequestItself. | MERGED | | NetLabManagerServer #73 | Old approach: cache the base64 once. Superseded. | CLOSED | | foundation.url:protocol:0.0.321 | The published artifact carrying the fix. | PUBLISHED (kotlin.directory) |

4. THE ROOT CAUSE + why the obvious fixes are wrong (the key insight)

I initially concluded "remove the manager's __bytecode_request handler — the framework serves it." THAT IS A PROD REGRESSION. Do not do it. Here is why, mechanism end-to-end:

• Prod url://netlab-hosted/ runs the manager in url-bind mode under ContainerNursery (URL_BIND_DOMAIN lazy-start; README lines 55/77/165). The manager binds protocol.bind(localhost.tcp:PORT, lambda) — a raw RPC handler with NO getImplementationJar.
• CN's UrlFacade registers netlab-hosted as a proxy (supportsSandboxedExecution() == false). So UrlProtocol's ServiceRoutingHandler.handleBytecodeRequest (ServiceRegistration.kt:826) takes the FALLBACK path (:901): it forwards __bytecode_request to the manager backend via service.handleRequest, and translateBackendBytecodeResponse (:960) requires a {jar:<base64>, className} map — it throws JSONObject["jar"] not found otherwise.
• Therefore the manager must answer __bytecode_request as a base64 map in this topology. The framework's optimized streaming path (:855, StreamingBytecodeResponse) only applies to services registered directly with getImplementationJar (P2P mode / the resolver's ServiceRoutingHandler) — never to a proxied backend. Removing the manager's handler → missing 'jar' key → bytecode serving broken in prod.

The OOM: that base64 materialization (impl jar + a stdlib jar the framework's own optimized path never even serves) is allocated per request, and every sandboxed client re-fetches bytecode on (re)connect → reconnect storms blow the -Xmx128m heap.

The fix (upstream, "stream the proxy path"): make the framework stream the bytecode on the url-bind/TcpServer path too, so the manager never materializes — mirroring what the resolver path already does. StreamingBytecodeResponse is a sender-side heap optimization only: its wire bytes are byte-identical to the materialized response (BytecodeResponseStreamer.kt:41), so the reader/consumer is unchanged. The passthrough (piece 2) lets CN's proxy forward the manager's now-streamed response without re-demanding the legacy keys.

Fail-first tests (verified RED→GREEN with the build cache purged — the warm ~/.aibuildcaches masks src reverts):

• UrlProtocol/tests/testProxyForwardsStreamedBytecodeResponseUnchanged.kts — proven RED on 0.0.314 with the exact JSONObject["jar"] not found; GREEN with the passthrough.
• UrlProtocol/tests/testBindStreamsServiceOwnBytecodeRequest.kts — e2e: a service bound via the new overload has __bytecode_request served by the framework via streaming; the bound handler is never invoked for it; the jar/className round-trip intact.
• Existing testServiceRoutingHandlerForwardsBytecodeWhenNotSandboxed (legacy keys) still passes — no regression. NetLab full suite 46/46 locally; NetLabManagerServer/tests/testManagerDoesNotServeBytecodeRequestItself.kts.

5. Prod-host investigation (why CI's kotlin.build (remote) lane flakes red)

Verified live on 198.199.106.165 via ssh -p 23 root@… (HardwareControlFabric daemon 8443 was DOWN, so SSH is the sanctioned fallback per CLAUDE.md's hierarchy):

• 2-vCPU / 3.8 GB droplet (nproc=2), ~16 CN-launched services each -Xmx128m (≈3.4 GB used, ~250 MB free, swapping ~715 MB). Genuinely near memory capacity with legitimate services — not one bloated leaker, and disk was FINE (72%). PSI mem full avg10=0.08%.
• The one anomaly: TWO live netlab-manager-server.jar JVMs (~250 MB each) — a duplicate. CN's danger_jarfile reaper failed to kill the old instance when it respawned a new one. This is downstream of the deployed manager's bytecode-OOM crash-loop (respawn → reaper leaks the old one). The buildtest service (also -Xmx128m) then times out under the memory pressure → the remote lane fails with "Service Unavailable: Read timed out" (or a random testNetlab*/timing-subset flake) on innocent PRs.
• /tmp is on-disk (/dev/vda1), NOT tmpfs — deleting stale heap dumps frees disk, not RAM. 0 orphaned container-nursery-jar-* dirs (dir cleanup works; only the JVM process reaping leaks).
• I did not kill the duplicate manager: CN's route (GET https://api.nursery.wasmserver.com/routes, provider danger_jarfile) does not expose the runtime port it proxies to, so I couldn't safely pick the orphan without risking a netlab-hosted outage (the e2e target). The owner's redeploy reaps it cleanly (CN restart kills all + starts one fresh).
• Chicken-and-egg: the prod-host degradation is partly caused by the deployed buggy manager (this bug). Deploying 0.0.50 breaks the loop → no more reaper leaks. So the prod-host "fix" is the deployment.

6. NEXT STEPS (the pick-up point)

A. Land UrlProtocol #328 (small, do first).

1. gh pr checks 328 --repo CodexCoder21Organization/UrlProtocol — the bld-all-tests failure is almost certainly a flake (it passed at 16m55s; only empty re-trigger commits followed). Re-trigger with an empty commit (git commit --allow-empty -m "re-trigger" && git push on branch fix-stream-proxy-bytecode-path) or via the merge queue. If it's a real failure, it would be staleness against a moved UrlProtocol main (concurrent agents raced protocol 0.0.314→0.0.320+ from unmerged branches) — rebase on latest main + renumber the published version (0.0.321 is taken by me; pick the next free one, republish, and bump NetLab's pin to match) only if truly needed. Local tests already pass; do not disable/bypass.
2. Merge #328 via the merge queue (squash; GraphQL enqueuePullRequest; 0 required reviews; never --admin/bypass). Note NetLab #76 already merged out of order — harmless because it pins the published 0.0.321 artifact, not #328's source; but #328 must merge so UrlProtocol main carries the 0.0.321 source.

B. DEPLOY (owner's step — "merge only, I deploy"). ORDER IS LOAD-BEARING:

1. Redeploy prod ContainerNursery onto foundation.url:protocol ≥ 0.0.321 FIRST. The manager's streamed __bytecode_request response is BytecodeProtocol format (jarBytes/implClassName); a CN on an older protocol rejects it with JSONObject["jar"] not found. (Prefer HardwareControlFabric; it was down → SSH:23. Use the container-nursery-deploy skill.)
2. Then deploy netlab-manager-server 0.0.50 to CN. This also reaps the duplicate manager JVM.
3. Re-run the NetLab NAT-traversal e2e (W3WalletTests netlab-e2e.yml, both js + jvm variants) and confirm it is reliably green across several runs (the manager should stop crash-looping; watch for new heap dumps in /root/ContainerNursery/heapdumps/ and jstat steady — remember the manager still has the OTHER OOM sources: the uploadFile materialization in the sibling handoff, etc.; those must ALSO be fixed before the manager is truly stable and the e2e is durably green).
4. Add the e2e as a required check in the W3WalletTests main ruleset (task originally requested). Do this only once it is reliably green.

C. Follow-ups (separate work).

• True zero end-to-end materialization: the CN→client hop still materializes the base64 on CN's (large) heap. Eliminating it needs a streaming-forward proxy primitive in ServiceRoutingHandler's fallback + ContainerNursery's UrlFacade. The -Xmx128m OOM victim (the manager) is already fixed; CN has headroom, so this is efficiency, not the OOM.
• Fix CN's danger_jarfile reaper so it kills the old instance when it respawns (the duplicate-JVM leak). ContainerNursery repo.
• Remaining netlab-manager OOM sources (sibling handoffs): uploadFile/downloadFile payload materialization (make the sandboxed-RPC InputStream streaming reliable, then revert netlab's ByteArray workaround), SJVM close leak. The manager will keep OOMing until ALL of these land + deploy.
• Other open NetLabManagerServer OOM-fix PRs existed in the thread (#58 TTL reaper / droplet leak, #64 keepalive) — audit/rebase/green/close as appropriate.

Considered but NOT chosen (so it isn't re-litigated):

• Remove the manager's __bytecode_request handler — a prod regression (§4). Rejected.
• #73's cache-the-base64 — a downstream mitigation; still holds/re-serializes the multi-MB base64. Rejected in favor of the upstream streaming fix.
• Raise -Xmx/metaspace, or admin-bypass the flaky required check — explicitly forbidden. Do not.
• Kill the duplicate manager JVM by hand — unsafe (can't identify the orphan; risks a netlab outage). The redeploy reaps it.

7. Key files / commands

• UrlProtocol: src/foundation/url/protocol/UrlProtocol.kt (the bind overload + wrapWithStreamingBytecodeServe), ServiceRegistration.kt (translateBackendBytecodeResponse passthrough + isAlreadyBytecodeProtocolResponse), BytecodeResponseStreamer.kt (StreamingBytecodeResponse — sender-side optimization), TcpServer.kt:224 (streams the marker). Build/test: scripts/test.bash --test tests/<file>.kts. Purge ~/.aibuildcaches/_home_hostusr_workspace_UrlProtocol before trusting a same-version src edit (warm cache serves stale jars).
• NetLab: src/netlabmanager/Main.kt (runUrlMode binds via the overload), NetLabManagerRpcHandler.kt (no more __bytecode_request case), build.kts (protocol 0.0.321, module 0.0.50).
• Publish protocol: coursier launch community.kotlin.maven.artifact.publishing:community-kotlin-maven-artifact-publishing:0.0.5 -r https://kotlin.directory -- --workspace <UrlProtocol> --buildrule foundation.url.protocol.buildMaven --api-url https://api.kotlin.directory/upload --force (409 = version already taken → renumber).
• Related memory: netlab-manager-bytecode-serve-must-stream-not-remove, cn-reaper-leaks-duplicate-netlab-manager-jvms.

🤖 Generated with Claude Code