← Priority list

Make per-droplet url:// management cheap so the buildtest WUI doesn't need a 2GB heap (efficiency root cause)

buildtest /runs was unreliable because the WUI OOM-looped on a 128MB heap under droplet load; interim fix bumped it to 2GB (holds 10/10 at 75 droplets but still OOMs a few times/30min). Root cause: each per-droplet url:// call runs in an interpreted SJVM sandbox and can re-fetch ~2.4MB of client-stub bytecode per connection, so heap+CPU scale with droplet count. Fix: cache/reuse each droplet's connection+bytecode, reduce lease-poll frequency, then revert the heap.

hf-2026-07-19-make-per-droplet-url-management-cheap-so-the-buildtest-wui-doesn-t-need-a-2gb-heap-efficiency-root-cause

Edit handoff

Title Summary Markdown body

# Make per-droplet url:// management cheap so the buildtest WUI doesn't need a 2GB heap (identified, unfixed)
**Written:** 2026-07-19 ~13:1xZ by Claude (supervisor session). RE-VERIFY all state live before acting.

## Mission (unfixed efficiency root cause)
buildtest.kotlin.build/runs was unreliable because the WUI (page front-end) OOM-crash-looped on a 128MB heap
under droplet load. INTERIM FIX (deployed, holding): bumped the WUI heap to 2GB (ContainerNursery route config,
memoryLimitMb 2048 + MaxDirectMemorySize 512m) — /runs now 10/10 at 75 droplets. BUT the WUI still OOMs a few
times per 30 min at 2GB, so this is a CRUTCH, not a clean fix. Deeper root cause: per-droplet `url://`
management is far more expensive than "passing a few messages" should be; heap+CPU scale with droplet count.

## Evidence
- Backend served bytecode ~282x in a few minutes; each `url://` SandboxedConnection does a `__bytecode_request`
  fetching ~2.4MB of client-stub bytecode (impl ~1.75MB + stdlib ~0.67MB) loaded into an interpreted SJVM
  sandbox. If connections churn, this 2.4MB fetch+parse repeats per reconnect → the byte-array/JSON heap
  pressure (dominates CN's heap) and CPU (ForkJoinPool SJVM workers + libp2p/RelayService).
- Load scales with droplet count: ~14 droplets -> load ~21 (/runs fast); ~90 -> load ~50 (/runs times out on
  128MB WUI). Cull droplets -> load drops -> /runs fast. Bump WUI heap -> /runs holds at 75.
- renewDroplet (~217/window) + renewBuildLease (~36) per-droplet RPCs, each SJVM-sandboxed.

## Fix directions
1. Cache/reuse each droplet's SandboxedConnection AND its fetched bytecode so bytecode is fetched+parsed ONCE
   per droplet, not per call/reconnect. Confirm connections are not churning (relate to 0.638/0.641/0.642
   dead-transport/lock/refcnt fixes).
2. Reduce lease-renewal/status-poll frequency or make it event-driven instead of O(droplets) polling.
3. Once per-droplet management is cheap, revert the WUI heap from 2GB to a lean value; confirm /runs holds at 75.

## Rigor (do NOT shortcut)
Fail-first test proving bytecode is fetched once per connection (not per call); comprehensiveness + adversarial
reviews via codex gpt-5.6-sol; final review; PR; drive to green CI and MERGE. A prior gpt-5.6-sol attempt drifted
into the reaper/lease area and was stopped; start fresh from profiling.

## Refs
Repos: UrlResolver (foundation.url.resolver.sandbox SandboxedConnection/bytecode), UrlProtocol, BuildTestEmbedded/
BuildTestServerService. Host 198.199.106.165. Related handoffs: droplet-reaper-leak, eliminate-o-n-bookkeeping.
Interim WUI-heap config in ContainerNursery config.json (buildtest-wui route memoryLimitMb 2048).

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Make per-droplet url:// management cheap so the buildtest WUI doesn't need a 2GB heap (identified, unfixed)

Written: 2026-07-19 ~13:1xZ by Claude (supervisor session). RE-VERIFY all state live before acting.

Mission (unfixed efficiency root cause)

buildtest.kotlin.build/runs was unreliable because the WUI (page front-end) OOM-crash-looped on a 128MB heap under droplet load. INTERIM FIX (deployed, holding): bumped the WUI heap to 2GB (ContainerNursery route config, memoryLimitMb 2048 + MaxDirectMemorySize 512m) — /runs now 10/10 at 75 droplets. BUT the WUI still OOMs a few times per 30 min at 2GB, so this is a CRUTCH, not a clean fix. Deeper root cause: per-droplet url:// management is far more expensive than "passing a few messages" should be; heap+CPU scale with droplet count.

Evidence

Backend served bytecode ~282x in a few minutes; each url:// SandboxedConnection does a __bytecode_request fetching ~2.4MB of client-stub bytecode (impl ~1.75MB + stdlib ~0.67MB) loaded into an interpreted SJVM sandbox. If connections churn, this 2.4MB fetch+parse repeats per reconnect → the byte-array/JSON heap pressure (dominates CN's heap) and CPU (ForkJoinPool SJVM workers + libp2p/RelayService).
Load scales with droplet count: ~14 droplets -> load ~21 (/runs fast); ~90 -> load ~50 (/runs times out on 128MB WUI). Cull droplets -> load drops -> /runs fast. Bump WUI heap -> /runs holds at 75.
renewDroplet (~217/window) + renewBuildLease (~36) per-droplet RPCs, each SJVM-sandboxed.

Fix directions

Cache/reuse each droplet's SandboxedConnection AND its fetched bytecode so bytecode is fetched+parsed ONCE per droplet, not per call/reconnect. Confirm connections are not churning (relate to 0.638/0.641/0.642 dead-transport/lock/refcnt fixes).
Reduce lease-renewal/status-poll frequency or make it event-driven instead of O(droplets) polling.
Once per-droplet management is cheap, revert the WUI heap from 2GB to a lean value; confirm /runs holds at 75.

Rigor (do NOT shortcut)

Fail-first test proving bytecode is fetched once per connection (not per call); comprehensiveness + adversarial reviews via codex gpt-5.6-sol; final review; PR; drive to green CI and MERGE. A prior gpt-5.6-sol attempt drifted into the reaper/lease area and was stopped; start fresh from profiling.

Refs

Repos: UrlResolver (foundation.url.resolver.sandbox SandboxedConnection/bytecode), UrlProtocol, BuildTestEmbedded/ BuildTestServerService. Host 198.199.106.165. Related handoffs: droplet-reaper-leak, eliminate-o-n-bookkeeping. Interim WUI-heap config in ContainerNursery config.json (buildtest-wui route memoryLimitMb 2048).