Handoff: buildtest scheduling question → CN outage fix → getBuildRun RPC flake root cause

Written: 2026-07-05. All state snapshots below were verified at write time — RE-VERIFY before acting. PRs merge, queues shuffle, services restart, and the host's memory picture changes hour to hour. gh pr view <num> --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative; never trust this doc's snapshot over a fresh query.

This workstream overlaps heavily with the other 2026-07-05 handoffs (2026-07-05-ci-stabilization-and-fleet-redeploy.md, 2026-07-05-urlresolver-race-and-leak-bug-bundles.md). They independently diagnosed the same root disease: the whole control plane runs on one 2-core / 3.9 GB droplet (198.199.106.165, ssh port 23, root) that swaps under load, so SJVM RPC handlers stall past their watchdog. My getBuildRun finding (below) is another expression of that disease. The fleet-redeploy task in that handoff is the highest-leverage fix for my remaining blocker (#661).

Mission summary

Original request (verbatim intent): a user looking at buildtest run d41cb83e asked "if tests run 4-at-a-time, how do we ever get 1 running + 1 pending?" It expanded, in order, into:

1. Explain the pending/parallelism behaviour.
2. Fix the root cause(s) with PRs that go green.
3. Investigate + fix the buildtest service outage that was blocking CI.
4. Fix the systematic cause of the recurring kotlin.build (remote) CI flake and fully resolve it.

What was found and done (the chain)

1. The question, answered. Parallelism is 16 — 4 static shards × a 4-wide intra-shard pool (kompile-core Workspace.kt val parallelism = 4, run on 4 separate droplets via BuildTestEmbedded/src/buildtest/embedded/TestSharding.kt). Run d41cb83e completed 912/912. The "1 running + 1 pending" was NOT a scheduler stall — the Phase-2 refill loop (Workspace.kt ~line 2441) provably cannot idle a thread while a test waits. It was the WUI live-status lagging (event-stream projection lag while the service was swap-slow), plus a no-package test defect. The real inefficiency was gross shard imbalance: one 22.5-min test (testNetlabDynamicFailureRecovery) was weighted as 30 s, so its shard ran 54.8 min while the other three finished ~31-37 min and their droplets idled ~20 min.

2. buildtest outage — root-caused and fixed (twice). ContainerNursery was running with no -Xmx (java -jar bin/container-nursery.jar config.json) → ~1 GB default heap → host swap → the -Xmx128m buildtest service answered its SJVM RPCs too slowly and OOM-restart-looped. Fixed by restarting CN with -Xmx128m (RestartCli 0.0.5). It recurred to a total outage and was restored via SSH. Full forensic detail + the SSH-only restore procedure are in the auto-memory file cn_no_xmx_host_swap_starves_buildtest.md. Current state: CN is correctly -Xmx128m and has held; the residual host swap (~500 MB at write time) is the fleet-wide leaked-memory problem the other handoff's fleet-redeploy task addresses.

3. BuildTestEmbedded #228 — MERGED (2026-06-30). Shard planner now weights a no-history test by its declared @Timeout instead of a flat 30 s default, so an expensive test (e.g. a netlab test declaring @Timeout(2_400_000)) gets its own shard instead of being piled onto one. Fix in TestSharding.kt (shardDurationEstimateMs helper, floored at DEFAULT_TEST_DURATION_ESTIMATE_MS). Regression test shardPlanningWeightsNoHistoryTestsByDeclaredTimeout (fails on old code: giant shares a shard Expected <1>, actual <2>; passes now).

4. UrlResolver #661 — OPEN, CI RED (infra), branch fix-no-package-test-files-and-guard, head 1b346e0. Adds package foundation.url.resolver to the only 2 test files that lacked one (stressTestConcurrentRegistrationsReliablyPropagateToRelay.kts, stressTestConcurrentLateJoinersDiscoverServicePromptly.kts — a no-package test is discovered with an empty project and addressed by a bare function name, the documented sharding hazard), plus a guard test allResolverTestFilesDeclareAPackage that fails and names any tests/*.kts missing a package. Locally verified both ways (passes clean; fails naming the file when a package is removed). Every CI red on this PR has been infra, not the change — see the root cause below.

5. The systematic CI flake — root cause understood (NO code fix exists for the named method). The recurring kotlin.build (remote) failure is SandboxException: SJVM host-side RPC interrupted during method=getBuildRun (watchdog timeout, cancellation, or shutdown). Investigation disproved the obvious hypotheses and found the real mechanism — see the dedicated section below.

Relevant PRs (state verified at write time — re-verify)

| PR | State | Notes | |---|---|---| | BuildTestEmbedded #228 | MERGED 2026-06-30T15:00:57Z | Shard balancing by declared @Timeout. Done. | | UrlResolver #661 | OPEN, both checks FAILURE (infra) | No-package fix + guard. Correct + locally verified; blocked only by the getBuildRun RPC flake. Last CI run 2026-06-30; needs a re-run once the host is stabilized. User is OK to merge on its merits if flakes persist, but prefers green. | | UrlResolver #663 | OPEN, red | NOT mine — separate work ("share one JVM-wide resolver thread pool"), same infra flake. Listed only so you don't confuse it for mine. |

The getBuildRun RPC flake — full root cause (this is the crux)

Symptom: kotlin.build (remote) fails with foundation.url.protocol.sandbox.SandboxException: … SJVM host-side RPC interrupted during method=getBuildRun … cause: java.lang.InterruptedException. It is labelled a "Build infrastructure error" by the check itself and comes with 0 test failures.

What it is NOT (verified, don't re-litigate):

• Not getBuildRun being slow. getBuildRun (BuildTestEmbeddedService.kt:564) is runs[runId] (in-memory ConcurrentHashMap) → copy() under a per-run state lock. O(1). It never replays the event log.
• Not lock contention. mutateRunAndSave (~line 1935) releases the state lock before the disk write (writes under a separate disk lock), so reads never block on I/O.
• Not buildtest heap/GC. Live buildtest at investigation time: 323 MB RSS, 0 swap, FGC=0, ~9 s total GC over 75 min, only 382 runs on disk. The in-memory runs map is unbounded in principle (loadFromDisk at ~2102 loads every run; archiveBuildRun at 670 only sets archived=true, never evicts) but at 382 runs it is not the bottleneck.

What it actually is: the resolver's own code documents it. In UrlResolver/src/foundation/url/resolver/sandbox/SandboxedConnection.kt:590-607, the catch (e: InterruptedException) that throws this exact SandboxException explains the cause: "a SandboxedProxyGenerator watchdog interrupting a blocking call inside the host-side rpcHandler during P2P cold-start." The watchdog is in SandboxedProxyGenerator.kt:857 (DEFAULT_TIMEOUT_SECONDS = 30, but the comment at :864 notes "120s for BuildTestClient"). So on the failing runs the RPC blocks for a full 120 seconds in P2P connection setup (bootstrap mesh → discover url:buildtest: → connect, possibly via relay + Noise) — it never reaches buildtest's method — then the watchdog interrupts it, the interrupt is translated to the recognizable SandboxException, and the poller does not retry, so the check goes red.

Why the cold-start blocks 120 s: buildtest is unreachable/unresponsive at that instant. Most acutely while buildtest was restart-looping under the CN outage (a restarted service kills the in-flight connection and the RPC hangs instead of detecting the dead peer), and under host swap pressure generally. The real defect: an unreachable/restarted url:// service should make the RPC fail fast (seconds), not hang to the 120 s watchdog. The slow-hang is what turns a transient buildtest blip into a red check.

Task list — next steps in priority order

1. Get UrlResolver #661 green and merged (user-approved to merge when green)

• It has not been re-run since 2026-06-30. First: rebase on latest origin/main (it will be BEHIND), force-push (--force-with-lease; the branch is solely mine, last commit 1b346e0 "Re-trigger CI"), which re-triggers CI.
• Do this only after the host is stabilized (task 2 / the fleet-redeploy handoff), or it will just hit the same 120 s cold-start flake. When buildtest is stable and un-swapped, the cold-start succeeds in seconds and both checks pass (this is exactly how #228 finally merged — it caught a stable window).
• Merge mechanism (BuildTestEmbedded and UrlResolver both use GitHub native merge queue; gh pr merge --auto does NOT enqueue — it only sets auto-merge and silently no-ops). Enqueue explicitly: gh api graphql -f query='mutation($id:ID!){enqueuePullRequest(input:{pullRequestId:$id}){mergeQueueEntry{state position}}}' -f id="<PR node id>" (get the node id from gh pr view <n> --json id).
• The merge-queue CI runs on the same buildtest infra, so it can be dequeued by the same flake — watch it and re-enqueue if evicted (this cost #228 two dequeues before it landed in a stable window). Only required check is kotlin.build (remote).

2. Durable fix for the getBuildRun RPC flake — fail-fast on an unreachable url:// peer

This is the systematic fix the user asked for. It is resolver/protocol transport work, not a buildtest or getBuildRun change (getBuildRun is already O(1); "caching" it does nothing).

• Goal: a sandboxed RPC / P2P cold-start to an unreachable or just-restarted service must fail fast (bounded, well under the 120 s watchdog and ideally under a few seconds) with a clean, retryable error — instead of blocking in connection setup until the watchdog kills it. A dead/unreachable peer is a "waiting for something that will never answer" hang; per the global rules that is a bug to fix at the transport, not a timeout to raise.
• Start in UrlResolver/src/foundation/url/resolver/sandbox/SandboxedConnection.kt (the rpcHandler blocking call that gets interrupted) and the resolver's connect/resolve path for a url:// service. Determine where the 120 s is actually spent (resolution retry loop? relay/Noise handshake? a connect with no bounded timeout?). Add a bounded, fast failure there. Do NOT raise the watchdog (global rule) — make the underlying connect fail fast instead.
• This overlaps the other handoff's leak/protocol fixes ([UrlResolver #669 SandboxedConnection.close() leak, merged]; [UrlProtocol #326 yamux backpressure, merged]). Read 2026-07-05-urlresolver-race-and-leak-bug-bundles.md first — part of this may already be addressed by the fleet redeploy (fixed libs → no swap → cold-start fast). Verify empirically before writing new code: if, after the fleet redeploy, three parallel UrlResolver remote builds go green with zero getBuildRun interrupts, this flake is resolved by stabilization alone and no transport change is needed.
• Whatever you change, follow strict TDD: an initially-failing test that reproduces a cold-start-to-unreachable-service hang, then the fail-fast fix. This is a foundation.url:resolver / foundation.url:protocol change → publish a new resolver/protocol version and it flows into every consumer.

3. Follow-ups (lower priority, all optional / already-noted)

• The 22.5-min testNetlabDynamicFailureRecovery (UrlResolver/tests/testNetlabDynamicFailureRecovery.kts, @Timeout(2_400_000)): no clean single-line fix — its cost is structural (3 sequential failure scenarios + a fresh netlab droplet per test + first-apply image pull). #228 already gives it its own shard. The real latency lever is pre-baking the eclipse-temurin images into the netlab droplet snapshot (a NetLabManagerServer change) — see NetLab-workstream-handoff-2026-07-05.md.
• Possible CN leak: at investigation time CN's live old-gen was ~181 MB after full GCs (> the 128 MB budget). If a fresh -Xmx128m CN OOMs within days, capture its heap dump (RestartCli wires -XX:+HeapDumpOnOutOfMemoryError, path /root/ContainerNursery/heapdumps) and root-cause the leak. Never bump -Xmx.
• Host capacity / fleet redeploy is the umbrella fix — see the ci-stabilization handoff's "Task 23: fleet redeploy with fixed libraries."

Key pointers / commands

• Host: ssh -p 23 root@198.199.106.165 (key ~/.ssh/id_ed25519). Forensics used: per-PID VmRSS/VmSwap from /proc/<pid>/status, jstat -gcutil <pid>, full cmdline from /proc/<pid>/cmdline, free -m.
• CN restart (correct, budget-honoring): coursier launch community.kotlin.ssh.restart.containernursery:ssh-restart-containernursery:0.0.5 -r https://kotlin.directory -M community.kotlin.ssh.restart.containernursery.RestartContainerNurseryKt -- --host 198.199.106.165 --port 23 --user root --key ~/.ssh/id_ed25519. Use 0.0.5 (0.0.3=-Xmx384m, 0.0.4=-Xmx1g).
• CN CLI: coursier launch containernurserycli:container-nursery-cli:0.0.16 -r https://kotlin.directory -- --url https://api.nursery.wasmserver.com <containers|health-detailed|container-logs --route-key ...>.
• Auto-memory files with deeper detail (in ~/.claude/projects/-code/memory/): cn_no_xmx_host_swap_starves_buildtest.md, urlresolver_656_serialize_once_and_meshleak_guard_flakiness.md, netlab_stream_closed_sjvm_watchdog_interrupt_leak.md.

Snapshot at write time (2026-07-05, verify before trusting)

• Host 198.199.106.165: Mem 3403/3919 MB used, 279 MB available, 503 MB swap. CN -Xmx128m (fix held). buildtest-server-service RSS 295 MB, restarted ~12 min prior (still occasionally cycling under pressure).
• #228 MERGED. #661 OPEN, both checks FAILURE (last run 2026-06-30, SandboxException infra error), head 1b346e0, BEHIND main.