Handoff: buildtest orchestration resource-cost workstream — virtual-threads stopgap in flight, with the wider plan to eliminate the persistent per-shard SSH connection via a url:// runner service

Written: 2026-07-11 (workstream began 2026-07-10).

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below (PR states, merge status, whether the virtual-threads PR exists/merged, host load) was true only at write time and goes stale fast. Re-check before doing anything:

• PRs: gh pr view <num> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• The in-flight virtual-threads PR: gh pr list --repo CodexCoder21Organization/BuildTestEmbedded --state all --search "virtual thread OR loom OR async orchestration"
• Host/infra: hit buildtest.kotlin.build and, if needed, ssh -p 23 root@198.199.106.165 'cat /proc/loadavg' (be gentle — heavy grep -r over /root/buildtest-data/runs/* once ballooned to 10 GB and tripped the kernel OOM-killer). A fresh query always wins over this document.

2026-07-12 correction (read this before the rest)

Since this handoff was written, the whole resource-cost mission has effectively landed — and a re-verification found the original forensic's headline DO-rate-limit claim was wrong (it was scoped to BuildTestEmbedded and never inspected the droplet service):

• The DO rate-limit / droplet-quota problem is solved by the DO gateway, not by anything left to do here. DigitalOceanDropletServiceServer #86 ("DO gateway: rate limiter + TTL cache + coalescing; sole API-key holder") + #287 ("DropletManager via DO gateway, no direct DO calls") mean getDroplet is served from a shared ~5 s cache and the gateway self-caps upstream calls (~3,600/hr). Per-shard getDroplet polling does not — and cannot — blow the DO 5,000/hr limit. The "shared readiness reconciler" this handoff called the highest-impact next step already exists.
• The thread explosion is solved by the virtual-threads stopgap, now MERGED as BuildTestEmbedded #304.
• Admission is covered by the merged redesign arc (#298, #288, #296, #300, #306).
• What genuinely remains: the url:// runner-service migration that eliminates the persistent per-shard SSH transport (the real structural target, still open — see Next steps #6), plus a minor optional analyzeArchive caching win (Next steps #4). The acute meltdown causes are resolved.

Mission summary

1. Original request: drive ContainerNursery PR #479 (the startup tmp-dir sweep fix) to completion — the open thread of an earlier handoff.
2. Landing #479's CI surfaced a systemic buildtest-infra meltdown: shared build host at load ~80, ~64 concurrent buildtest runs, DigitalOcean droplet-count quota and 5,000-requests/hour API-rate both exhausted, and the digitalocean-droplets service OOM restart-looping — failing every kotlin.build (remote) run org-wide.
3. The user directed fixing the structural causes, not papering over them with a concurrency cap: "the underlying problem occurred without any such limit so clearly something more structural is wrong." That expanded the work into a forensic analysis of the orchestration and three merged structural fixes, plus this handoff to preserve the wider architectural plan so it is not lost after the interim virtual-threads stopgap.

What was found and done (the chain)

1. Three structural fixes — all MERGED this session (evidence-based, each with tests):

• ContainerNursery #479 — the startup temp-dir sweep no longer deletes container-nursery-jar-* dirs that are in use by a live process (it skips any dir whose path appears in a live process's /proc/<pid>/cmdline; a running container's child JVM carries -Djava.io.tmpdir=<that dir>). Was a production hazard, not just a CI flake.
• GithubWatchmanServiceServer #47 — the kotlin.build webhook receiver was leaking ~663 callback-executor threads via an unbounded Executors.newCachedThreadPool in WatchmanService.fireCallbacks (callbacks block behind a serialized UrlResolver RPC, so a new thread spawns per callback). Replaced with a bounded pool + bounded queue + CallerRunsPolicy backpressure.
• BuildTestEmbedded #298 — fleet admission was disabled-by-default (BUILDTEST_DROPLET_LIMIT unset → dropletLimit=0) and, when enabled, called listDroplets() (a live RPC to the overloaded droplet service) on every admission check. Now admission uses in-process reservation accounting as the fast-path authority, reconciles the real droplet list only periodically off the hot path, seeds reservations for restart-recovered runs (they previously bypassed admission), and logs the effective caps at startup. (The user separately set BUILDTEST_DROPLET_LIMIT in prod.)
• Infra incident recorded as a challenge: challenges/2026-07-10-1529-digitalocean-api-rate-limit-and-droplet-quota-exhaustion.md.

2. Forensic profiling (measured by codex gpt-5.6-sol with a throwaway harness; numbers are measured, not speculative):

Problem 2 — thread-per-shard blocking orchestration (the dominant, incident-scale problem). BuildTestEmbeddedService.kt:2820 does shardSpecs.mapIndexed { Thread { runShardPipeline(...) } } (one blocking platform thread per shard, joined by the orchestrator). runShardPipeline runs the whole lifecycle synchronously — provision → SSH connect → install → stream compile/test → collect → destroy — blocking on socket I/O and Thread.sleep poll loops the entire time, holding one long-lived JSch SSH connection per shard.

• Threads/run ≈ 1 + S explicit, ~1 + 2S including JSch's SSH threads, ~1 + 3S during a lease-renewal burst. At 64 runs × S=10: ~704 explicit / ~1,344 incl. JSch / ~1,984 during renewal — nearly all parked on blocking I/O (native stacks + scheduler cost).
• DigitalOcean API: DropletManager.waitForDropletReady polls getDroplet every 10 s per shard (~1 + ⌊T/10⌋ GETs/shard for readiness time T). At S=10 / 90 s readiness that is ~7,680 GETs across 64 runs — crossing the 5,000/hour DO limit in ~70 s from polling alone, before lease renewals (~3,840/hr) and watchdog checks. (Pre-#298, admission additionally did an account-wide listDroplets per capacity check.)

Problem 1 — analyzeArchive under the global lock (secondary). analyzeArchive (BuildTestEmbeddedService.kt:5596) runs under a process-wide synchronized(ARCHIVE_ANALYSIS_LOCK). Measured: PSI parsing is 96–97 % of the cost; ~238 ms cold IntelliJ-env init, then 0.5–2.5 ms/file, linear in file count (~0.08–0.22 s warm for 50–200 files). It is not O(files×tests) and does not re-parse all files per test (a hypothesis that was explicitly disproven — it loops once per file). The global lock is a genuine correctness requirement: the kompile-buildscript PSI parser uses a JVM-global IntelliJ application environment that corrupts under concurrent use (real prod symptom: garbled project names / "Corrupt GZIP trailer"). Remaining waste is second-order: a fresh, empty BuildscriptCache per call (no cross-call/run caching), duplicate helper functions, and recovery re-analysis.

3. Interim STOPGAP — virtual threads (in flight, being merged): codex gpt-5.6-sol is migrating the per-shard, per-run-orchestrator, and lease-renewal threads in BuildTestEmbedded from platform threads to Java 21 virtual threads (Loom) — keeping the exact blocking code but collapsing the ~1,344 platform threads onto a small carrier pool. Includes a carrier-pinning audit (-Djdk.tracePinnedThreads=full; convert any synchronized-around-blocking to ReentrantLock) and a fail-first test asserting shard pipelines run on virtual threads. The user authorized merging this PR. This is an interim reduction of the thread cost — not the target architecture.

The wider plan (the point of this handoff — do not lose it)

The real fix is to eliminate the persistent per-shard SSH connection used to monitor builds. A held-open JSch SSH connection per shard, with a blocking read thread streaming output, is why the orchestration is thread-per-shard and blocking; virtual threads only make that cheap, not unnecessary. There is no need for the orchestrator to hold a socket into each droplet.

Target shape: the droplet runs a url:// runner service and streams build/test results back over the negotiated UrlProtocol/UrlResolver transport, instead of the orchestrator SSH-ing in and holding a connection. This is already the shape of the plan — see:

• workstreams/ExecutionEnvironments.md §C "The test-runner service contract and the default pool": define a TestRunnerApi url:// contract; make url://buildtest/ the default implementation so droplets become runner nodes that pull prepared tests and return/stream results; replace static sharding with a work-stealing scheduler, retiring the per-droplet --test allow-lists in BuildTestEmbedded.
• workstreams/KompileRemoteBuild.md: the remote-build optimizer (capacity planning + admission queueing) and "build-watching absorbed into the CI service", which sizes capacity around that scheduler.

The gap to close: ExecutionEnvironments.md §C does not yet explicitly state that the url:// runner contract retires the persistent SSH monitoring transport and the thread-per-shard blocking model, nor that virtual threads are the sanctioned interim stopgap until it lands. Recommended plan refinement (needs review — do NOT auto-merge into the aspirational plan): add a concise, high-level bullet to ExecutionEnvironments.md §C making that explicit and cross-linking the interim virtual-threads reduction. Keep it plan-level (shape/objective), not implementation detail.

Relevant PRs / refs (write-time state — RE-VERIFY)

| Item | State @ write time | Link / ref | |---|---|---| | ContainerNursery #479 — tmp-dir sweep skips in-use dirs | MERGED 2026-07-11T11:57:25Z | https://github.com/CodexCoder21Organization/ContainerNursery/pull/479 | | GithubWatchmanServiceServer #47 — bounded callback pool | MERGED 2026-07-11T09:26:19Z | https://github.com/CodexCoder21Organization/GithubWatchmanServiceServer/pull/47 | | BuildTestEmbedded #298 — robust/default-safe admission | MERGED 2026-07-11T16:02:10Z | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/298 | | BuildTestEmbedded #304 — virtual-threads (Loom) orchestration | MERGED 2026-07-11T23:42:41Z | https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/304 | | DigitalOceanDropletServiceServer #86 / #287 — DO gateway (rate limiter + TTL cache + coalescing) — this is the "shared readiness reconciler" | MERGED (2026-07-10) | #86 · #287 | | Infra challenge (DO quota/rate-limit exhaustion) | filed | challenges/2026-07-10-1529-… | | Wider-plan homes | — | ExecutionEnvironments.md §C · KompileRemoteBuild.md |

Next steps (ordered — start by re-verifying)

See the 2026-07-12 correction at the top of this document first. The original "Follow-up #1 (shared readiness reconciler)" is already done (the DO gateway), and the acute DO-rate / droplet-quota / thread problems are resolved. The list below is the corrected remaining work.

1. Re-verify the current state (all PRs merged? current main HEAD? the DO gateway present in DigitalOceanDropletServiceServer?).
2. ~~Shared readiness reconciler~~ — DONE. The DO gateway (DigitalOceanDropletServiceServer #86: rate limiter + ~5 s TTL droplet-list cache + request coalescing, sole API-key holder) plus #287 ("DropletManager via DO gateway, no direct DO calls") already provide this: getDroplet/listDroplets are served from the shared cache and the gateway self-caps upstream at ~3,600/hr, so per-shard readiness polling cannot blow the DO 5,000/hr limit. (The original forensic mis-attributed this because it was scoped to BuildTestEmbedded and never inspected the droplet service.)
3. ~~Batch lease-renewal / admission~~ — largely DONE via the admission-redesign arc (#298, #288, #296, #300, #306). Re-verify against current main before assuming any of it is still open.
4. (minor, optional) analyzeArchive caching. Still builds a fresh empty BuildscriptCache per call (new temp dir; ~4 call sites) with no cross-call/digest cache. Digest-keying the ArchiveAnalysis result (or reusing one long-lived PSI environment) would skip re-parsing on recovery / identical resubmits and shorten the global-lock hold. Low priority — measured sub-second, one analysis per normal submit.
5. (plan) ExecutionEnvironments.md §C refinement — explicit SSH-elimination + interim-stopgap note, as a reviewable plan PR (human eye; do not auto-merge).
6. TARGET (the real remaining structural work): the url:// runner-service migration (ExecutionEnvironments.md §C). Persistent SSH per shard is still present (ssh.connect() held for the shard lifetime) — #304 only made those threads cheap (virtual threads); it did not remove the SSH transport. Making droplets url:// runner nodes that stream results over the negotiated transport is the durable fix that makes thread-per-shard moot. Large, multi-repo; design with the user before starting.

Reusable / operational knowledge

• kotlin.build (remote) is the only required merge-queue check in these repos (required_status_checks: ["kotlin.build (remote)"]); GHA jobs (bld-all-tests, etc.) are informational for the queue. gh pr merge --auto is rejected — enqueue via GraphQL enqueuePullRequest.
• Merge-queue evictions are usually infra, not code. A PR whose head is green but that keeps falling out of the queue (OPEN/CLEAN, no gh-readonly-queue/main/pr-<n> ref, not merged) is almost always the queue-ref kotlin.build (remote) flaking on the shared host (droplet quota / kotlin.directory Connection reset downloading kompile-cli / droplet-service restart gap). Fix by re-enqueuing during a clean host-load window, not by touching the code.
• Do NOT run heavy grep -r over /root/buildtest-data/runs/* on the build host — one such sweep grew to 10 GB and tripped the global OOM-killer. Prefer the buildtest HTTP API (/api/test-results?id=<run>, /api/build-rules?id=<run>) and single-run stat/tail.
• Diagnose infra-vs-code on a kotlin.build (remote) failure by grepping that one run's build.log for provisioning_failure|droplet limit|rate-limit queue|No service registered|shard_orchestration_interrupted (infra) vs BUILD FAILED|COMPILATION ERROR|AssertionError (code).
• codex gpt-5.6-sol did the implementation/profiling: codex exec --dangerously-bypass-approvals-and-sandbox -m gpt-5.6-sol -C <repo> -o <out> "<brief>". It correctly stopped rather than guessing when a reproduction wasn't reliable (e.g. the concurrent-run cap, blocked by the global analyzeArchive startup serialization) — trust that signal.