Handoff: BuildTest droplet-limit admission — hermetic e2e tests + 4 admission bug fixes across 4 PRs (all OPEN, CI red/stuck at write time, driver session died)

Written 2026-07-12 ~17:05 UTC (workstream began 2026-07-10)

⚠️ RE-VERIFY BANNER — all state below is a write-time snapshot and is already partly stale. The driver session and its codex exec worker are dead, CI runs referenced here were mid-flight or failed on infrastructure, and main has moved substantially (six related PRs merged 2026-07-11→12; see the chain). Before acting, re-verify every claim:

• PRs: gh pr view <n> --repo CodexCoder21Organization/BuildTestEmbedded --json state,mergeStateStatus,statusCheckRollup,mergedAt (and same for the BuildTestServerService PR).
• A buildtest CI run's real status: https://buildtest.kotlin.build/run?id=<runId> (the GitHub check can read IN_PROGRESS for a run whose executor died hours ago — see PR 294).
• Merged main state: gh pr list --repo CodexCoder21Organization/BuildTestEmbedded --state merged --limit 12. A fresh query always wins over this document.

---

UPDATE 2026-07-12 ~22:30 UTC — reconciliation done; conclusion shape decided (supersedes the original "Next steps")

A fresh session drove this forward. Main independently landed the same fix series (#288–#308), so two of the four PRs are now largely redundant. A codex gpt-5.6-sol reconciliation pass (rebased each branch onto origin/main, dropped hunks main already implements, ran hermetic tests locally) produced these evidence-backed verdicts:

| PR | Verdict | Detail | |----|---------|--------| | BuildTestEmbedded #294 | KEEP | Rebased + force-pushed 00359cc3. Dropped reservation/fleet-listing hunks redundant with #298/#300; kept the 14 hermetic droplet-limit admission tests + unique src (provider-quota watchdog authority during retries, durable admission-grant timing, persisted shard partitions). Local: 19/19 changed + 13/13 regression pass. CI IN_PROGRESS at write time (run started 21:42 UTC). ⚠️ Retains reserved artifact coord 0.0.276 (not main's published 0.0.267) because kompile dedup discarded the local build otherwise — verify this is sound before merge. | | BuildTestServerService #209 | KEEP | Rebased + pushed 2ec26c2f (downstream VT-carrier-pinning fix, needed because main's #304 moved orchestration to Java-21 virtual threads). CI IN_PROGRESS (run started 22:16 UTC). An earlier run this session (ee8f6dd4) failed — re-verify the new run's outcome. | | BuildTestEmbedded #291 | SUPERSEDED (partial) | Bug A (restart recovery bypasses admission) → main #300; Bug B (quota-422 resume → requeue PENDING) → main #297. Unique survivors worth salvaging: stale-persisted-droplet-ID preflight before treating restart reservations as usable, and persisting PENDING before deletion is externally observable. Reconciled local head preserved at branch salvage/291-reconciled-ae2c77d on origin. | | BuildTestEmbedded #301 | SUPERSEDED (partial) | Bug C → main #296; Bug D (watchdog counts queue-wait) → main #289/#290; execution-timing → main #300's admissionGrantedAt. Only unique survivor: a 15-line ManualClock test-stabilization hunk in e2eShardedMixedProvisioningAndTestingFailureFinalizesStartedTests.kts. Reconciled local head preserved at branch salvage/301-reconciled-c7e0fd1 on origin. |

Revised conclusion plan (replaces the original Next steps below):

1. PR 294 & PR 209 — let CI finish; do the Claude final review (incl. the 0.0.276 coordinate check on 294); then leave for the user to merge (never auto-merge code PRs).
2. PR 291 — decide with the user: salvage the two unique correctness bits (stale-droplet preflight + PENDING-before-cleanup ordering) from salvage/291-reconciled-ae2c77d into a small, comprehensively-tested new PR, then close #291. The salvage material is preserved on origin regardless.
3. PR 301 — its lone unique 15-line test hunk (preserved on salvage/301-reconciled-c7e0fd1) is marginal; fold it into 294 or a tiny test-only PR, or drop it, then close #301.
4. Closing #291/#301 is the only user-gated step; both PRs' unique work is already preserved on the salvage/* branches so closing loses nothing.

Note: the driver session paused here at the user's request (workers/loops stopped). At pause, PR 294 & 209 CI were still IN_PROGRESS — re-verify their outcomes first before the final review.

---

Mission summary

The user's concern, escalated over four messages (verbatim intent preserved):

1. "I'm concerned that the build server is currently setup to have a maximum number of droplets of 100, but this limit may not be robustly tested in a hermetic environment (do not start real droplets, use an in-memory implementation that simulates droplets/builds). We need comprehensive e2e tests of these edge condition behaviors, or else things fall apart when they are most needed."
2. "Ultrathink are the tests comprehensive? Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."
3. Worry specifically about droplet admission control: "if the droplet allocation is full then new requests should queue instead of breaking." Established that BUILDTEST_DROPLET_LIMIT is a real concurrency cap — waitForAdmission() blocks a run until activeDroplets + reservedDroplets + requestedDroplets <= dropletLimit, and <= 0 disables admission entirely (unlimited). Source default is 0.
4. "We should fix those bugs. We should have hermetic tests (do not hit real digital ocean, simulate droplets in memory) which verify all this functionality works properly."

Work followed the global CLAUDE.md division of labor: codex gpt-5.6-sol (high reasoning) did all implementation, tests, reviews, and CI-shepherding; Claude only defines problems and does the final review (findings routed back to codex, never patched by Claude). Never merge these PRs — the user merges. Tests are hermetic (FakeDropletService/in-file DropletService fakes + ManualClock); no real DigitalOcean, no mocks.

What was found and done (the chain)

~11 real defects were found and fixed across the session (verified by reproducers that failed before the fix). The production trigger: url://buildtest/ had no BUILDTEST_DROPLET_LIMIT set (=0=admission disabled) until 2026-07-10; the only cap was DigitalOcean's account quota (422s). See memory buildtest-droplet-limit-config (history UNSET→100→85→65; the live value is now 65 in /root/ContainerNursery/config.json, set by a different lane for DO-account oversubscription).

The fixes were split across four PRs (three in BuildTestEmbedded, one downstream in BuildTestServerService):

1. Wake-on-release convoy (PR 294): under saturation, freed capacity idled up to 5 min because waitForAdmission slept on a backoff instead of being notified on release. Fixed with a lock-protected wait/notify + a clock.scheduled fallback notify; also fixed reservation double-counting (consume the reservation exactly when the droplet becomes provider-visible) and a missed-fallback-notification race.
2. Restart recovery bypassed admission (PR 291, "Bug A"): recovered runs re-provisioned without going through waitForAdmission. Quota-422 during resume → permanent FAILED ("Bug B"): resume-time DO quota errors now requeue as PENDING via a shared allocation-failure helper. Recovery now sorts oldest-first and enqueues admission placeholders; shard plans persisted across restart; readmission state made observable atomically.
3. Cancel-while-queued zombie + watchdog reaping queued runs (PR 301, "Bugs C+D"): Bug C (cancel while in the admission queue → permanent PENDING zombie) was found already fixed on main — not duplicated. Bug D (watchdog counted queue-wait toward the 3h max-build-duration, reaping queued runs during saturation) fixed by budgeting max-duration from admission, not submission (new duration-basis timestamp, backward-compatible JSON). Plus unlimited-mode stale-placeholder and zero-delay-busy-loop watchdog fixes.
4. Virtual-thread carrier pinning during droplet creation (PR 209, BuildTestServerService — downstream): surfaced during the PR 301 adversarial review after main merged #304 ("Run build orchestration on Java 21 virtual threads"). The remote-create serialization lock could pin both VT carriers. Reproducer: tests/idempotentDropletProvisioningDoesNotPinVirtualThreadScheduler.kts (a scheduler-sentinel VT must still be schedulable while a create is gated).

Why this is being handed off now (verified):

• The codex exec wrap-up worker (PID 506256, ~12h old) died sometime after 05:08 UTC. No codex exec process is alive (ps confirms). Its log froze at 451 MB.
• The status-report cron loop is gone (CronList → "No scheduled jobs"), which is why updates stopped between ~05:08 and ~17:00 UTC.
• All four git workspaces are clean — nothing uncommitted or unpushed. Every branch matches its PR head. No work was lost. (git status clean + git log @{u}.. empty in each of /code/workspace/BuildTestEmbedded, BuildTestEmbedded-bugfix, BuildTestServerService-aux-fix; BuildTestEmbedded-main-diagnostic is a detached read-only checkout at ba22f32.)
• The PR 301 review amendments codex spent hours drafting (the carrier-pinning test, failed-boot droplet hardening) did NOT land on PR 301's branch — PR 301 head is still b68acd34 (its last commit is "Preserve admitted execution timeout across restart"). The carrier-pinning test instead became PR 209 in the downstream repo. Whether the failed-boot droplet hardening I watched being drafted in the log was committed anywhere is unverified — grep for it before assuming PR 301 is complete (see Next steps).

Main has diverged significantly (merged since these branches were cut — re-check, several overlap these PRs' subject matter):

• #308 "Requeue builds blocked by DigitalOcean rate-limit exhaustion instead of hard-failing" (merged 2026-07-12 10:17Z) — overlaps PR 291's quota-requeue-on-resume directly; PR 291 may now be partly redundant or conflicting.
• #306 (per-run droplet lifecycle records, 0.0.266), #304 (Java 21 virtual threads — the cause of PR 209), #303, #300 ("Honor droplet admission across restart and queue wait"), #298 ("Make fleet admission robust and default-safe"). All touch admission.

Relevant PRs / refs (write-time state — RE-VERIFY)

| PR | Branch / head | CI (write-time) | Notes | |----|---------------|-----------------|-------| | BuildTestEmbedded #294 | droplet-limit-e2e-tests @ d22ae7b3 | STUCK — run e4b18150 IN_PROGRESS since 03:50 UTC (13h+ = lost/zombie dispatch) | 13 hermetic scenarios + wake-on-release/reservation/race fixes. Never got a green verdict. Prior attempts failed on infrastructure (DO account rate-limit exhaustion; run d8130f89 = "0 passed, 0 failed, 528 total"). | | BuildTestEmbedded #291 | fix-restart-recovery-droplet-admission @ 74771f73 | FAILURE — run a10cb66a at 06:22 UTC | The 4 post-rebase test failures were addressed by 74771f73 ("Make restart readmission state observable atomically"); this run then failed on a different infrastructure error: SandboxException: SJVM virtual method invocation timed out after 120s: BuildTestServiceClientImpl.getTestResults. Needs a re-run, not (necessarily) a code fix. | | BuildTestEmbedded #301 | fix-queued-run-cancellation-and-watchdog @ b68acd34 | SUCCESS (run from 2026-07-11 17:17Z) | Green but stale — never rebased onto current main (was DIRTY; now UNKNOWN). Codex's intended review amendments did not land here (see chain). | | BuildTestServerService #209 | fix-virtual-thread-droplet-create-pinning @ 87867ba | FAILURE — CI at 02:47 UTC | "Prevent droplet creation from pinning virtual-thread carriers." Failed CI predates the DO rate-limit reset — likely infrastructure; re-verify the failure category. |

Supporting context in session memory (~/.claude/projects/-code/memory/): buildtest-droplet-limit-config, codex-gpt55-invocation, merge-queue-eviction-playbook, w3walletdaemon-merge-path.

Next steps (ordered — re-verify current state FIRST)

1. Re-verify everything with the commands in the banner. Especially: is PR 294's run e4b18150 still IN_PROGRESS, or did it finally resolve? A 13h+ run is a lost dispatch — do not wait on it.
2. Decide relevance vs. merged main before spending CI. Because #308/#306/#304/#303/#300/#298 merged, re-assess whether PR 291 is still needed or now conflicts/overlaps with #308's quota-requeue. Rebase each of 294/291/301 onto latest origin/main, resolve conflicts faithfully, and re-run each branch's own new tests locally (export PATH="$HOME/bin:$PATH" for working coursier; scripts/test.bash). This is codex's job per the division of labor — dispatch codex gpt-5.6-sol (high reasoning) per the codex-gpt55-invocation memory; Claude does only the final review.
3. PR 294: re-request a fresh CI run on a healthy builder (the stuck one is dead). Prior real failures were infrastructure (DO rate-limit), not the code. If it ever fails on the specific stream-read trio (see the diagnosis file), consult scratchpad/pr294-ci-failure-diagnosis.md (TailingInputStream liveness hypothesis) — do not weaken tests/timeouts.
4. PR 291: re-run CI (last failure was an SJVM sandbox 120s timeout — infrastructure). If it recurs deterministically, treat the getTestResults RPC timeout as a real defect to root-cause, not to paper over.
5. PR 301: confirm what its review amendments should be. Grep first: git log --all --oneline -S 'scheduler-sentinel' and grep -rl 'failedDropletDeletionWaiters' /code/workspace/*/ to confirm whether the failed-boot droplet hardening codex drafted was committed anywhere or was lost when the worker died. Then rebase and re-run.
6. PR 209 (BuildTestServerService): re-verify the CI failure category; re-run if infrastructure. This is the downstream fix for the VT-pinning bug introduced by main's #304.
7. Do NOT merge any of these four — they are code PRs; the user merges. (This handoff PR itself is the only pre-approved merge.)
8. Restart the status-report loop only if the user still wants it (/loop 20m /status-report).

Reusable / operational knowledge

• codex gpt-5.6-sol invocation + failure modes: memory codex-gpt55-invocation (exact codex exec ... --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check command; always < /dev/null + task-in-a-file to avoid the stdin hang and the pkill -f self-kill; model-capacity crashes leave the tree intact — resume from git status).
• A green GitHub check is not a live run. buildtest's GitHub check can sit IN_PROGRESS for many hours after the executor droplet died. Always cross-check https://buildtest.kotlin.build/run?id=<runId>; treat >~2.5h as lost and re-request.
• Distinguish infrastructure failures from code failures before touching code: "0 passed, 0 failed, N total" = suite never ran (provisioning/quota); SandboxException: SJVM ... timed out after 120s = foundation.url sandbox flake; explicit named test failures = code. Only the last warrants a code change.
• DO account rate-limit was the dominant CI-failure cause this session (shared account, POST /v2/droplets ... ratelimit-remaining=0). Re-requesting after the window resets (the run page prints provider reset=<ts>) is the remedy; a timed rerequest via gh api -X POST repos/.../check-runs/<id>/rerequest works.
• Working coursier on the /code box: export PATH="$HOME/bin:$PATH" (the /usr/local/bin/coursier is wrong-arch — memory coursier-broken-on-code-box).
• Workspaces used: /code/workspace/BuildTestEmbedded (294), /code/workspace/BuildTestEmbedded-bugfix (currently on 291's branch), /code/workspace/BuildTestServerService-aux-fix (209), /code/workspace/BuildTestEmbedded-main-diagnostic (read-only main checkout). Scratchpad task files + the pr294-ci-failure-diagnosis.md are under /tmp/claude-1000/-code/40d56168-7a5d-423f-a0f5-7cb52b42f4d4/scratchpad/.