BuildTestEmbedded full-suite verification on current main 3f3acdd plus the dynamic-dispatch

Reported (UTC): 2026-07-16 11:37

BuildTestEmbedded full-suite verification on current main 3f3acdd plus the dynamic-dispatch command fix exposed three deterministic non-hermetic fake-droplet tests and one suite-only concurrency failure. e2eShardedRunRecordsDistinctPerDropletPhaseTimestamps and provisioningRetryReplacedDropletKeepsOnlyProvisioningStamp each fail alone because BuildTestEmbeddedService defaults to dynamic dispatch and remoteWorkRoot=/root; their FakeDropletService interceptors do not handle resetWorkQueue's rm command and their service constructors omit remoteWorkRoot="root", so the fake SSH command falls through onto the non-root host and fails exactly with "rm: cannot remove '/root/btr-work-queue-<run>.jsonl': Permission denied". dropletTestingPhaseStampIsFirstEntryNotRewritten has the same constructor defect and terminates before observing TESTING, reporting that no TESTING phase stamp was observed. Full stack traces point to assertions at e2eShardedRunRecordsDistinctPerDropletPhaseTimestamps.kt:189, provisioningRetryReplacedDropletKeepsOnlyProvisioningStamp.kt:170, and dropletTestingPhaseStampIsFirstEntryNotRewritten.kt:166. e2eShardedRestartReadmissionAndPartialAllocationFailure also exited 1 only during the parallel full suite but passed immediately alone, indicating suite-level interference/flakiness. Impact: local full-suite verification as a non-root user reports 588/592 with four failures even though the mechanism-focused tests all pass, and the first three fixtures accidentally touch the host filesystem instead of remaining hermetic. Durable fix: update those fake-droplet BuildTestEmbeddedService fixtures to pass remoteWorkRoot="root" (as modern neighboring E2Es already do), reproduce the restart/readmission concurrency failure at >=50% with stress before changing it, and add a test invariant preventing fake SSH commands from falling through to absolute host paths. No timeout or iteration workaround was used.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.