BuildTestEmbedded full local suite on commit 573875e (fresh branch rebased onto origin/main)

Reported (UTC): 2026-07-16 09:08

BuildTestEmbedded full local suite on commit 573875e (fresh branch rebased onto origin/main) deterministically failed three existing FakeDropletService E2E tests because their command interceptors do not handle the dynamic-dispatch reset command 'rm -f /root/btr-work-queue-<run>.jsonl'. FakeDropletService therefore executed that absolute-root command on the local host as user helena, producing 'rm: cannot remove ...: Permission denied'. e2eShardedRunRecordsDistinctPerDropletPhaseTimestamps and provisioningRetryReplacedDropletKeepsOnlyProvisioningStamp then failed terminal-status assertions; dropletTestingPhaseStampIsFirstEntryNotRewritten terminated before observing TESTING. All three reproduced in an isolated 4-test run. Impact: the suite is not hermetic and passes/fails based on host UID, obscuring unrelated regressions. Workaround for this task: keep the production change scoped, verify its focused tests, and rely on remote root-run CI. Durable fix: each FakeDroplet command handler should explicitly emulate the work-queue reset/append commands in its per-droplet temp root (or FakeDropletService should translate absolute remote paths consistently) instead of falling through to the host shell.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.