BuildTestEmbedded PR CI for the DEFAULT_IMAGE warmcache-v3 pin was repeatedly invalidated by the

Reported (UTC): 2026-07-15 19:26

BuildTestEmbedded PR CI for the DEFAULT_IMAGE warmcache-v3 pin was repeatedly invalidated by the unrelated test e2eCancelResumedBuildProducesCanceled. In run 3d734d7f, after cold-cache compilation had progressed to 106 passed / 428 pending with no image-pin assertion failures, the test failed with exact error . The same test had timed out in an earlier rerun 72d4fc8f, forcing repeated deletion and re-request of the entire 535-test, 10-shard check; each rerun spends roughly 20–24 minutes compiling because production CI still boots the old uncached image 236908613. Impact: a deterministic image-pin PR cannot obtain a green check without gambling on an unrelated load-sensitive cancellation test, wasting droplet capacity and operator time. Workaround used: confirm the full persisted test-results.json to distinguish product assertion failures from the unrelated timeout, delete only the doomed run, warm github-webhooks.wasmserver.com, and rerequest the check suite. Suggested durable fix: apply the flaky-test forensic workflow to reproduce this timeout at >=50% frequency under equivalent cold-cache/load conditions, identify why line 165 can exceed the fixed 60-second test budget, and fix the underlying cancellation/resume mechanism without increasing the timeout.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.