BuildTestEmbedded PR CI run de5f3823 was invalidated by the unrelated load-sensitive test

Reported (UTC): 2026-07-15 19:52

BuildTestEmbedded PR CI run de5f3823 was invalidated by the unrelated load-sensitive test replayTestEventLogStreamsLargeEventLogUnderTightHeap. After 159 tests had passed and no DEFAULT_IMAGE assertions had failed, it produced: java.lang.RuntimeException: Test 'replayTestEventLogStreamsLargeEventLogUnderTightHeap' timed out after 30000ms, stuck at buildtest.embedded.ReplayTestEventLogStreamsLargeEventLogUnderTightHeapKt.replayTestEventLogStreamsLargeEventLogUnderTightHeap(replayTestEventLogStreamsLargeEventLogUnderTightHeap.kt:128). The same test had timed out in earlier runs bd4dec01 and 31daf6b1. Impact: repeated full 535-test, 10-shard CI reruns on old cold-cache image 236908613 consume 20–24 minutes of compilation before this unrelated tight-heap test probabilistically times out, preventing a deterministic image metadata PR from obtaining green CI. Workaround: verify the full persisted test-results.json, delete the doomed run, and rerequest the suite. Suggested durable fix: use the flaky-test workflow to amplify this failure under equivalent shard load and tight heap until it reproduces reliably, then fix the underlying event-log replay performance or memory mechanism without increasing its 30-second timeout or reducing its workload.

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.