Handoff: Review the readiness fixture premise and continue module-cache timeout diagnosis

Written 2026-10-09 08:29 UTC.

RE-VERIFY: this is a write-time snapshot. Check the remote diagnostic branch with git ls-remote and re-read main. Use gh run view and gh run download for each linked Actions run. No fix PR exists. Do not merge this branch: its readiness diagnostic is deliberately red.

Mission

The user requested execution of bteflake1: investigate two cross-PR first-attempt flakes in BuildTestEmbedded, reproduce each at >=50% before any fix, then mutation control and 20/20 first-attempt green plus 3/3 originals. Stop an item that cannot reach the reproduction threshold within 40 minutes; hard deadline 85 minutes. Standing rules prohibit assertion weakening/deletion, timeout increases, iteration reductions, CPU spinners, pin changes, merges/enqueues, deployment and production-host access. The investigation started 07:44 UTC and applied the cache stop rule at 08:24 UTC. This belongs to the BuildTest utilization/startup-compaction campaign mentioned in the brief; it did not claim or modify the campaign's larger handoff.

What was found and done

1. Read the complete fix-flakey-test skill, Testing Architecture and Engineering Philosophy, standing brief, project overview/build/testing and relevant README contracts. No project AGENTS.md exists. Actual pin 0.0.112 only; later common rule supersedes the older dual-pin request. Repository README recommends CPU spinners, which contradicts current rules; none were used.
2. Read supplied first-attempt XML from https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37882967093 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37884015361. Preserve selected exact assertion traces and relevant timeout thread blocks on the evidence branch.
3. Readiness fixture holds only beforeTestResultProjectionRebuild reason startup recovery. The independent run-resume worker calls rebuildTestResultsProjection for terminal evidence after service restart and legitimately publishes a complete pending snapshot first. Holding one writer does not imply that a committed snapshot is missing. Original remote test failed first attempt 1/10 (11187ms), retry passed; original local passed1/1. Controlled ordering retains the original assertion and makes the valid snapshot publication occur first: remote exact-assertion failure10/10 first attempts plus local failure1/1. Independent review agrees the fixture premise is false, without proving every historical CI failure came from that writer.
4. Strengthened diagnostic explicitly proves both public full/paginated result reads complete with pendingTest while startup replay remains held, then fails the unchanged original missing-snapshot assertion (local first attempt11737ms). Thus no public-read liveness bug is demonstrated. The common brief explicitly says to stop with evidence if an assertion is wrong; no original assertion or product behavior was changed. This is the readiness item's evidence-backed refutation.
5. Cache-family baseline first-attempt failures0/30: ten each ByteBoundEvictsAcrossReleases, ImageChangeMisses and ProfileChangeMisses, no retries. Concurrent first sample durations105797/99004/98607ms; later cases78–92s. Isolated fresh java.io.tmpdir cold Image samples pass3/3 first attempts in41636/40353/40522ms; ordinary local Image passes1/1 in51.7s using an already existing seed. Cold-cache state alone does not reproduce or explain the reported timeout on this runner.
6. The brief's scheduled/lost-poll claim is unsupported: TailingInputStream directly waits200ms on the reading thread, then rechecks file/status; no scheduled polling pool exists. close sets a volatile flag and signals under the same lock. Supplied failure dumps catch preparation child-output reading or installation cp work while the main log reader is normally polling. Some last-stage children are only1–8seconds old at the overall120s deadline. Actual timeout cause remains unestablished. Do not replace this with a generic machine-speed explanation.
7. Added a separate unchanged-product diagnostic preserving all29 original Image assertions and120s timeout, with64 genuine public streamBuildLog followers per real cache-miss build and explicit close/join cleanup. It did not finish compilation within the300s local command limit; no body result or rate credit. An earlier aborted pre-rebase launcher left a timeout child group alive during rebase, which produced a0ms FileNotFoundException scanning .git/rebase-merge. That compiler result is excluded, not a cache flake. Do not signal only a wrapper group when canceling a command whose timeout child has its own group.
8. Stopped cache item at required40-minute limit with rates below50%, without any speculative fix. Mutation/20-of-20 green/3-of-3 original-green/PR-CI gates do not exist. All four targeted remote runs are terminal and disposable branches were removed by their runner. No deployment, publication, merge or enqueue was issued. Only the own fresh checkout/cache is scheduled for deletion after this handoff is created; all needed work is pushed first.

Relevant refs

• Reproduction and full findings: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-bteflake1-probe/handoff-artifacts/bteflake1/REPRODUCE.md and https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/sol-bteflake1-probe/handoff-artifacts/bteflake1/findings.md.
• Original readiness baseline https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37901759634, base3403a3fb1c00d97c0c4deeb9d6e4e6b7ed27e549.
• Controlled readiness10 https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37902042315, source5348866aa508e0b21ad08dab9a71dcd5bb664e0b on that base.
• Concurrent cache-family10 https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37901787109, same base.
• Cold Image3 https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37902904503, source084cee626e5005aceb8a5e04258376140befbc2c, base660dfb9d9.

No deployed/published-but-unmerged code: this session issued no service or artifact mutations and created no PR. Tests ran only locally and on disposable Actions branches. Dependency caches/build outputs/launcher jars were deliberately excluded from evidence. Selected original thread blocks and full assertion traces are saved; full XML/logs remain in Actions artifacts. The cold archive accidentally included512MB of dependency caches; only evidence was extracted and that archive was removed.

Next steps

1. Re-verify current main, remote diagnostic head and the linked evidence. Clone fresh in the new scratchpad workspace. Treat the deliberately failing test as a diagnostic, not a regression to ship.
2. Review the readiness premise finding. In a newly scoped fix, consider holding every writer that can publish the pending snapshot while preserving the meaningful missing-snapshot/readiness/result assertions. Do not change correct product reads to suppress a valid committed snapshot. The previous brief required stopping rather than rewriting a wrong assertion.
3. Investigate the cache timeout from the actual child preparation/installation work, capturing the nested process state and output at the failing point. Do not infer a lost poll from a200ms waiting-reader stack. The copied64-follower scenario is available but unverified; build it on a stable checkout before using its result.
4. Name the mechanism and deterministic condition before a fix. Establish >=50% red first attempts, mutation control, then the required green gates, independent review and one PR per defect. Do not merge or deploy. If a dependency owns the defect, stop with evidence for a separate upstream task.

Operational knowledge

Use ~/bin/coursier first on PATH; the other installed launcher is broken per the brief. Multi-test/repetition gates use the saved finite gha-gate runner on Actions with --local, not the coordinator. Place its scripts under the new scratchpad/out directory before invocation so its scratchpad root resolves correctly. Local commands acquire one of three prescribed build slots and have a finite deadline; local compilation can consume most of a five-minute short-run budget. Read first-attempt XML attempt-history, not aggregate success, since automatic retries conceal first failures. Saved cold runner now excludes cache directories from upload. Report-challenge CLI was not invoked because it auto-merges, contrary to the explicit no-merge scope.