Finish the BuildTestEmbedded concurrent reader timeout fix

Written 2026-10-19 03:30:20 UTC.

RE-VERIFY: This is a write-time snapshot. Recheck main and the draft PR with gh before resuming. The original flake remains unresolved.

Design decisions in force

User brief, 2026-10-19: "NEVER raise the test's 30 s limit or any timeout, reduce iteration or reader counts, weaken assertions, or disable tests." "Do NOT mark ready, merge, enqueue, publish or deploy. Do not re-run CI blindly." "Clone fresh. Use ~/bin/cs, never /usr/local/bin/coursier. Targeted runs only (scripts/test.bash --remote --test <name>; local runs are fine for constrained-CPU amplification). Count first-attempt results from the runner XML." Read ~/.claude/CLAUDE.md and follow ~/.claude/skills/fix-flakey-test/SKILL.md. Every commit message ends with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. The PR description ends with 🤖 Generated with [Claude Code](https://claude.com/claude-code). Every background wait must have a hard deadline and be killed before finishing; wait on tracked PIDs. Commit trailer and PR footer remain as stated in the original brief.

Mission summary

Fix concurrentRunningAttemptReadsDoNotTearIngestion without changing its workload or time limit. It predates https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1531. Original evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/r47-reader-classification-2026-10-18/handoff/r47/reader-classification.md . Current main has an unbounded join inside the runner's 30000ms whole-test timeout; the runner expires at that join.

What was found and done

Public getRunningTestAttempts and getTestAttemptResultsPaginated call AttemptHistoryReads.read. Failed source/metadata admission causes a per-canonical-run reconciliation flight. Callers capture the committed newline boundary they require, then wait on flight.done.get(). Its owner takes the per-run writer ReentrantLock plus SYSTEM file lock, captures a validated source prefix, writes and syncs derived index nodes and roots, publishes and compacts, and only then completes the future. This proves the future is the direct serialization point behind the captured fsync stacks. A read needs a root covering its captured cut, but this does not establish that fsync is the dominant timeout cause.

Unchanged b28d028005 original workload under ten real test copies, two CPUs and four workers: 10/10 first-attempt timeouts; automatic retries do not count as passes. Ordinary single local and remote cases passed. Setting only history-index sync to instantaneous in a diagnostic dependency did not remove the timeout: 4/4 measured-sync cases and 4/4 no-sync cases timed out first attempt. Sync-only theory is refuted as the complete cause. Captured stacks also show run-metadata JSON decoding, source-copy work and projection-lock waits. An 84-sample profile supports repeated reads/JSON work but is too small to quantify dominance.

A prototype completed read futures after validated root publication, before compaction. A held-compaction deterministic reproducer flipped from 10/10 failures to 11/11 passes, but the unchanged original still timed out 20/20 first attempts. More importantly, existing bounded page work tests observed two still-open worker handles after a read and later writes during an unchanged second page. That prototype was withdrawn. Do not adopt its lifecycle behavior. It also includes same-stamp invalidation checks needed only because it extended the active-flight lifetime.

The current narrow candidate only reuses immutable reads within a snapshot and already-proved checks. A 32-row public page on unchanged main made 452 attribution reads, including 440 duplicates across eight ranges. Ten baseline counter copies failed first attempt; eleven candidate copies passed. Cache maps belong to one snapshot and exact immutable generation, each bounded to 1024 entries and 1MiB weighted content; oversized entries are skipped, and close/detach clears maps. Values are keyed by offset AND length. No DTO or open handle is retained across requests. Cache-only still timed out 20/20 original first attempts; its 29-case remote selection passed25 and timed out4. It is not a complete fix.

Two further counters proved a duplicated root parse and32 duplicate row-length reads. Both failed first attempt and both retries. Modern nonempty/nonlegacy root generations now return after successful opened-file header validation; legacy/empty roots and failed opens retain the reread. Row decoding uses its already-checked length. All three counters plus oversized-field behavior passed first attempt on pre-rebase9754c980f0. Independent source reviews found no verified defect. Latest original and remote checks remain to be filled below.

Latest verified candidate is ae5207f3d6 at the draft PR. Its production tree remains the one that failed 10/10 constrained original first attempts at1b0bdd4880 (all 30 attempts fail; 32828–33408ms runner durations). Remote 9/9 and targeted 7/7 passed first attempts; the latter was repeated after mandatory fetch/rebase and again passed 7/7. Both independent source and public-test reviews accepted the narrow scope after a headerless fallback fixture was added. This fixture zeroes the generation magic but leaves modern immutable offsets/padding; it proves empty/legacy fallback classification and reread, not a byte-for-byte old writer round trip.

One final experiment found eager full run.json decoding before an unchanged attribution-root version check. Its public journal-only append counter failed first attempt and both retries with 779 bytes read. The proposed shortcut 33c13d79a3 is withdrawn: metadata file version omits contentDigest, and supported in-place writes can keep inode/size/mtime unchanged. The shortcut could skip malformed JSON that the old reconciliation ordering would report. This is a source-review finding, not a runtime-proved malformed-error reproducer. Post-fix 11-case validation was terminated during compilation (exit143), so no post-fix counts exist and no original comparison was run for that shortcut. The zero-read assertion is too strong without a content proof; do not merge it. Keep the useful finding that eager decode occurs, but preserve error behavior when pursuing it.

PR write-time state: OPEN, DRAFT, head ae5207f3d6, mergedAt null; 5 CI shards passed, 11 remain in progress; failures: none reported; no final verdict.. No CI reruns, ready action, merge, enqueue, publish or deployment occurred. The earlier referenced provisioning PR is now MERGED. Main advanced again during final evidence checkpoint to daf1d2ad51; the verified code and tests are based on 7485f604fc. Rebase and revalidate before any further code push; do not present this older head as tested against the latest main.

The unresolved items are the original timeout's full mechanism, a production fix that makes its reliable baseline flip, and statistical pass confidence for that original scenario. The direct future/lock/fsync connection is proved; its sufficiency as the timeout cause is refuted. Draft creation, source/test review, counts and evidence are complete. CI has no final verdict.

Reproduce

Clone fresh under workspace and read README. Required docs: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/playbooks/DELEGATION_BRIEFS.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md . Do not use the existing constrained-java wrapper; it contains CPU spinners. This effort used taskset and real test callers only.

Trial sources and XML/auditors are in https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/r56-reader-evidence-2026-10-19/handoff/r56 . Copy the r56OriginalTrial00..09 scripts from evidence into tests; they differ from the original only in the function name. Run targeted selectors on two CPUs, with JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2. Leave all256 attempts,12 readers and160 minimum read cycles unchanged. Wrap execution in timeout --signal=TERM --kill-after=15s 720s (with ordinary spaces between flags and values). Use four default runner workers. Audit first-attempt XML with audit-results.py; exclude filtered rows and inspect numbered attempt1 before any canonical final row. Do not rerun red CI to diagnose.

Relevant PRs and refs

Deployed or published but not merged

None. This effort performed no publish, deploy, route update or container mutation. The command record consists of git branches, read-only GitHub queries and targeted test runs. Test execution through buildtest did not deploy this code as a production service.

Next steps

1. Re-verify main, every branch head and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1538 before acting. It must stay draft and must not be merged/enqueued/published/deployed under this brief.
2. Start from a fresh clone. Read project instructions/README and the linked testing/delegation docs. Fetch the evidence branch and copy the exact trial sources from handoff/r56/trial-sources. Audit first attempts from XML, not final retry status.
3. Investigate the remaining original workload before changing code. Separate time spent in source-cut capture/copying under RunProjectionLock, run-metadata decoding, index preparation, root sync and actual page decoding. Capture phase counters/profile samples for the real public path while keeping every original assertion/count/timeout. The existing 84-sample profile and deadline dumps are insufficient to rank those costs. Do not repeat the withdrawn early-return or metadata-skipping shortcuts.
4. Once the exact condition is named, force it deterministically through a public API test and require failure before a fix. Preserve malformed-source error behavior and all source/generation/lifecycle contracts. A passing repeated-read counter alone does not prove the original timeout fixed.
5. Verify the unchanged original against the same two-CPU/four-worker workload. With a baseline rate of at least 50%, require at least 10 clean independent first attempts (P<=0.5^10<1%). If main changes, retain versioned controls; do not attribute a base change to the fix. Never reduce workload, raise timeouts or use artificial load.
6. Perform independent test/code review, fetch/rebase before builds/pushes, targeted verification only as the user requested, and update the existing draft only while it remains OPEN. Preserve the required coauthor trailer and Claude Code PR footer. Watch CI without blind reruns. Ask for no merge/deploy permission as a substitute for finishing the bug diagnosis.
7. Preserve all work remotely and remove fresh checkouts when finishing, as requested.

All local owned waits were terminated/reaped before cleanup; explicit checks found the last compiler/runner PIDs absent. The original clone and two worktrees have been removed after final remote verification. No local-only code is required to resume: the six remote branches above cover verified, withdrawn and evidence states.

Operational knowledge

Use ~/bin/cs. Builds compile slowly; warm artifacts before constrained comparisons. Keep versions explicit: main advanced from b28d028005 to 7485f604fc during this effort when the referenced provisioning PR merged. Do not attribute a difference between those bases to the candidate. Main and PR check reads must be at least 60 seconds apart. Every long task in this effort used a finite outer timeout; no owned wait may be left running after handoff. No timeout, assertion, reader count or iteration count was changed.

Operational note: gh pr edit --body-file failed with GraphQL: Projects (classic) is being deprecated ... (repository.pullRequest.projectCards). The body update succeeded via gh api --method PATCH repos/CodexCoder21Organization/BuildTestEmbedded/pulls/1538 --input <JSON file containing body>. The report-challenge CLI was not invoked because its automatic main merge conflicts with this brief's no-merge rule; this handoff preserves the workaround.