Handoff: Reproduce and fix BuildTest cold shared-cache preparation before opening a PR

Written 2026-10-21; final cleanup verified 2026-10-21T10:35:14.828050+00:00.

RE-VERIFY: this is an investigation snapshot, not a completed fix. Fetch current source, inspect the evidence branch, re-read the linked check/run, and verify current runner release identity before acting. Query the PR with gh pr view https://github.com/CodexCoder21Organization/UrlProtocol/pull/729 --json state,headRefOid,statusCheckRollup. Do not infer current deployment from the historical source references below.

Design decisions in force

Original brief, 2026-10-21: "BuildTest's cold shared module-cache preparation overruns its 20-minute deadline on UrlProtocol. Reproduce, fix, open a PR".

"No merge, enqueue, deploy or restart." "Do NOT raise the 20-minute preparation deadline or any other timeout. The fix makes preparation finish well inside it." "No reduced iterations, no weakened tests, no mocks, no reflection." "NEVER stop any codex process. Stop only JVMs you started, by recorded PID." "One full scripts/test.bash at a time machine-wide. Check uptime first." "If the mechanism turns out to be capacity, i.e. the work genuinely needs more than 20 minutes, STOP and write that up with numbers rather than raising anything."

Commits must end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. PR descriptions must end 🤖 Generated with [Claude Code](https://claude.com/claude-code). Work in a fresh clone under the lane's workspace directory; read root README and instructions. Preserve milestones remotely and in the progress file. The final report is out/r115-final.md, followed by removal of clones and lane-owned caches.

Mission and verified findings

The required check on https://github.com/CodexCoder21Organization/UrlProtocol/pull/729 failed: https://github.com/CodexCoder21Organization/UrlProtocol/runs/119122378116. BuildTest run https://buildtest.kotlin.build/run?id=48f1ffcd failed all ten shards with elapsed 1249375 ms against 1200000 ms, zero tests. At 10:30 UTC the PR remained OPEN at 15c01b85747be9541723dd1f6487f9b39923c5e6 and that check remained FAILURE.

1. The previous diagnosis was read from lane r112, together with its 69 MB raw log (public equivalent https://buildtest.kotlin.build/log/raw?id=48f1ffcd) and the older serial-chain memory. The older separate-runner chain does not describe this run: one runner builds fourteen roots.
2. The churn helper's child owns legacy simpleCache key 380cb1bf1c7bac2399df4d2f80d421d81cffa93377363c3e1820544a74061e7b. The explicit buildMaven child waits for it and then gets a HIT. The submitted source calls buildMaven inline from buildChurnOutputTestSupport: https://github.com/CodexCoder21Organization/UrlProtocol/blob/15c01b85747be9541723dd1f6487f9b39923c5e6/build.kts#L2284. Raw lines 17095–17103 show wait/HIT, and 22366–22442 show the owner's miss, Kotlin compiler diagnostics and successful store. Buffered child output prevents interpreting printed line order as acquisition order.
3. This is process-local cache coordination, not a stale persisted lock or another BuildTest run. The current owner calculates outside the coordination lock. A waiter cannot read an unfinished value. Source at the released executionenvironment 0.0.44: https://github.com/CodexCoder21Organization/kompile-executionenvironment/blob/1cb38081a0f9db671ee69c4f117410cc48595857/src/kompile/executionenvironment/BuildScriptCacheFlights.kt#L98 and JvmBuildScriptLauncher.kt#L1273. The 300-second diagnostic is a socket liveness poll during the wait, not an additional delay after 525 seconds.
4. Both explicit roots retain their full memory reservation. Raw CHILD_BUDGET lines report each 1463780352 bytes, two total 2927560704, child budget 3032354816: a third root cannot fit. BuildTestRunner owns that admission in BuildRuleAdmission.kt and Main.kt runBuildOnlyBatch. The owner compiles inline and does not ask this gate for another reservation. No cache-wait deadlock was identified.
5. The preparation deadline is intentionally one phase, including bootstrap, roots, independent verified copy, archive, serving and ready commit: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/f3889729412bd642738540324734a1f9f598e1b5/src/buildtest/embedded/BuildDriver.kt#L1056 and line 1120 onward. A successful last root therefore does not prove cache readiness.
6. A rule's duration starts before result validation, while build_rule_started is emitted after validation/lock acquisition. Adding the duration to that event timestamp does not establish completion time. The prior report's 09:03:20 final-completion estimate is not established by that arithmetic.
7. Production emits 73570 version warnings, exactly 14 x 5255. Manager's warning latch holds 256 identities, so a scan larger than the latch can evict its own previous scan. This is repeated work, but its contribution to the production deadline has not been measured. Another repeated-work issue is already fixed on main: https://github.com/CodexCoder21Organization/kompile-core/pull/397 removes unused full-workspace fingerprinting for empty parallel selections. It is absent from published manager 0.0.205. Its own description explicitly does not claim the full deadline solved. Do not duplicate that fix downstream or assume a dependency bump proves this brief.
8. The runner 0.0.124 published POM verifies manager 0.0.205, buildscript 0.0.43 and executionenvironment 0.0.44: https://kotlin.directory/buildtest/runner/buildtest-runner/0.0.124/buildtest-runner-0.0.124.pom. Historical live coordinator identity in r112 was embedded 0.0.69274771 from f3889729412bd642738540324734a1f9f598e1b5; this lane did not re-query or change deployment.

Two-sentence mechanism: the churn child calculates buildMaven inline while a separately admitted explicit buildMaven child waits for that same successful cache value, retaining its child memory reservation. Bootstrap, all fourteen roots, and copy/archive/readiness publication share one unchanged twenty-minute preparation deadline, but the exact code-level cause of the production calculation's duration remains unproven.

Diagnostic and its limits

A local standalone build used the exact submitted source (81 Kotlin files, 61199 lines, 3.1 MiB source, 26 MiB tree), published runner 0.0.124 and a fresh cache. It requested only buildMaven, no tests or other roots. Its outer cache key exactly matched production. Parent heap 768 MiB, child 979 MiB, ActiveProcessorCount=2. The invocation script, all event records, full error and three child stacks are on the evidence branch.

It failed once at the existing rule deadline: launcher elapsed 1307149 ms, deadline 1198812 ms; runner root duration 1363621 ms. Shell real=23m57.137s, user=6m32.352s, sys=2m57.655s. The parent initially parsed test declarations and wrote metadata cache JSON; the child remained RUNNABLE in Kotlin type analysis. This was a build diagnostic, not a test or the full preparation pipeline. There was no cache waiter, so a waiter is not a necessary condition for this local symptom.

Do not call this capacity proof: during the run host load rose to 158.11, CPU pressure some.avg60=74.67, memory pressure some.avg60=49.38, I/O pressure some.avg60=51.50. Child live-thread totals showed 250.06 seconds runtime versus 1577.44 seconds CPU queue wait. The timing was materially affected by unrelated work. No shared process was stopped. Both recorded JVMs exited through the launcher's unchanged deadline; no lane-owned JVM remains.

No deterministic main regression, product fix, mutation control, neighboring/full suite or PR exists. First-attempt results: one explicit diagnostic build failed, zero tests ran. The brief's capacity STOP condition has not been established. No code was deployed, published, merged, enqueued or restarted by this effort.

Remote work and references

Start with r115-evidence/r115-progress.md, r115-mechanism.md and cold-maven-record.md on that branch. The exception/timeout dump is cold-maven-failure.txt. The saved source fixture and build caches are deliberately removed at final cleanup. They can be reconstructed from the exact Git source and published runner coordinate. JFR binaries and the full repeated-warning transcript are not needed to resume; textual stacks and findings are preserved remotely. The child JFR was empty after deadline termination, so there is no child CPU profile to cite.

Next steps

1. Re-verify current source, check state, published/deployed toolchain and availability of a controlled execution slot. An asynchronous question asking for such a slot was sent to the owner; it has no answer as of this snapshot. Do not identify local host pressure as the production root cause.
2. Profile the complete exact cold fourteen-root preparation, including publication, with per-boundary work counts, child CPU and CPU-wait attribution, and stage copy/archive bytes. The raw production log has no timestamped compiler start/finish or individual publication command timing. Do not reuse duration-plus-start arithmetic.
3. State a specific code-level mechanism and force it deterministically through public in-process components using ManualClock. Demonstrate the test failing on current main. If no defect is identified, do not guess at the scheduler. If the necessary workload genuinely exceeds twenty minutes, stop with measured numbers under the original brief.
4. Fix at the owning layer, retain all timeouts, prove the same test passes and reverting the fix fails it. If upstream, follow the upstream/consumer reproduction and dependency workflow.
5. Run neighbors and one full suite from final head with the machine-wide exclusivity rule; preserve native first-attempt counts. Review test coverage and code independently.
6. Fetch/rebase, create the PR with the failing check as its opening motivation and requested footer, monitor required CI with watch-build, write the final report and clean only owned clones/caches. Never merge, enqueue, deploy or restart.

Operational notes

Read ~/.claude/CLAUDE.md, ~/.codex/AGENTS.md, the linked TESTING.md including extreme-condition testing, relevant READMEs and cache contracts, and the prior r112 diagnosis before acting. The evidence root was /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/r115. Local-only timings cannot replace a deterministic reproducer. The cached source/result relationship was preserved rather than changing wait policy to make a benchmark pass.

Final cleanup: eight lane-owned clones, the extracted exact-source fixture and the lane-owned cold cache were removed. Both recorded JVM PIDs are absent. Text evidence and the pre-cleanup final report are on the verified evidence branch; the final local report and this handoff include cleanup verification. Shared dependency caches were not removed.