Handoff: Verify the module-cache follow-up CI result without merging

Written: 2026-10-19 18:28:11 UTC.

RE-VERIFY: this is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1559 with gh pr view --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid before acting. Check the current CI run, not a superseded head.

Design decisions in force (owner, 2026-10-19)

The owner requested “follow-up fixes on BuildTest's module-cache lock change … failing tests first.” The following original requirements remain in force; none was superseded:

• “Disk: df -h / first; stop under 10 GB.”
• “Only stop processes you recorded. Only delete ~/.aibuildcaches entries containing your path (r97).”
• “Work in /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/r97.”
• “Prefer scripts/test.bash --remote --test <selector>. If you must build locally, cap memory and run one build at a time. Never run the whole suite locally.”
• “No new production test callbacks.”
• “For every new or changed test, report first-attempt counts from the runner's native XML on both the current base (must fail, where it is a reproducer) and the fixed branch.”
• “Never raise timeouts, reduce iterations, weaken or skip tests, or use reflection or mocks.”
• “Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.”
• “Do NOT merge, enqueue, publish or deploy.”
• The PR description must open with why, link the original PR by full URL and end 🤖 Generated with [Claude Code](https://claude.com/claude-code); read it back after edits.
• Watch with build-watchman, watch only, never --to-merged, with a hard deadline. This session's fixed watch ceiling was 18:27:30 UTC; it was not extended.

Read ~/.claude/CLAUDE.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md before any new work. The original takeover context is https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1555#issuecomment-6272884320. Repository-specific AGENTS/README must be read after a fresh workspace clone.

Mission and remaining work

All five requested implementation, test and issue items are complete. The remaining work is the final CI verdict for the open follow-up PR, and any evidence-driven repair if CI identifies a real defect. Do not merge or enqueue it. This session created a reviewable PR, read the description back, watched until the fixed deadline and cleaned its own checkout/cache/watchers.

At the fixed watch cutoff, the follow-up remained OPEN/BLOCKED on head8f3baf5d4ad1f8d68d8b90c7438b64c0c4aecc40. The current Actions run had4 successful shards and12 still running, all twelve in tests, with no failed conclusion. The separate kotlin-build-ci-test suite was queued with zero runs and kotlin.build (remote) had not reported. The aggregate bld-build job depends on all shard jobs and therefore had not started; this is explained by the active workflow rather than a missing Actions dispatch. Full CI green is NOT established. The timeout wrapper exited124 at its absolute deadline; watchman emitted RESULT: INTERRUPTED exit130 and both recorded watcher PIDs exited.

What was found and done

The cache constructor unconditionally removed .reading-, .writing-, and .discarded- paths before the service obtained its live-instance registration. A second service over the same directory could therefore delete a first service's active link or stage. Two public service reproducers failed at the precise file-survival assertions on every native attempt before the fix. The fix removes constructor sweeping and calls the existing cleanup only inside the service's existing registration.maySweep startup window, preserving the existing sweep completion/failure handling. No new production callback was introduced.

The original lock change is merged at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/451ba11b10 (original PR merged 2026-10-19T18:01:56Z). Only the three follow-up commits were replayed onto that main head with git rebase --onto origin/main 6139d13166b05843838daa0c64441f4588edcafb. Both pre- and post-rebase full source trees equal e0cf4f78c8d732f9cf54b4488f2b55923ea6f00a, so native results and independent reviews apply to exactly the pushed source. No duplicate test run was used to replace an earlier outcome.

The PR description below is the detailed implementation and native qualification record:

The owner requested failing-first follow-ups to https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1555 after the TinySite outage investigation found admitted BuildTest runs waiting behind a 762.5 MB archive hash under the cache index lock. A fresh review of that change identified startup deleting another live instance's files, missing generation and admission coverage, a process-wide digest fixture, and uncounted concurrent staging work.

The branch started at the original PR's head and was rebased onto main after it merged. This diff contains only the three follow-up commits. The rebase preserved the exact tested source tree (e0cf4f78c8d732f9cf54b4488f2b55923ea6f00a), so the native results below apply unchanged to the rebased head.

Startup now reclaims module-cache temporaries only within the existing LiveInstanceRegistration.maySweep window. A second registered service preserves a live reader link and publication stage. The production generation guard, archive validation, atomic publication, and existing byte/entry policies are unchanged.

Six public service tests cover live overlap, a stale invalid reader against a repaired generation, admission while copy or archive validation is held, and same-key publication racing a warm restore. The copy and validation tests use actual named-pipe I/O, the existing public Clock, and the unchanged 10-second provisioning guard. The original SHA-256 admission scenario and the new digest fixtures run in dedicated child JVMs, so their provider installation cannot affect sibling tests. No production test callback was added.

First-attempt counts come from the runner's native XML; retries are reported separately:

The copy and validation controls were applied in one build with the guard control. The validation fixture releases and finishes the held copy before holding validation, so its admission assertion tests the validation boundary independently. All temporary controls were removed. The final targeted register passed 9/9 first attempts with zero retries, including the unchanged restore/concurrent-eviction and restart/uncommitted-stage neighbors. Native records: base-targeted.xml, base-io.xml, base-isolated-admission.xml, mutations.xml, and fixed-targeted.xml.

The initial local wrapper compile failed because the new TestWorkerSlots annotation needed an explicit annotations artifact. That fixture error is recorded separately and is not counted as cache-bug reproduction. The first remote invocation failed during connection before submission and produced no XML; the listed counts are targeted local runs with a 1536 MiB build heap and 512 MiB metaspace cap. No whole local suite was run. No timeout was raised, iteration count reduced, assertion weakened, or test skipped.

Independent test and code review passes completed. One README sentence overstated publisher shutdown behavior; it was corrected and the correction was reviewed. Publisher shutdown ownership remains the separate work in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1551.

The measured staging and same-key sharing gap is tracked in https://github.com/CodexCoder21Organization/BuildTestEmbedded/issues/1558. Two same-key misses held two 179-byte stages (358 bytes total, two distinct inodes) while the committed-entry target was one byte. A direct run of the same compiled public fixture supplied that measurement; native qualification remains separate. A simple hard-link substitution was not included because source/cache paths have no same-filesystem guarantee and the current independent-copy ownership contract would change.

🤖 Generated with Claude Code

Review and scope evidence

A dedicated independent test-comprehensiveness pass found no gap. A separate code review found one README overclaim about publisher shutdown. That sentence was replaced with the precise condition: startup leaves temporaries in place when the ownership check finds another live instance, and the reviewer confirmed resolution. Base publisher shutdown can release registration while an interrupted shard finalizer leaves publication briefly alive; this is unchanged and separately addressed by https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1551. Do not claim this PR fixes that separate ownership issue.

Hard-link assessment: production archive download uses temporary output, verification and atomic rename, with cleanup unlinking; no in-place production writer was found. However source and cache paths have no same-FileStore guarantee (nested mounts or restricted filesystems are possible), and replacing the independent staged-copy ownership contract needs separate proof. The owner allowed issue-only handling when a link-only substitution was not clean. The measured staging issue is therefore filed, not silently dropped.

Relevant PRs and refs

The remote head was checked with git ls-remote and equals local HEAD. There was one worktree, no stash, no uncommitted/untracked source and no local-only commits. All implementation/reproducer source is on this branch. Earlier checkpoint commits were replayed by the completed rebase; their content survives in these three commits.

Deployed/published but unmerged: none. No deployment, artifact release, queue mutation or merge command was performed; the only external writes were the follow-up PR/branch, measured issue, and this handoff. Original PR merge was observed, not performed here.

Raw runner XML, logs, review reports and patch remain under the original r97/out directory. They were deliberately not committed as build/test artifacts, consistent with the owner's repository instructions; the counts, precise failure assertions, source, control methods and reviews needed for continuation are captured here and in the PR. No generated jars, cache or prepared classpaths are needed to continue CI observation.

Next steps

1. Re-verify current state first. Query the PR's current head and checks; verify whether the Actions run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/39299488923 is still its current run. If all checks are green, record that verdict and leave the PR open for review; no merge is authorized.
2. If more watch time is authorized, use a tracked, bounded cs launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/BuildTestEmbedded --pr 1559 --check 'kotlin.build (remote)'. Never use --to-merged. Stop its owned PID before the new deadline/final report. Do not hand-roll a polling loop or silently widen the deadline.
3. If CI reports a defect, read the full native output, reproduce it through the relevant public test locally or remotely before editing code, and fix the cause. Do not blindly requeue. Fetch/rebase before any build. Prefer one named scenario. A targeted example is scripts/test.bash --remote --test buildtest.embedded.crossRunModuleBuildCacheOverlapPreservesReader --log /tmp/r97-reader.xml.
4. The nine qualified selectors are crossRunModuleBuildCacheOverlapPreservesReader, crossRunModuleBuildCacheOverlapPreservesStage, crossRunModuleBuildCacheStaleInvalidReaderPreservesRepair, crossRunModuleBuildCacheCopyDoesNotBlockAdmission, crossRunModuleBuildCacheValidationDoesNotBlockAdmission, crossRunModuleBuildCacheSameKeyPublishAndRestore, crossRunModuleBuildCachePublicationDoesNotBlockAdmission, crossRunModuleBuildCacheRestoreSurvivesConcurrentEviction, and crossRunModuleBuildCacheRestartsFromUncommittedStage, all in buildtest.embedded. The runner supports repeated --test selectors.
5. Check the PR is still OPEN before any future push. If it has merged, create a new branch/PR for any new fix. Review tests and code independently and read any changed PR description back. Keep the required trailer/footer. Never merge, enqueue, publish or deploy under this brief.

Operational details

The initial remote targeted invocation failed before submission/run ID/XML: peer 12D3KooWLMyXNfwhcX1YsiNx3hnjk3GGSfsU1fydRa8bzrE6scMT at /ip4/198.199.106.165/tcp/35000/p2p/12D3KooWLMyXNfwhcX1YsiNx3hnjk3GGSfsU1fydRa8bzrE6scMT could not negotiate /aicli-rpc/1.1.0 within30000ms. This gave no native test count. Targeted local fallback used JAVA_TOOL_OPTIONS='-Xmx1536m -XX:MaxMetaspaceSize=512m -XX:+UseSerialGC', one build at a time. Child fixtures use128m and SerialGC. No whole local suite was run.

The runner's native XML has canonical test nodes and a separate root-level attempt-history list. Qualify first attempts from attempt number1; don't count eventual successes as first passes. The initial wrapper compile0/2 was a missing explicit build.kotlin.annotations:build-kotlin-annotations:0.0.10 dependency and was recorded as fixture error, never bug reproduction. All actual overlap/control failures have full native stacks, repeat consistently and target exact public assertions. All temporary controls were removed before the final fixed run.

Successful native test console output is omitted, so the staging measurement was independently repeated with the exact already-compiled fixture in a child JVM. Runner prepared-test-classpath paths are deleted at runner exit; the successful standalone launch mapped exact jar basenames to retained cache originals. Its exit0 and measurements were recorded separately from native qualification. Repeating that measurement is not necessary to observe CI.

The PR was initially stacked against the original open branch. GHA filters main/master targets, so it was retargeted to main and reopened once to emit its first eligible workflow event. After the original merged, a force-with-lease push of the exact-tree rebase started the current CI. No failed check was blindly rerun. The old stacked run is superseded. Old gh pr edit hits deprecated GraphQL projectCards; structured gh api PATCH with JSON body/base worked and exact readback was verified.

Central handoff health returned OK with incidental Netty StacklessClosedChannelException warnings; those warnings alone did not mean the probe failed. The current handoff skill creates through handoff-cli, not git; the format README's old merge instructions conflict with the current process and the owner prohibition and were not followed. report-challenge CLI would auto-merge, so infrastructure friction was recorded locally rather than invoking it.

Cleanup completed: removed only /home/helena/.aibuildcaches/tmp_claude-1000-code_2f854167-89e3-433d-bb44-9afc59221d1d_scratchpad_r97_workspace_BuildTestEmbedded, /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/r97/workspace/BuildTestEmbedded. All recorded test/build/watch PIDs are gone; no r97-cache Java process remains. Native XML, logs, reviews and findings remain in /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/r97/out. No source clone or owned build cache is required to continue; clone fresh under workspace and fetch the remote branch only if new code work is needed.