Handoff: Decide occurrence landing scope after the final diagnostic upload failed

Written 2026-10-05T06:15:44.438267+00:00.

RE-VERIFY (2026-10-05T06:15:44.438267+00:00): This is a write-time snapshot. Recheck handoff claims, GitHub PR heads/required checks, git ls-remote, and buildtest public HTTP runs/raw/all result pages. Canonical passes do not erase failed attempts. The supervisor owns every merge, enqueue, deploy, publication and handoff-completion decision.

Mission

The user asked to “root-cause and fix the PIPELINE runner's tail stall — full suites on the PIPELINE coordinator leave 3–8 tests that never complete,” then required “completions must identify the terminating occurrence.” Original review: https://github.com/CodexCoder21Organization/BuildTestRunner/pull/125#issuecomment-5887637496 . Finish normal-dependency/full-suite validation and preserve the supervisor's final review/merge gate.

What was found and done

1. Discovery-order tickets could block a READY child behind an undispatched predecessor. A file-less same-name completion could release an active sibling ticket. Source now reserves tickets synchronously at dispatch, selects terminating canonical source occurrences, retains captured metadata after deletion, and leaves ambiguous process samples unavailable. Original source/reproducer evidence is retained. Source-path effects are merged in https://github.com/CodexCoder21Organization/kompile-core/pull/343; typed failure transport is merged in https://github.com/CodexCoder21Organization/kompile-core/pull/340. The historical unpublished-dependency premise is refuted; no publication is needed for this source.

2. Real-child regressions cover parked/pending pre-READY exits, reversed callbacks, out-of-order tail/refill, deleted-source/control, and compile-failed sibling. Preserved fail-first baseline,10 fixed original passes and19 neighbor passes are historical normal dependency evidence. Strong no-body fixtures assert zero measured compilation rather than warming the timed body. The supervisor-approved optional processTreeRunQueueWaitMs presence correction and sibling fixture patch are folded; all required fields/order/identity/error/product comparisons remain exact. Public formatter absent/present cases and real-sampling neighbors remain. Identical-tree fixture selectors4 PASS / 0 FAIL and build-only neighbors2 PASS / 0 FAIL are retained.

3. Source remains 1d2b4cfb9838d32417379d381806bcf5ffa1b356 on main 9b21080331af10178195f19cce060d6c741ab04b, with normal manager 0.0.205 / executionenvironment 0.0.44, independent of https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145. Exact-head https://buildtest.kotlin.build/run?id=a8ede5c5 COMPLETED324 canonical PASS / 0 FAIL, but330 passing attempts / 1 failed attempt. FailedA212's unchanged 60-second batch-entry assertion had no entered bodies; later attempts passed. Older https://buildtest.kotlin.build/run?id=30149e8b and https://buildtest.kotlin.build/run?id=65309f6e warm/batch failures remain unexplained. Canonical success is not attempt-clean proof. Required remote CI is SUCCESS; latest plain PR watcherGREEN/exit0. Informational kompile check is FAILURE; it is not a required gate. No rerun was requested.

4. Two prior diagnostic runs retained original real public batch scenarios and limits: https://buildtest.kotlin.build/run?id=43cae9ef 2 PASS / 0 FAIL; https://buildtest.kotlin.build/run?id=6d2227f4 2 PASS / 2 FAIL/321FILTERED_OUT, first attempts only. The two failures reported first and third bodies entered. Canonical stdout was empty over permitted HTTP views; documented result/feed output restrictions prevent claiming a deployed output bug. The diagnostic snapshot was therefore placed in the failure message on investigation-only d84d80e.

5. Run 6 authorized exactly ONE more submission of d84d80e. Confirmation/refutation criteria were pushed before launch. https://buildtest.kotlin.build/run?id=8d1d87a8 ended0 PASS / 4 FAIL / 321 FILTERED_OUT, four first attempts. Raw coordinator, tracked clientexit1, watcherRED/exit1 and fresh runs row FAILED agree; runs refreshFailing=false when read. Cached results initially showedPENDING, then4FAILED with complete=false/reconciliationPending=true and stale endpoint flags. No duplicate full suite, failed-check rerun or second submission occurred.

6. Before cleanup, revised diagnostic's unchanged60-second assertion had calls={} and only discovery events. Workers1/2/4 waited on ReentrantLock$NonfairSync@6a41eaa2 owned by worker 3; owner CPU 8,414,860,590 ns. Visible waiters are in BuildscriptCache.compileWorkspaceOwned:1272 -> compileBuildscript -> rule-index fingerprint validation -> runner.runOneExecution. No child-process row or entered body was recorded. This establishes a preparation wait at this boundary and refutes held-body execution coupling for this snapshot; it does not explain the owner's operation or the earlier two-entered failure.

7. Correction: prior planning incorrectly assumed65,536 failure characters. Protocol.describeTestFailure retains16,384 leading characters. Returned diagnostic16,435 characters ends during worker 2 stack with28,384 characters omitted, before active owner 3's stack. Pre-registration therefore classifies causal diagnosis asINCONCLUSIVE. Investigation revision 68916b9 puts waited-on owners first from one public JMX dump and retains indexed first 12/buildscript/runner frames. The exact normalized public scenario still matches the original, with 8 assertions / all bounds unchanged. A small Java probe through public describeTestFailure deterministically loses a labelled owner after preceding waiter text (expectedFAIL1/1) and retains it first (PASS1/1) under the unchanged cap. This is formatting proof, not a product fix or execution of the revised scenario. No source fix is justified yet.

8. Separate sixth-invocation test-comprehensiveness and adversarial reviews are written and remote. The draft source PR's reason-first description includes0 / 4, the snapshot limit and all remaining failures. All 28 satellite branch heads were re-read; only this lane's evidence/diagnostic branches changed. No PR/branch was closed, discarded or merged. Strong occurrence/no-body tests, baselines and evidence remain worth keeping.

9. Run-7 used its sole authorized remote submission of68916b999a34d479e8f6bd2f15c22fc3c72da6b0 after confirmation/refutation criteria were pushed. https://buildtest.kotlin.build/run?id=c34ed33c ended0 PASS / 4 FAIL / 321 FILTERED_OUT, four first attempts. Tracked client/watcher exit1, raw and fresh runs FAILED agree; runs refreshFailing=false/snapshotAgeMs117. All325 result rows were returned untruncated; the result endpoint itself still reports complete=false/reconciliationPending=true and a stale refresh flag. Its failure rows agree with fresh runs/raw. The normalized diagnostic remains byte-identical to the original scenario with8assertions/all bounds unchanged.

10. Registered result: INCONCLUSIVE. The diagnostic returned677characters containing only IllegalStateException: Check failed from its unchanged30-second executor-shutdown check in the outer finally. The original shared-workspace test returned the same cleanup check. Two other neighbors reported zero entered bodies; the width scenario included late build_rule_started events. No owner stack, snapshot marker, earlier cause or suppressed exception was returned. The snapshot is attached only to an entry AssertionError; a later finally exception replaces any pending exception. This establishes a measurement-loss path, but does not prove which earlier assertion failed or explain the delayed preparation mechanism. Run6's compilation-lock chain cannot be assigned to run7 without its owner evidence.

11. Source PR remains draft at1d2b4cf on current main9b21080. Its reason-first description now records the result and remaining gap. Separate test-comprehensiveness/adversarial reviews and follow-up are preserved. All28 satellite heads were rechecked; strong occurrence/no-body fixtures, fail-first baselines and owner observations remain worth keeping. No source patch, limit change, assertion weakening, publication, merge, enqueue, deploy or handoff completion occurred.

12. Run8 authorized a local retention proof followed by exactly one final remote submission. The source-extracted local fixture captures a real JMX snapshot, cancels the real accepting Future, and formats the resulting error through public describeTestFailure. The unchanged cleanup replaced the primary assertion with CancellationException (1expectedFAIL). Diagnostic57ed5b4 retains the primary, attempts each original cleanup action and suppresses later failures. Primary-plus-cleanup and cleanup-only checks pass2/2. Complete batch body is byte-identical to68916b9; all8assertions and60/120/30-second bounds/checks remain. This is diagnostic retention proof, not a batch fix.

13. The sole final remote client exited1 during uploadChunk: Persistent RPC connection to service 'buildtest' was closed while requests were still pending. Health had returnedOK. No run ID or test verdict was returned. Fresh public runs reads show refreshFailing=false but runMembershipReconciled=false and no identifiable BuildTestRunner-eighth row in the returned page; this does not prove server-side nonexistence. No further submission was made. The registered result isINCONCLUSIVE: there is no new owner snapshot, and the preparation cause remains undetermined.

14. Draft PR description now records all local proof and upload evidence. Source1d2b4cf remains unchanged on current main9b21080 after no-op rebase. Required remoteSUCCESS, informational kompileFAILURE; plain watcherGREEN/exit0. Separate review passes are preserved, and all28satellite refs were reverified. No product fix, merge, enqueue, publication, production-host access, deployment or handoff completion occurred.

Relevant PRs / refs

Current PR states (recheck through gh pr view):

Salvage: keep occurrence/no-body tests, optional-telemetry fixtures, fail-first baselines and the new diagnostic retention change. No PR or branch closure/deletion is recommended now. Separate identity, cancellation and pin PRs remain with their owners. Reassess warmup PR closure by content only after its stronger replacement lands. No satellite was closed.

Next steps

1. Re-verify current state first. Supervisor question: keep the occurrence PR draft until the retained preparation failures are explained, or make a separate scope decision about landing the occurrence fix using its focused evidence? Recommendation: keep draft. The final experiment allowance is exhausted; no further remote submission is delegated.
2. Read investigations/L41-occurrence/eighth on the linked evidence branch: invariant table, probe generator and before/after/cleanup-only traces, pre-registration, observation.md, client.txt, reviews.md, challenge.md, runs records and PR description. Diagnostic57ed5b4 is remote and recoverable.
3. The missing causal evidence is the active lock owner's preparation stack and held-resource/progress chain before cleanup. A compiler frame alone is not a cost explanation. A later separately scoped investigation must force the established condition deterministically through the public API before fixing its owning layer. No product patch is justified from the present evidence.
4. Keep earlier warm/two-entered failures explicit. The historical exact-source full run remains324canonicalPASS/0FAIL but330passing/1failed attempts. Any eventual landing needs the supervisor's disposition of those failures, exact-head validation and final review. No full-suite rerun was made in run8.

Operational knowledge

Every shell exports PATH=/code/ws/bin:$PATH. Fresh clone /code/ws/L41/BuildTestRunner-eighth and its evidence worktree are lane-owned. The repo launcher fetches digest-verified JVM coursier; do not use the wrong-architecture /usr/local/bin/coursier. Local retention compilation used128MiB heap and waited until cgroup memory fell below4.5GiB; no large local build/full suite. The source-extracted proof replaces only the expensive scenario body with a real JMX snapshot and Future cancellation; it is not a production batch regression test.

The diagnostic now keeps the primary failure before cleanup and attaches subsequent cleanup errors as suppressed. It attempts every original cleanup action. Failure detail still has the original16KiB bound; owner-first ordering protects the decisive prefix, but a very long primary can still truncate later suppressed text. Retained local checks prove the primary and cleanup errors render together for the forced cleanup case. No formatter or product behavior changed.

Remote client PID909472 and plain PR watcher are terminal. No accepted run id was available, so no run watcher could be attached. Every launched helper has a tracked session and hard bound; cleanup uses exact recorded PIDs only. Public API membership is not reconciled, so no accepted record identifiable is narrower than no run exists. No production-host access or direct RPC history reads are allowed.

The first handoff update returned an ambiguous outcome after request write. Re-fetch verified it had not landed; a subsequent explicit update succeeded. Do not blindly replay an ambiguous mutation. The report-challenge skill automatically enqueues/merges, conflicting with the lane's absolute prohibition; challenge.md preserves the exact errors and suggested investigation for the supervisor instead.

Required reading: PlanRepository handoffs README(Triaging/Format/Partial work); DocumentationRepository PHILOSOPHY(root cause/refutation), architecture/TESTING(public API/end-to-end/determinism), CODE_REVIEW(host claims/locks/no weaker tests); complete runner README and original review. No root AGENTS.md. The lane brief overrides general document suggestions to merge, publish, deploy, or inspect production hosts.

No code was deployed or published by this lane, verified against the action record. No build jars/classes are committed; probe sources/logs are intentional evidence on the evidence-only branch. Product source PR contains no diagnostic/evidence addition. Findings file: /tmp/claude-1000/-code/4127b3f9-6050-446f-a28d-b0511dacd396/scratchpad/hq/out/L41-findings.md.