Handoff: Restore run evidence and finish compiler diagnosis and tail-stall validation

Written 2026-10-04T22:11:48.078670+00:00. Original effort began 2026-09-29.

RE-VERIFY: this is a write-time snapshot. Check live claims before every update, each PR's state/head/required checks with gh, main and every branch with git ls-remote, and authoritative buildtest records plus complete attempt histories before acting. Green canonical totals can contain failed attempts. A stale projection saying not found or zero results is not a verdict. Do not resubmit an existing run because its client disconnected.

Mission and remaining work

The original user asked to “root-cause and fix the PIPELINE runner's tail stall — full suites on the PIPELINE coordinator leave 3–8 tests that never complete.” Restore authoritative evidence for existing exact-head run https://buildtest.kotlin.build/run?id=30149e8b, explain the three historical build-only initial failures, then finish required/full-suite validation and supervisor final review of https://github.com/CodexCoder21Organization/BuildTestRunner/pull/125. The optional telemetry decision is resolved and the approved correction is pushed and selected-tested. The immediate gate is BLOCKED_INFRA: the stored run projection is stale/refresh-failing and coordinator-backed log reads fail.

Sibling lane L41 owns shared source/rebase/PR updates and its bounded required-CI watcher. L42 owns the fixture evidence and historical build-only analysis; do not race a shared-source push. The supervisor owns every merge, enqueue, deploy, publication and handoff-completion decision. No such action, failed-check rerun, timeout increase, iteration reduction or additional remote submission was performed here. No other handoff was edited.

Findings and actions, in order

1. The dispatch-order mechanism remains supported, with original retained baseline 2/2 failures and fixed 20/20 passes plus 19 neighbor passes. Discovery-order tickets differed from compiled-method dispatch order, so a ready child held the slot needed by its undispatched predecessor. Source125 registers tickets synchronously at dispatch and retains captured file occurrence through callbacks, completion and release. Main is still4377987a08c924ebee823f5dee97614a9c4c423b; the source fix is not on main or obsolete. Rebased source is63a3e4558e5105150e47989e01edaa03f7d8e89e, including main's BuildRuleStartScope. Original incident/dumps and later occurrence/source-deletion/no-body validation remain in the refs below.

2. Historical run https://buildtest.kotlin.build/run?id=6fc1befd on8d536c0988c501fbadd30c57cb29e085c804cea5 has303canonical PASSED/0FAILED, but312complete attempts:309PASSED/3FAILED. Failures are testBuildOnlyBatchSharesWorkspaceAndBoundsRules attempts1and3 and testBuildOnlyBatchWaitsForDependenciesAndIsolatesFailures attempt1. Body-entry assertions see calls={} and discovery events after60seconds. The fixtures' finally shutdownNow then interrupts compilation/cache operations. This explains the interruption stacks, not the initial failure. Full consoles and all attempts are preserved remotely; do not describe later passes as a fix or host capacity as the cause.

3. All original seven broader failing selectors passed their first recorded attempts in6fc1befd; the saved claim they remain red is refuted. This does not prove every historical cause resolved. The separate cancellation source122 is still open and its PreparedLeasePipeline changes are not imported into source125/main. Older source120 adds in-body singleton warmup/phase diagnostics; current125 instead carries real fixture prerequisites with private relocated source/cache and an exact zero-compilation guard. Preserve the stronger tests, keep120 until they land and are compared, and keep122 with its owner. No PR was closed or branch deleted.

4. Supervisor approval on2026-10-04 20:56UTC permits only independent optional processTreeRunQueueWaitMs presence in two queue fixtures. Retained L291 19PASS/1FAIL differs solely at that key; README/sampler/formatter permit its absence when unmeasured. Both fixtures retain exact other-key/event-ordinal, lifecycle/attempt/lease/verdict/error/product-frame checks and all limits. Any supplied wait must be integer/non-negative on test_completed. Public formatter coverage verifies default/null/negative absence and zero/17/Long.MAX_VALUE exact presence, with complete error and all other fields. No product formatter changed.

5. L42's exact fixture patch was folded by L41 into63a3e455; its tested tree matches source tree4131bddbcd7c288aba0b9b8b4915bcf6ec91383a. Four fixture/formatter selectors passed4/4 with0failures, then two unchanged build-only selectors passed2/2 with0failures. These are canonical console observations: the old pinned remote client prints no successful run ID and creates no requested XML. They are not complete attempt-clean proof or a reliable historical-failure reproducer. Exactly two L42 manual submissions are consumed; no more were submitted on resume. Separate focused test-comprehensiveness/adversarial passes are retained and refreshed; L41's full-source passes remain separate.

6. Required remote check111534200723 remains IN_PROGRESS for30149e8b, claiming building/testing. GET /api/runs?limit=500 reports outOfDate=true,refreshFailing=true,upToDateAsOfEpochMs1791101091862 and omits it; /run?id=30149e8b says Build run not found; /api/test-progress says the recorded count is unavailable; /api/test-results returns0rows/complete=true. Coordinator-backed /log/raw timed out45seconds with0bytes, after prior RELAY_FORWARD_FAILED. Bounded build-watchman --run30149e8b --once exits0 PENDING with a not-found PROBLEM; that is not GREEN. Fabric certificate directories are absent and no fabric connector is exposed. Direct published CLI recovery already failed for the absent coordinate. No run was deleted, repeated or cancelled.

7. The alternate source check is informational FAILURE because its durable deferred queue is full at64checks: the build never started. This is not a source/test execution failure, and no requeue was performed. Toolchain pin145 remains position1/AWAITING_CHECKS at queue head14f87c63e089201cf1fac776d979f9a4d1f3d894; required head check SUCCESS. Leave the queued branch untouched. Do not replace the required current-source gate with historical checks or the pin PR's313canonical passes, which also note failed re-leases.

8. The earlier missing-worker-annotation hypothesis is not an established remedy. Current runner README Work-Queue Lease Mode, PreparationAdmission.kt and owner UTILIZATION_ARCHITECTURE.md CPU admission say declarations reserve no lane/CPU capacity. Build-only width and memory gate do not read workerSlots; the two fixtures deliberately admit all four rules through their memory provider. Capture actual compiler-stage/lock-owner/CPU/GC evidence before fixture teardown and force the identified condition deterministically before any fix. Shared memory readings remain above the brief's4.5GiB heavy-launch threshold, so no heavy local compilation was started; that launch constraint is not a historical root-cause claim.

Relevant PRs and refs

Snapshot checks are from resumed read-only queries around21:58-22:04UTC; re-query before acting. Required check is kotlin.build (remote).

All branch refs were verified with git ls-remote on resume. Keep every baseline/evidence branch until its insight and regression tests survive landing. Evidence branches have no product PR and must not be merged into product main.

Two recovered local commits, bded8b0668f957608b6d3742343042da62e26e26 and3f769df, are aliases of already-preserved content, not lost work: their complete source and exact fixture patches are on the L42 approved-telemetry evidence branch and their tested source tree matches the shared source125. No second source-owner branch was pushed. All checkouts are clean with no stashes; the only local-only commits are those recovered aliases.

No artifact was published and no service was deployed, restarted or reconfigured by this lane; its recorded commands are read-only plus Git evidence checkpoints and this handoff/report. This makes no claim about the current production rollout. Future publication/deployment is an operator decision after source landing, outside this brief.

Next steps

1. Re-verify current state first: live claims, source125/main, L41 ownership/watcher, queued145 and all refs. Restore an authoritative read path for existing30149e8b and the historical worker evidence; retain existing runs. Do not duplicate unidentified submissions, rerequest a failed check, or touch someone else's queue.
2. Retrieve actual full-suite verdict and complete attempt histories for current source63a3e455. Empty cached results and historical green totals do not discharge this gate. If a required check becomes red before execution finishes, compare its run record before classifying it. L41 owns source updates; communicate through lane files.
3. Explain the three historical initial build-only failures using the narrow real public path and pre-teardown compiler/lock/CPU/GC evidence. Current selected passes and interruption stacks do not settle it. Produce an initially reliable/deterministic failure before any source fix. A confirmed upstream defect needs consumer and upstream reproductions and an upstream PR; publishing is not authorized here.
4. Preserve every timeout, iteration and assertion except the already-approved optional presence correction. No guessed worker annotation, retry, hardware change, or source mitigation. If the test contract needs another change, present the evidence and exact decision to the supervisor first.
5. Once all review/execution gaps are closed, obtain required green exact-head validation, then leave supervisor final review/merge and handoff completion. Keep122's cancellation work separate; keep120's useful diagnostics until125's stronger prerequisite lands; keep145 with its queue owner. Do not close satellites merely because a feature branch supersedes them.

Operational knowledge

• Export PATH=/code/ws/bin:$PATH in every shell. Current runner scripts use their own digest-verified JVM Coursier. Clone fresh into the assigned lane directory and read README/AGENTS; no AGENTS exists in Runner. No full local suite/large local build on this shared6GiB host. Exactly two L42 manual remote submissions already occurred.
• Full selector is buildtest.runner. The old0.0.93 remote client does not expose successful run IDs/XML; preserve console counts separately from complete attempts. Historical /api/test-attempts returned0:0 while server-rendered test-activity-data contained312:312; compare both before calling evidence empty.
• All long commands are tracked and bounded. PR source owner uses plain build-watchman, never --to-merged. Poll GitHub no faster than60seconds; kill only owned recorded PIDs. All L42-owned processes are stopped at terminal report; do not stop L41's watcher.
• New evidence is https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/L42-approved-telemetry-evidence/investigations/L42-approved-telemetry/resume3; exact patches,6selected passing consoles and3historical complete failure consoles are in its parent. Complete historical312attempt payload and L291XML are on wip/L42-tail-validation-evidence. Original baseline/fixed journals are on the p35 evidence branch.
• Four required whole documents remain byte-identical to prior full reading, verified by current GitHub blob SHA: handoffs README a196623f0d6874d00ac04a6babb443ea0d87feda, PHILOSOPHY82e280be0d314cb10209aaf1c7b27d02a2989b52, TESTING1d2f1eece9eef27bd246e6761d5a20114182d8a8, CODE_REVIEWa416f1d57196db9ecc5d07cd1c707798885b3d48. Relevant sections: Triaging/Format; root cause/reproduce/refutation; hermetic public API/no calibration; test weakening/Claims Host Overloaded. Runner/core/buildscript READMEs and owner utilization contract were read. Report-challenge CLI auto-enqueues/merges, prohibited by this lane; its full service/tool friction is preserved for supervisor reporting.