Handoff: Resume nested batch-rule diagnosis after provisioning transport recovery

Written: 2026-10-04 14:58 UTC

RE-VERIFY: This is a write-time snapshot. Check claims with handoff-cli claims before updates; use gh pr view for live PR state/head/checks and git ls-remote for every branch. Direct getBuildRun/listBuildRunsPaginated RPC wins over the stale WUI run list. Never merge, enqueue, deploy, publish, weaken tests or complete this handoff as a delegate.

Mission

The original investigation was asked to distinguish a single-flight upstream bug from a runner concurrency-test contract error after run 2be3ba61 passed 291/293. The two failures were testBuildOnlyBatchSharesWorkspaceAndBoundsRules and testBuildOnlyBatchWaitsForDependenciesAndIsolatesFailures, both with calls={} after their existing 60-second rendezvous. Prior session names rv142 and rv142b are historical lane codenames, not product versions. This continuation must recover complete evidence, verify the failing stage, then reproduce a named mechanism before fixing its owner.

Findings and actions

1. Prior investigators passed all 16 unchanged local invocations, including the old droplet harness, genuinely cold 128MiB children, two CPUs and four real concurrent callers. Those records remain on the two historical evidence branches below. The cache-flight global-serialization theory was unsupported. Cold helper download cost was measured but was never proved to cause the CI failure.
2. Complete historical console is now recovered through published read-only RPC: getTestResults and paged getTestAttemptResultsPaginated. All six failed attempts retain the same no-entry assertions. Five are interrupted during nested fixture compilation or a BuildscriptCache file-lock wait; one dependency attempt has already started a nested child and is interrupted while handling dependency rule-lock requests. The outer test functions ran. Outer test compilation/setup/JVM-start failure, and a universal before-child-launch failure, are refuted.
3. These interruptions do not prove the initial cause. Test finally calls runnerExecutor.shutdownNow(); runBuildOnlyBatch then cancels worker futures(true). This explains the cancellation traces after the assertion. No pre-assertion thread sample is retained. The initial no-entry mechanism remains unproven; no source fix is justified. The complete 4,582,928-byte coordinator log and all relevant attempt consoles are preserved.
4. Latest required failure on the original release PR points to run 3eab56e5, an interrupted chunked upload with one chunk and no assembled archive. No tests ran there. Do not use it to explain the original 2be3ba61 test failures.
5. The newer open release-pin proposal uses manager 0.0.203, EE 0.0.43 and CLI 0.0.119 and has a required 293/293 pass. It may supersede the old release dependency proposal, but a passing newer suite does not establish the historical cause. No sibling branch/PR was changed.
6. The fresh unchanged-source experiment 9bda1278 ended FAILED before tests: failureCategory=INFRASTRUCTURE/PROVISIONING, createdDropletIds=[], dropletId=0, 0 passed/0 failed, existing 600000ms deadline. Its complete coordinator error records digitalocean-droplets listDroplets losing its connection after request write began with EOF read 0 of 4 bytes; no conclusive async allocation outcome arrived. CLI 0.0.93 had failed upload; outer CLI 0.0.119 accepted this run at source fa75da5a01a09772c384254ed63ea02ce80e4adf. This is no test verdict. Client exited 1. Cleanup cancel request raced terminalization and returned cancelled=true, but authoritative follow-up remains FAILED with provisioning error. Watcher stopped and all local task processes are terminal.
7. Independent test-comprehensiveness and adversarial review passes are recorded separately in findings. Recovery is complete; initial historical cause remains unproven; fresh execution is blocked by provisioning transport. Final lane outcome BLOCKED_INFRA. No source fix/product PR justified. Raw jstack deliberately retains original whitespace; no product-code formatting claim is made.

Relevant PRs and refs

Evidence entry: https://github.com/CodexCoder21Organization/BuildTestRunner/blob/wip/L16-runner-ci-evidence/handoffs/artifacts/L16/findings.md . Full original handoff is retained on the older evidence branch. No deployment, artifact publication, merge or enqueue performed.

Next steps

1. Re-verify current state first: claims, PR/branch heads and terminal run 9bda1278. Supervisor routes provisioning transport recovery to its owning infrastructure work. No production action is authorized for this lane.
2. Once allocation transport works, submit one targeted unchanged-source experiment selecting the two named tests with timeout 5400. Use direct exact-run RPC if WUI projection is stale. Preserve selected canonical results and complete console. Do not rerun failed CI checks; unexecuted runs remain no test verdicts.
3. If no-entry recurs, capture the public nested runner path before assertion cleanup. Identify and force the actual compiler/file-lock/dependency ordering deterministically. Only then fix upstream with fail-first consumer/upstream tests. Preserve assertions, counts and deadlines.
4. If selectors pass unchanged, do not claim the historical mechanism is fixed. Supervisor decides whether continued historical investigation is warranted and whether the old release proposal should be replaced by the newer pins. Newer PR is not yet on main; do not close old PR as already landed.
5. Keep all evidence branches and both original concurrency tests. Complete six-attempt console archive and prior cold controls are useful salvage. Supervisor owns review, merge, enqueue, production actions and handoff completion. No product PR ready from this lane.

Operational knowledge

Use PATH=/code/ws/bin:$PATH. This host has no rg; grep is the fallback. Clone is /code/ws/L16/BuildTestRunner; no other checkout reused. No local full suite/fatjar under the shared 6GiB ceiling. Remote clients use timeout 5400; never rerun a failed CI check.

Published RPC probes and invocation notes are archived alongside evidence. WUI bounded test results returned output="" while legacy direct getTestResults retained complete console. WUI runs page reports outOfDate=true/refreshFailing=true and omits current accepted runs. A challenge draft is saved; report-challenge-cli was not invoked because it auto-enqueues/merges, forbidden by this brief. Canonical-results/projection sibling lanes own those API discrepancies; do not modify their branches.