← Handoffs

id: hf-2026-09-24-fix-the-buildtestrunner-warm-lane-tests-that-fail-their-first-attempts-on-the-120-s-limit-and-pass-on-retry url: url://handoff/handoffs/hf-2026-09-24-fix-the-buildtestrunner-warm-lane-tests-that-fail-their-first-attempts-on-the-120-s-limit-and-pass-on-retry title: Resolve deleted CI and finish verification for https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135, then continue warm-lane work summary: Resolve the required check pointing to deleted failed run ca8d84a6, prove actual suite execution, and rerun the changed test when the shared lock is available before final review. Both static reviews found no pin defect at b9d0aa9; local gate executed zero tests because its 1500-second lock wait expired. All broader mission work is retained. created: 2026-09-24T18:17:58.795Z completed: null blocked-reason: Required check for BuildTestRunner pin PR points to deleted failed run ca8d84a6; actual suite execution is unavailable. dependencies:

  • url://handoff/handoffs/hf-2026-09-24-resume-the-2026-09-24-handoff-sweep-once-buildtest-allocates-droplets-again-rerequest-the-seven-parked-prs-re-enqueue-aiclisupervisormanagerembedded-pr-25-land-buildtestrunner-119-and-116-then-drive-the-blocked-and-held-items

rev-btr135b final review result — 2026-10-02 14:48 UTC

RE-VERIFY 2026-10-02 14:48 UTC: this is a write-time snapshot. Re-read https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 checks and remote refs before acting. Every prior Mission, finding, and branch/PR row is retained below. Broader cancellation and warm-lane work is unchanged and remains owned by the supervisor.

Remaining first: resolve the required check pointing to deleted failed run ca8d84a6, obtain a valid run with complete executed-versus-filtered records, and finish the changed-test local gate once the compile lock is available. Then the supervisor can finish final review and decide whether to merge. This review is REVIEW_BLOCKING on execution evidence; no pin source defect was demonstrated and no fix was made.

OBSERVED: The PR remains OPEN at b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c (live read 14:38 UTC); required check 110853208564 is IN_PROGRESS with URL https://buildtest.kotlin.build/run?id=ca8d84a6. Hardware Fabric read of the coordinator's run.json states CANCELED, archived=true, deletedPriorStatus=FAILED, deleted at 2026-10-02T13:51:18.742Z via deleteBuildRun with artifacts removed. Its counters testsTotal=0/testsPassed=0/testsFailed=0 and empty testSpecs are post-delete values, not proof of a selector mismatch. The original failure, deletion caller and complete execution rows are unavailable. This lane issued no deletion or rerequest.

OBSERVED: Main's last green commit 960881e4ec86b8331316b27f6f5628264b2ca38a reports 293/293 in GitHub for https://buildtest.kotlin.build/run?id=e3d1bbe4, with 3 retry warnings. The current checkout has 293 top-level test functions in 292 files; 5 nested build scripts and 4 barrier fixture scripts bring total .kts files to 301. Dashboard rows and retained baseline run.json are missing, so complete executed-versus-filtered/attempt comparison cannot be made.

OBSERVED: Both separate static reviews found no proven defect in the three pin lines. Both wrappers and the nested bootstrap test use 0.0.115; the version's POM declares buildscript 0.0.44 and its JAR endpoint returns HTTP 200. No active old CLI pin remains. Existing public CLI/barrier assertions, limits, lifecycle cleanup and README relationship are preserved. PR description opens with why and includes full links and the required footer. Source diff has no build artifacts. This PR adds no fail-first test; the historical missing-class fixture is on the separate cancellation PR, and upstream fail-first evidence is carried forward without claiming this review reproduced it.

OBSERVED: The requested local changed-test command waited on flock -w 1500 <scratchpad>/local-build.lock scripts/test.bash --local --test testE2EExplicitShardPreparationDoesNotCopyLiveCache, under UTF-8. It exited 1 after failing to acquire the lock; the test script and its JVM never launched. Full local counts for this review: 0 passed / 0 failed / 0 executed, 1 lock-acquisition failure. Shell syntax checks passed. The prior implementing lane's results below are separate. No three-repeat result is claimed; this lane's specific instruction asked for the changed test once. No second local or remote run was submitted, no lock was removed/bypassed, and no limit was changed.

Repo Branch Remote head SHA PR What is on it State
BuildTestRunner review evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/rev-btr135b-2026-10-02 08eeaf6fc85da7cdae7e77ea9a0ef58990337322 none lane-evidence/rev-btr135b: full review findings, deleted-run record, current/baseline GitHub snapshots, publication headers and stopped watcher output Review blocked on execution evidence; no source fix; do not merge evidence
BuildTestRunner pin source, carried forward current state https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/K15-kompile-cli-0.0.115 b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 Unchanged three-line pin bump OPEN; required check still references deleted failed run

No merge, enqueue, new CI/remote submission, deployment, production restart, route change, artifact publication or handoff completion occurred. The local watcher was stopped and its shell/JVM exited; no remote run was canceled by this lane. The report-challenge CLI is not invoked because it automatically enqueues/merges, forbidden for this delegate; stale/deleted-run evidence is preserved for the supervisor.

OBSERVED: The first final update failed during service API bytecode fetch before its write. Full output is on the evidence branch in handoff-update-failure.txt. This update uses a fresh handoff read and retains all latest rows below.

rev-btr135b live CI discrepancy — 2026-10-02 14:24 UTC

RE-VERIFY: write-time snapshot. Re-read PR checks and the coordinator run record before acting. Every prior Mission, finding and branch row is carried forward below.

Remaining first: the supervisor must resolve the required check that still points to a deleted failed run and establish actual suite execution; the lane's local changed-test run is still waiting on the shared compile lock. https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 remains OPEN at b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c. Static test and code review found no proven pin defect. A clean final verdict is not justified yet.

OBSERVED: Hardware Fabric read of /root/buildtest-data/runs/ca8d84a6/run.json states CANCELED, archived=true, deletedPriorStatus=FAILED, deleted at 2026-10-02T13:51:18.742Z through deleteBuildRun with artifacts removed. GitHub check 110853208564 is still in_progress with URL https://buildtest.kotlin.build/run?id=ca8d84a6 and all progress boxes unchecked. The post-delete record has testsTotal=0, testsPassed=0, testsFailed=0, testSpecs=[]; these values cannot prove a selector mismatch or reconstruct the original failure. Main's last green commit 960881e4ec86b8331316b27f6f5628264b2ca38a reports 293/293 in GitHub for https://buildtest.kotlin.build/run?id=e3d1bbe4, but its dashboard rows and retained run.json are unavailable. This checkout has 293 top-level test functions in 292 files, plus four nested fixture test scripts and five build scripts. No executed-versus-filtered comparison can be made from missing records.

Repo Branch Remote head PR State
BuildTestRunner review evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/rev-btr135b-2026-10-02 b0453a1babb2deb75c3832f5e325ee8b51dab80a none Separate review findings, deleted-run record and GitHub head/baseline check snapshots; no source fix; do not merge evidence

No deletion, rerequest, new remote run, source push to the PR, merge, enqueue, deploy, restart, route change, artifact publication, or handoff completion by this lane. Local results remain zero completed; the 1500-second flock acquisition bound is unchanged. The challenge CLI is not run because it automatically merges, forbidden for this lane. The CI mismatch evidence is retained for the supervisor.

rev-btr135b review checkpoint — 2026-10-02 14:19 UTC

RE-VERIFY: this is a write-time snapshot. Re-read the PR and remote branch refs before acting; every existing Mission, finding, and branch row is retained below.

Remaining for this lane: finish the changed test locally under the shared lock, verify actual remote execution counts, then give the review verdict. https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 is OPEN at b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c. The required check is still IN_PROGRESS at https://buildtest.kotlin.build/run?id=ca8d84a6; the brief's green state is not current. Both review passes found no proven code defect in the three pin lines. The dashboard cannot currently establish test counts: the ca8d84a6 detail is missing and test rows are empty, and main's last known-green e3d1bbe4 also has empty rows. A Hardware Fabric read-only lookup confirms /root/buildtest-data/runs/ca8d84a6 exists; its per-test event file was absent on the first read. No remote run, deletion, rerequest, production change, merge, enqueue, or artifact publication was made.

Repo Branch Remote head PR State
BuildTestRunner review evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/rev-btr135b-2026-10-02 117b8cc570f33a723b9113e14bb23a94a6c308f5 none lane-evidence/rev-btr135b has the separate review findings and live PR snapshot; no source fix; do not merge evidence

Local tests in this review: zero completed, one queued on the shared compile lock. The pin POM is published and declares buildscript 0.0.44; no active 0.0.93 CLI pin remains. Prior lane results are retained below and are not counted as this review's runs.

btr135 time-box result — 2026-10-02 13:22 UTC

RE-VERIFY: all states here are write-time snapshots. Re-read PR checks and git ls-remote before acting. This section supersedes the earlier btr135 checkpoint; every original Mission, prior finding, and branch/PR row is carried forward below. The cancellation/launcher/adopted-outcome/CAS/listener work is outside this lane and remains for the supervisor.

Remaining first: finish watching CI for https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135, then perform the supervisor's final review and merge decision. The delegate stopped new work at 13:19:40 UTC (75 minutes after start). The PR is pushed and locally verified, but CI green is not claimed. No merge, enqueue, deployment, artifact publication, or handoff completion occurred.

Repo Branch Remote head PR State
BuildTestRunner pin source https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/K15-kompile-cli-0.0.115 b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 OPEN; rebased onto main 962511127006ff6d9842042c88e59d29299376ac and force-with-lease pushed after local verification; CI queued
BuildTestRunner evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/btr135-2026-10-02 85ce64a0bcf95448646eb3efb61c21a84bff7327 none Final source/evidence checkpoint; lane-evidence/btr135 contains findings, full local outputs, thread dumps, PR text, and watch output; do not merge evidence

OBSERVED: The original required check passed 279/279 tests at https://buildtest.kotlin.build/run?id=0c06dde7. The original failed informational check also reported 279 passed/0 failed tests, with five unsuccessful rules listed in the earlier btr135 checkpoint. Its exact rule errors are unavailable: retained CI build d6819795-08b3-4c65-a64c-c6f8708d2bcb returns zero rule rows and a 27-byte rerun-marker log; underlying run reads return 404. This refutes the failing-test premise but does not explain that old build-rule failure.

OBSERVED: Local UTF-8 initial pin run completed 1 passed/1 failed: explicit-shard preparation passed; heartbeat preparation failed because runner fat-jar construction exited 137 before the test body. Cgroup oom_kill=1 accompanied that attempt, but kernel logs cannot identify the killed process directly. The same heartbeat selector on current main with CLI 0.0.93 then passed 1/1. The focused pin heartbeat rerun on unchanged b9d0aa9 also passed 1/1. Total local attempts observed: 3 passed/1 failed. Both selected pin scenarios passed on the same source head; no full suite was run. Successful main and pin runner builds both printed jar SHA256 a5284166f77cd7957c8b3077dac376a3a8606b33a0a77dc265509fc41e6478d4. No source fix or test-limit/assertion change was made, and no local process failure is claimed fixed.

OBSERVED: Main's separate required test failure remains documented at https://buildtest.kotlin.build/run?id=63ed37b7; it was not fixed or attributed to the pin. Artifact-directory highest listed CLI is still 0.0.115; its maven-metadata XML is stale at 0.0.2. The pin was not increased further. Independent test/code reviews found no issue in the three version edits.

OBSERVED: A single plain --pr build-watchman watch was launched after pushing. At cutoff it reported head b9d0aa92 has no dispatched check runs; check suite 100251347361 is queued/in_progress with zero runs. No underlying buildtest URL was available yet. Checks: https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135/checks. The local watcher was stopped at the time box; no remote run was canceled or rerequested. Resume with coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/BuildTestRunner --pr 135 --idle-timeout-minutes 60 (no --to-merged for a delegate).

OBSERVED: PR description now includes why, both handoff links, current local outcomes and pending CI, and the required footer. gh pr edit failed on deprecated repository.pullRequest.projectCards; a structured REST PATCH succeeded and its body was verified. One progress upload returned an uncertain transport outcome, but reports readback confirmed it persisted; it was not duplicated. Tool/retention/metadata issues are recorded in findings for supervisor challenge filing; report-challenge-cli was not invoked because it automatically merges.

INFER: No pin regression was reproduced. The old rule error remains unknown; current selected pin behaviors pass. This is INCOMPLETE only because the new CI verdict remains pending at the explicit time box. If CI fails, reproduce the named failure locally before editing or submitting another head; do not use blind rerequests. The supervisor owns remaining review, any merge decision, and the broader mission below.

btr135 checkpoint 2026-10-02 12:54 UTC

RE-VERIFY: every state below is a write-time snapshot. Re-read GitHub check state and remote branch SHAs before acting. This lane owns only https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135; the original Mission, all prior rows, and the cancellation/launcher/adopted-outcome/CAS/listener follow-ups below remain in force.

Repo Branch Remote head PR State
BuildTestRunner evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/btr135-2026-10-02 2d5123c71cde3ec51787165b00a0f01b2ccadbec none Rebased pin source plus lane-evidence/btr135 observations and raw local failure output; do not merge evidence
BuildTestRunner pin source https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/K15-kompile-cli-0.0.115 d548cd9d0e6e908fcfea5a34db2177978ef94df6 https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 OPEN; unchanged remotely; local rebase b9d0aa924d9745f09a8c5ae7a1eecbd3dedb125c onto main 962511127006ff6d9842042c88e59d29299376ac is preserved in the evidence branch

OBSERVED: The original PR required kotlin.build (remote) check passed 279/279 tests (https://buildtest.kotlin.build/run?id=0c06dde7). Its red informational kotlin.build (kompile-remote-build) check also reported 279 passed/0 failed tests, but five unsuccessful rules: buildtest.runner.buildFatJar(), e2e.heartbeat.cadence.fixture.buildMaven(), e2e.isolated.preparation.probe.buildMaven(), fixture.explicit.shard.buildMaven(), fixture.transported.failure.buildMaven(). CI build d6819795-08b3-4c65-a64c-c6f8708d2bcb retains its summary but returns zero rule rows and a 27-byte rerun marker as its whole log. Old underlying runs 9325c88f and bb38174a return 404. That evidence does not identify the original rule error.

OBSERVED: Latest main required check is red at https://buildtest.kotlin.build/run?id=63ed37b7 (292 passed/1 failed: testE2EWarmLaneDuplicateNoBodyTicketsDoNotBlock). Main informational check separately failed three tests: testE2EMainSanitizesControlCharactersInProcessOutput, testE2EQuietCpuBuildKeepsProgressWatchdogAlive, testE2ESlowBuildOutputKeepsProgressWatchdogAlive. These tests are not this pin lane's scope.

OBSERVED: Fresh local UTF-8 validation on rebased b9d0aa9 completed 1 passed/1 failed. testE2EExplicitShardPreparationDoesNotCopyLiveCache passed. testE2ERunnerHeartbeatCadenceContinuesAcrossDiscoveryAndWorkLoop failed before its body because runner build preparation exited 137. The build child disappeared while JarBuilder.toJar was packaging; cgroup memory.events records oom_kill=1, memory.max=10737418240, but kernel logs are unreadable, so the killed process cannot be directly confirmed. The same heartbeat selector is now running locally on origin/main with CLI 0.0.93. No PR push or remote/CI run was launched because the lane gate is not green. No test limits or assertions changed.

OBSERVED: Artifact-directory listing at https://kotlin.directory/kompile/cli/kompile-cli/ has 0.0.115 as its highest listed version. The maven-metadata XML is stale at 0.0.2. No further bump was made. Independent test and code review passes found no issue in the three pin changes; README has no pin value needing an edit. Testing Architecture and Engineering Philosophy were read fully.

INFER: All-test-pass evidence refutes the initial failing-test premise, but does not explain the informational build-rule failure. Exit 137 and the cgroup event support a local resource-exhaustion failure, without proving the pin caused it. Finish the main comparison before changing source. Then rerun the pin gate, push only a locally verified PR head, and watch one CI run with plain --pr. Never merge, enqueue, deploy, publish, or complete this handoff from the lane. Preserve broader work below for the supervisor.

Snapshot 2026-09-30 19:15 UTC (worker-K15-0930)

RE-VERIFY banner: everything here is a write-time snapshot. Re-check each PR with GraphQL (state, headRefOid, statusCheckRollup) and each branch with git ls-remote before acting.

Mission: land the runner cancellation fix in https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122, then the separate warm-lane follow-ups below.

DECISION (orchestrator): fix the root cause (path a). Moving the manager-0.0.190 code out of the fixtures is rejected. The fixtures really do need PreparedTestBatch: they precompile the real batch before the timed test.

Root cause (settled with evidence):

  • OBSERVED: kompile-buildscript src/BuildscriptCache.kt (release 0.0.33 inside kompile-cli 0.0.93; unchanged on main 0.0.43 at line ~1840) post-processes every compiled script class with a plain ClassWriter(ClassWriter.COMPUTE_FRAMES or ClassWriter.COMPUTE_MAXS). ASM's default getCommonSuperClass loads types through the compiling process's class loader rather than the script's compileClasspath.
  • OBSERVED: a script declaring org.apache.commons:commons-csv:1.10.0 with two sequential CSVParser.parse(...).use { } blocks fails with "Type org/apache/commons/csv/CSVParser not present" under kompile-cli 0.0.93 and 0.0.114. With a single use it compiles.
  • OBSERVED: with kompile-cli 0.0.93, a minimal workspace containing cbe3154's fixture (one prepareTestBatch(...).use) compiles. The same workspace with 799f9e4's aggregate fixture (two sequential uses) fails with "Type kompile/PreparedTestBatch not present".
  • INFER: run 93c907d0 passed on cbe3154 because no script there triggered the lookup. It built the probe artifact fresh, so this was not a stale artifact.
  • OBSERVED: the six kompile:manager pin messages in 06f19c67 are warnings (kompile-core WithArtifactVersionMismatchWarningEffect). The real failure is cut off in that row ("932183 more characters of stack trace omitted"). The pin alignment on 122 is harmless hygiene; it is not the fix.
  • INFER: kompile-cli 0.0.114 only hides the defect for PreparedTestBatch, because its bundled manager 0.0.193 has that class. The Kotlin compile INTERNAL_ERROR seen under 0.0.114 in the fixture shard runner is a separate and still undiagnosed issue. It only matters if someone tries the toolchain bump without this fix.

Fix in flight:

  • https://github.com/CodexCoder21Organization/kompile-buildscript/pull/84 (branch wip/K15-frame-superclass-script-classpath, head b28c34eef48fc6f7467f77352d81dd876fe0062d after one review round).
  • The new ScriptFrameClassLoader searches the script's compiled classes and compile classpath first and falls back to the compiling process. ScriptClasspathClassWriter uses it for getCommonSuperClass.
  • Fail-first test tests/compileWorkspaceComputesFramesAgainstScriptDeclaredClasspath.kts: FAILED on unchanged main (commit d12f5ec4, TypeNotPresentException scriptonly/ScriptOnlyHandle at BuildscriptCache.kt:1840) and PASSED with the fix.
  • Six neighbouring compile tests pass locally.
  • End to end: the fixed jar swapped into kompile-cli 0.0.114 builds the commons-csv script, which fails with the stock jar.
  • The PR's CI and its reviews are still pending. Nothing is published.

PR 122 state: head 34206f1874a9af99463329b715267c21d901d293, rebased onto main e4a64906 with the six test pins aligned to 0.0.190. Main's https://github.com/CodexCoder21Organization/BuildTestRunner/pull/131 already released 0.0.119, so 122 no longer bumps RUNNER_RELEASE_VERSION. The required check (run 6ad7f80e) ran during the 2026-09-30 buildtest incident and is not a verdict. It is expected to fail on the same test until the toolchain carries the fix. Branch wip/122-close-interrupt-F22 (b76d800f) is superseded by commit 1be8e6e on 122 and can be deleted.

Refs: | Repo | Branch | Remote head | PR | State | | --- | --- | --- | --- | --- | | kompile-buildscript | main | 97819d821469586475b856e98ffdff9fcb3cbdcc | https://github.com/CodexCoder21Organization/kompile-buildscript/pull/84 | MERGED; 0.0.44 published | | kompile-cli | https://github.com/CodexCoder21Organization/kompile-cli/tree/wip/K15-buildscript-0.0.44 | c7e710b507d53058c416ff181c5c7c852e0f3732 | https://github.com/CodexCoder21Organization/kompile-cli/pull/179 | MERGED; kompile-cli 0.0.115 published | | BuildTestRunner | https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/K15-kompile-cli-0.0.115 | 4ed3abdcde5be570c8cd115efb2a767cb3b0e635 | https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 | kompile-cli 0.0.115 pin bump; checks running | | kompile-buildscript | https://github.com/CodexCoder21Organization/kompile-buildscript/tree/wip/K15-frame-test-drafts | ddec82f9cb5b97288427b6672cc46341281f002e | none | evidence drafts only, do not merge | | BuildTestRunner | https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/late-warm-ownership-W87 | 34206f1874a9af99463329b715267c21d901d293 | https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122 | waits for the toolchain fix | | BuildTestRunner | https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/K15-122-rebased-pins | 34206f1874a9af99463329b715267c21d901d293 | none | checkpoint copy | | BuildTestRunner | https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/122-close-interrupt-F22 | b76d800fb134ac597e59b150f782402177373815 | none | superseded, delete |

Progress 2026-09-30:

  • https://github.com/CodexCoder21Organization/kompile-buildscript/pull/84 MERGED (97819d82). kompile-buildscript 0.0.44 PUBLISHED and verified: jar sha256 b74ffa9d33e12f5b93344a4145e4b12bffdb087861806feea2dbbb212383415e.
  • https://github.com/CodexCoder21Organization/kompile-cli/pull/179 MERGED 18:04 UTC (db2d6b58). kompile.cli:kompile-cli:0.0.115 PUBLISHED from merged main and verified:
    • jar sha256 a18ee41e66d7693060d9bc68392bb67cd674d701e65fa7b540c36abbe0f5a652 and pom sha256 c1cf39c01cd19410eaff05f0760d6fbf74191cd336470e4d04400377212dc65a, both equal to the local build and the sidecar;
    • the POM declares kompile-buildscript 0.0.44;
    • a fresh consumer resolve yields kompile-buildscript-0.0.44 and manager-0.0.193.
  • https://github.com/CodexCoder21Organization/BuildTestRunner/pull/135 is OPEN (branch wip/K15-kompile-cli-0.0.115, head 4ed3abdcde5be570c8cd115efb2a767cb3b0e635). It changes KOMPILE_CLI_VERSION in scripts/build.bash and scripts/test.bash, and the explicit-shard test's kompile-cli pin, from 0.0.93 to 0.0.115.
    • Local run under C.UTF-8: 5/5 passed (explicit-shard test plus PR 121's four neighbours).
  • The "INTERNAL_ERROR under 0.0.114" is explained: it is a sandbox-locale artifact.
    • Under a POSIX/ASCII file-name encoding, the assembled runner fat jar stores javaslang/(lambda).class (the Greek letter lambda) as javaslang/?.class, and kotlinc inside the runner fails with ClassNotFoundException javaslang.?.
    • The fat jar is identical under CLI 0.0.93 and 0.0.115. Under UTF-8 it is correct and the test passes.
    • The locale sensitivity of fat-jar assembly is a separate latent defect.

What remains, in order:

  1. Orchestrator review and merge of BuildTestRunner PR 135, after its branch checks are green.
  2. Rebase PR 122 onto it. Then targeted remote runs of testE2EExplicitShardPreparationDoesNotCopyLiveCache, the six re-pinned tests, W87's 40 selectors (wip/w87-cancellation-evidence w87/selectors-W87.txt) and testE2EWarmMemoryGateDeniesFourthAggregateReservation. Get the required kotlin.build (remote) green.
  3. Independent reviews of PR 122's final head, covering commit 1be8e6e and fixture commits b0dc93b, cbe3154 and 799f9e4. Then the orchestrator's final review and merge.
  4. Separate follow-ups (unchanged): W66 launcher readiness diagnostics/release; adopted PARKED execution outcome ownership; deterministic proof for the fail() CAS window; failing child/thread evidence for the historical listener first-attempt timeout.
  5. Separate latent defects:
    • kompile-buildscript's four compile-cache tests without @Timeout run at 70-110% of their 30 s default when co-leased on 2-vCPU droplets;
    • fat-jar assembly mangles non-ASCII entry names under a non-UTF-8 host locale.

Previous snapshots (provenance)

Quota-stop snapshot 2026-09-28 18:13 UTC

RE-VERIFY banner (added 2026-09-28 18:25 UTC by fable-code-2026-09-28): everything below this banner is a write-time snapshot taken when the supervising session stopped because its codex delegates hit the account usage limit (reset 2026-10-05 05:43 UTC); re-check every PR with gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,headRefOid,statusCheckRollup and every branch with git ls-remote before acting. Session artifacts (lane briefs lanes/*.md, findings out/*-findings.md, review verdicts out/r*-verdict.md, the supervisor's review log out/supervisor-reviews.md, slot queue out/followups.tsv) are at https://github.com/CodexCoder21Organization/PlanRepository/tree/handoffs-artifacts/2026-09-28-fable-board-drain-quota-stop/handoffs/artifacts/2026-09-28-fable-board-drain . The board-level resume procedure lives in the handoff titled "Resume the open-handoff drain". The shared droplet service dropped its RPC connection twice (17:13?17:18 and again about 18:04 UTC), failing every remote run in those windows before any test executed; runs that report zero tests executed with a provisioning error are infrastructure verdicts, not code verdicts.

State: https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122 OPEN at 799f9e49 (branch wip/late-warm-ownership-W87). The warm-lane 120 s first-attempt fix is on the PR (lane o22 reproduced the warm memory-gate denial 1 of 1 in run 6e3461a5 and fixed it). The required kotlin.build (remote) check fails deterministically on one test out of 276 (runs 0e3e8ea7 manual gate and 06f19c67 required check, 275 passed): testE2EExplicitShardPreparationDoesNotCopyLiveCache ? its bootstrap fixture copies the repository's tests/ tree into a workspace and kompile refuses it because the PR's two new fixtures (tests/testCancelClosesRealBatchBeforeWarmExecution/build.kts, tests/testE2EWarmMemoryGateDeniesFourthAggregateReservation/build.kts) pin kompile:manager:0.0.190 while the pre-existing tests/testBuildRuleDeduplicationPreventsDoubleEmission.kts and tests/testBuildRuleEventsEmittedFromEffectHandler.kts still pin 0.0.154. The non-required kotlin.build (kompile-remote-build) check is red fleet-wide and does not gate. Next steps: run the brief lanes/o22c.phase.md from the artifacts branch ? align every kompile:manager pin under tests/ with build.kts (0.0.190), prove the two older tests pass at that version, record fail-first (unchanged head) and green verdicts by run id, push with --force-with-lease, get kotlin.build (remote) green, then the two independent reviews, the supervisor's final review, and enqueue via the GraphQL enqueuePullRequest mutation.

Review and land the runner cancellation fix, then finish the remaining warm-lane work

Written 2026-09-27 16:45 UTC. RE-VERIFY: GitHub GraphQL, git ls-remote, kotlin.directory POMs and complete paginated buildtest rows override this snapshot. The orchestrator alone merges/enqueues. W87 made no production change and published no artifact. Keep this same handoff for the remaining work.

W87 result

CHECKPOINT: scoped engineering, both reviews and targeted gates complete; required PR CI still active. https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122. Current source aaefd0cb32b7a5077e1261c94993611002b58e7c at https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/late-warm-ownership-W87. It is rebased on main bf535bb01acdf3f19dc6e16573b3cc057cc06304. The original W87 scope is complete at source: verify the origin guard, resolve the real-batch closed-before-execute error, resolve the queued-cleanup fixture error, run the original 33-selector gate, review and open a runner PR. Required CI state: required kotlin.build (remote) run https://buildtest.kotlin.build/run?id=5c6560f1 BUILDING at 16:44 UTC, with 276 PENDING / zero executed at 16:45 UTC; informational kotlin.build (kompile-remote-build) pending. One empty-suite dispatch rerequest started both checks; no failed test was retried.

The original 33 selectors plus seven focused controls/review cases passed 40/40 on af427acf587faf1d5ba9bae537c2bb4a1d4216cb in https://buildtest.kotlin.build/run?id=592a9cf6. All were first attempts, no retries; 236 FILTERED_OUT rows are excluded. testE2EWarmMemoryGateDeniesSingleReservationBeyondBudget passed in 88406 ms; the F7 32-caller test passed in 3646 ms; queued-cleanup passed in 2982 ms; real-batch close-before-execute passed in 63095 ms. The final test-only assertion strengthening at aaefd0cb was reviewed by both independent reviewers. Final head passed four focused checks locally through the JVM-slot wrapper, 4 passed / 0 failed; the preceding remote submission executed zero tests because allocation failed.

Demonstrated mechanisms and fixes

  1. The origin guard at W81 d7b90b9 is verified: local testPreparedHandleCancellationOriginRemainsCleanup passed 1/1 in 995 ms under the shared JVM slot. Its unchanged pre-fix failure is retained in W81 evidence.
  2. Real-batch cancellation: closeReady closes the prepared handle while its warm thread has not entered batch.execute. Manager 0.0.188 throws a general IllegalStateException for that terminal batch state; W81's classifier recognized only admission cancellation. A public real-batch latch forces close to finish before execute. It failed on unchanged production in https://buildtest.kotlin.build/run?id=6597df43 (11825 ms), while the independent same-text error control passed. Runner now consumes the already-published manager 0.0.190 and recognizes its PreparedTestBatchClosedException only for fully cancelled CLOSING groups. Cause traversal remains bounded and any suppressed independent error still survives. No text matching and no upstream publication by W87.
  3. Queued cleanup fixture: only one speculative preparation is admitted at once. W1 preparation leaves W2 as a one-item tail behind the existing 250 ms collection deadline. The fixture froze its ManualClock at zero while awaiting both warm workers. The assertion failed before cancellation, and final cleanup then blocked on W1's worker-only close latch. Diagnostic 6597df43 captured the event order and full child thread snapshot before releasing diagnostic holds: only WARM W1, ready=1/queued=1, then main-thread final cleanup and the original warm-latch assertion. The fixture now drives the manual deadline through the public capacity callback. The normal 20-second limit and all ownership/error assertions are preserved; final assertions also reject premature attempt completion and duplicate saved-error attachment.
  4. W81 review gaps are covered: two handle-close/outcome orderings, separate attempt and lease delivery failures with stored cleanup errors, and both terminal boundaries while outcome/cleanup is pending. Both independent final reviews found no remaining scoped issue after the assertion correction. Review artifacts are w87/test-review-W87.md and w87/code-review-W87.md on the evidence branch.

Relevant refs

Repo Branch Remote head PR State
BuildTestRunner source https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/late-warm-ownership-W87 aaefd0cb32b7a5077e1261c94993611002b58e7c https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122 final reviewed source, validation above
BuildTestRunner evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/w87-cancellation-evidence 93a12e4bcd866f0b1fa2f8e82f1063a793ba15d8 (validation snapshot; terminal snapshot appended below) none w87 exact commands/results/thread evidence/reviews/report; inherited w75 and w81 preserved

Historical refs below are preserved for provenance; W87 supersedes W81/W75 for the scoped runner fix.

Repo Branch Remote head PR State
BuildTestRunner https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/late-warm-ownership-W81 d7b90b937fe4b612090c5c863f9c412a97292a3b none W75 ownership candidate rebased on main, close wrapper,7boundary tests, origin reproducer/guard; gate RED on64c390f, final guard unverified
BuildTestRunner evidence https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/w81-cancellation-evidence cf350b7024836efd80288c2efd4a586510a8b9a7 none w81 findings, all API snapshots, exact stacks/logs/XML, two reviews, unrun review drafts, selected and repeat commands; inherited w75 evidence retained
W75 source https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/late-warm-ownership-W75 94e6c27a0b0d94bd82432d17b9d2bca88751e4ea none Superseded for continuation by W81 source; do not reapply old candidate patches
Launcher source https://github.com/CodexCoder21Organization/community.kotlin.kompile.testrunner.jvm/tree/wip/relay-cleanup-W66 f72640aa2fedfb04c99ab11f960b5d50056c2a3e none Separate follow-up, unchanged by W81
Launcher evidence https://github.com/CodexCoder21Organization/community.kotlin.kompile.testrunner.jvm/tree/wip/w66-cancellation-evidence 890e3014e572a81e5d22a19aa2304396878c2e9f none Readiness diagnostics and prior fail-first evidence

Next steps

  1. Resume build-watchman for https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122 at 300-second cadence, without --to-merged; finish active required run 5c6560f1, then orchestrator alone performs final review and decides merge/enqueue. No new source change remains in this lane scope. The local watcher was stopped at lane exit; no remote run or production service was stopped.
  2. The latest W87 instruction required the same 33-selector gate before PR creation, and that gate is green (40 total with controls). No eight-repeat batch was launched in W87. The latest W87 continuation narrowed its stop condition to the 33-selector gate; the older eight-repeat instruction was not carried out, and no repeated-run result is claimed.
  3. Explicitly separate remainder: finish W66 launcher readiness diagnostics/release and consume its future published launcher; finish already-adopted PARKED execution outcome ownership; add deterministic proof for the fail() CAS window; obtain failing child/thread evidence for the historical listener first-attempt timeout. Manager 0.0.190 and executionenvironment 0.0.38 are now consumed on the W87 candidate, while launcher remains the published 0.0.29. The new typed close classifier does not claim to solve adopted execution outcomes.
  4. https://github.com/CodexCoder21Organization/BuildTestRunner/pull/120 remains OPEN/BEHIND at 17c0b7bf5faac2e0c03f7971802a96b0ff191b21 with failed checks; it was not changed. https://github.com/CodexCoder21Organization/BuildTestRunner/pull/116 and https://github.com/CodexCoder21Organization/BuildTestRunner/pull/121 are MERGED. https://github.com/CodexCoder21Organization/kompile-core/pull/339 is MERGED and manager 0.0.190 is published. Runner 0.0.119 remains reserved and unpublished.

Commands and operational evidence

From a fresh lane layout with source in ws/BuildTestRunner and the w87 evidence copied into evidence/, run bash evidence/run-selected.bash for all 40 selectors. bash evidence/run-final-assertions-local.bash uses the shared JVM-slot wrapper and only four focused tests. No full local suite was run.

Remote final-assertion run https://buildtest.kotlin.build/run?id=5a284469 failed during allocation: digitalocean-droplets listDroplets transport closed after request write, and the coordinator could not verify cleanup. It executed zero tests (4 PENDING, 272 FILTERED_OUT), so it is not a failing test result. W87 used the local focused fallback. The raw allocation log and challenge note are preserved.

build-watchman 0.0.18 still counts FILTERED_OUT rows as completed and reads only the default 100-row page. W87 kept the mandated watcher but counted complete durable rows separately, excluding FILTERED_OUT. The initial diagnostic timeout lost child stdout; the second diagnostic saved its full trace in an assertion after capturing the stuck state and releasing fixture holds. The diagnostic-only script was removed from the source diff and retained in evidence. One remote client printed UrlProtocol2 host.stop TimeoutException warnings after the successful 40-test result; its exit was 0.

No production deployment, restart, route/data change, direct cloud administration, artifact publication, merge, enqueue, protection change, test disabling, limit increase or assertion weakening occurred. report-challenge-cli was not invoked because it automatically merges; its observations are preserved for the orchestrator in w87/allocation-challenge.md and w87/watchman-filtered-challenge.md.

Terminal evidence snapshot: https://github.com/CodexCoder21Organization/BuildTestRunner/tree/wip/w87-cancellation-evidence @ dca044e332ea5782bc0d5612131f2873237d6e44.