← Handoffs

Repository · handoffs

Handoff: Restore the runner gate and finish fleet policy verification

View on GitHub ↗

id: hf-2026-09-30-buildtestrunner-rotating-remote-timeouts-choose-and-land-a-test-child-jvm-tier-policy-that-recovers-the-suite url: url://handoff/handoffs/hf-2026-09-30-buildtestrunner-rotating-remote-timeouts-choose-and-land-a-test-child-jvm-tier-policy-that-recovers-the-suite title: Restore the runner gate and finish fleet policy verification summary: Restore provisioning, resolve the retained process-accounting failure, and obtain a clean exact-head required gate before supervisor review of the current pin PR. Scaling5 adoption and Embedded runner122 pins are already landed; admission/cancellation validation and operator running-byte verification remain. Current required check failed before runner launch. created: 2026-09-30T20:19:20.817Z completed: null blocked-reason: Required runner gate failed before execution: the droplet create response was lost after transport EOF. Provisioning recovery and clean validation are required. dependencies:

Handoff: Restore the runner gate and finish fleet policy verification

Written 2026-10-04T16:13:33.815292+00:00. RE-VERIFY: This is a write-time snapshot. Re-check handoff claims, GitHub heads and required checks, served Maven POMs/JAR digests, and the live coordinator's actual running versions before acting. The required kotlin.build (remote) check is the merge authority; prior local final passes do not replace it.

Mission summary

The original mission was to choose and land a test-child JVM tier policy that recovers the rotating remote-timeout suite. Historical investigation measured C1, scaling10, scaling5 and default; it did not establish one common cause for all failing tests. Scaling5 was selected and landed upstream. This continuation re-verified that adoption, compared the later manager pin proposals, and reviewed the exact current pin head. Remaining work is provisioning recovery, clean validation, the original admission/cancellation findings, and operator verification of actual deployed bytes. No merge, enqueue, publication, deployment, or handoff completion was performed in this continuation.

What was found and done

  1. Historical evidence at the measurement branches below reported 20 tests with four child callers: median total CPU378s C1,414s scaling10,422s scaling5,519s default. Every policy still had failures; those are historical observations, not proof that changing one JVM flag resolves every victim. Preserve the measurements and separate root-cause insights; do not remove failing tests or raise their budgets.
  2. Scaling5 merged upstream via https://github.com/CodexCoder21Organization/kompile-core/pull/350 at commit https://github.com/CodexCoder21Organization/kompile-core/commit/da5d7f40256e3db61a65f5322dc7a534b0dcbcfc . Runner manager198 adoption merged via https://github.com/CodexCoder21Organization/BuildTestRunner/pull/139 at https://github.com/CodexCoder21Organization/BuildTestRunner/commit/7f7de523789fd8151723fae48509470c216b0b76 . Release120 merged via https://github.com/CodexCoder21Organization/BuildTestRunner/pull/141 at https://github.com/CodexCoder21Organization/BuildTestRunner/commit/962511127006ff6d9842042c88e59d29299376ac . These original adoption steps are done.
  3. Fresh metadata latest manager205 and runner122; served runner120/121/122 POMs all pin manager198 and executionenvironment38. Runner122 bytes have identity122 and SHA256547ee83616f856302d2b7d92bb9ff308b27f9278757a3d75f3e747d3c445fc10. Embedded main pins runner122/manager198 in both SERIAL and PIPELINE at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/054b253144d6c4f72f24a6e15922098cd0941afe . Latest Embedded is0.0.69274749. This pin step is done. Manager205 JAR also contains the scaling5 child flag, independently checked by class bytes and current source.
  4. Published-but-unmerged discrepancy: runner main4377987 still declares release120. Published runner122 comes from the identity-only commit e40e465e below, not on main. Preserve that branch and reconcile source identity before a future release; never overwrite published120/121/122. The current manager205 pin PR publishes no new runner.
  5. Current https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145 at cf8a405 is NOT ready for merge. Required check failed in https://buildtest.kotlin.build/run?id=495cc5ef . Raw log shows createDropletAsync response transport EOF after request write began; shard0 failed before workspace preparation, followers were not created, and no runner launched. Synthetic run counters0/313 are not313 executed test failures. The read API additionally declares outOfDate=true/refreshFailing=true. No remote run or check was requested here.
  6. Two separate review passes at exact cf8a405 are in the evidence findings. The ten-file diff changes pins/README/launchers only, no test body, budget, assertion or implementation. Six direct manager fixtures move with production; the intentionally older producer fixture remains unchanged, but its annotation alone does not establish an isolated old runtime. Existing mirrored dedup code is formatter coverage, not proof of runner integration. No pin-code edit was identified. Prior-session XML records7/7 targeted and313/313 final full-suite, with312 first-attempt passes and one first-attempt process-tree accounting failure: tree3870ms versus children4000ms. Its automatic second pass does not establish that defect fixed. The entire stack is preserved in imported prior findings.
  7. Older https://github.com/CodexCoder21Organization/BuildTestRunner/pull/142 has manager200/EE42 in the same seven declarations and only release121 metadata beyond that. Current proposal has205/44 and CLI121; no unique test body or implementation is lost by closing the old proposal. Since published121 contains198, that release121 proposal cannot now publish as written. Recommend supervisor close it as superseded; it was not closed here because its exact content is not on main.
  8. Original two consecutive main fleet gates remain unproven and blocked. The build-only admission latch still includes cold buildscript work on current head; passing it once at9749ms does not fix or refute its historical intermittent failure. Cancellation/close ownership proposal below remains open, red and conflicting; current warm/single reservation passes once also do not prove those defects fixed. Keep those original next steps for their existing owner rather than copying an unreviewed fix into the pin PR.
  9. Live ContainerNursery route url:buildtest: is SERIAL and names /root/ContainerNursery-uploads/jars/buildtest-server-302856bf-20261004-deploy6.jar. The named source commit https://github.com/CodexCoder21Organization/BuildTestServerService/commit/302856bf97ecaabfdce7da6af83c0ad078b8e09f pins Embedded49; filename alone does not prove running bytes. Published server0.0.26709273972 instead has an Embedded43 POM, so launching that coordinate is not an Embedded49 deployment. The conditional source-build/operator-upload recipe is linked below.
  10. Overlap: the occurrence/tail-stall handoffs' manager publication prerequisite is obsolete (manager200/205 published), and their runner proposal below now has both checks green. This lane did not prove their original10/20-run execution gates or exact occurrence review, and neither overlapping handoff was edited or completed.

Relevant PRs / refs

PR/ref Remote head or artifact identity Write-time state
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145; release/pub1-manager-cli-pins cf8a4058d6d14787cec6ece0607b07f6d8d1e874 OPEN; kotlin.build (remote) FAILURE, kotlin.build (kompile-remote-build) IN_PROGRESS
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/142; release/runner-0.0.121-manager-0.0.200 67e3da0a5bb5a15c08818b150f77d89d3a88ff82 OPEN; kotlin.build (remote) FAILURE, kotlin.build (kompile-remote-build) SUCCESS
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/122; wip/late-warm-ownership-W87 34206f1874a9af99463329b715267c21d901d293 OPEN; kotlin.build (remote) FAILURE, kotlin.build (kompile-remote-build) NEUTRAL
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/125; wip/p35-pipeline-tail-stall-2026-09-29 8d536c0988c501fbadd30c57cb29e085c804cea5 OPEN; kotlin.build (kompile-remote-build) SUCCESS, kotlin.build (remote) SUCCESS
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/139; wip/btrbump-manager-0.0.198 f8ea3835af33b52d052d5180afbb8d45a446775c MERGED; kotlin.build (kompile-remote-build) SUCCESS, kotlin.build (remote) SUCCESS
https://github.com/CodexCoder21Organization/BuildTestRunner/pull/141; release/runner-0.0.120 643d0e74e8db2ad5bbb7e3a1994911957b1056c1 MERGED; kotlin.build (kompile-remote-build) SUCCESS, kotlin.build (remote) SUCCESS
Scaling5 upstream proposal; wip/jitflag-compile-threshold-2026-09-30 5d5b2631b6eb8e56dfc7a59fb1e8e0cb708fe6c1 MERGED; all six checks SUCCESS
CLI121 release; wip/PUB86-2026-10-04 c23ae0f57f5493f51f8ddd8261511f60d20f575c MERGED; both checks SUCCESS;121 served
release/publish-runner-0.0.122-runner122 e40e465e0bc0a3cdd29e6814e20fbbcea47d41d1 Published122 identity change; not merged; preserve/reconcile
wip/L26-tier-policy-review 698ccd8c2f2dc7aea0b8f765573996918538ef19 Evidence only; no product PR; preserve
wip/PUB87-evidence-2026-10-04 ae19f2c02d159a851f26599265e4a9f3b5fc4ea2 Prior-session exact-head gate evidence; preserve
wip/K48-tier-policy-measurement 82d15bdd442a34c768e57df9d67da27f0981e27e Historical policy measurement only; preserve
wip/K38-c1-cap-evidence 1e61131b19c2add7d0f3295466a921ba3a2063cc Historical C1 evidence only; preserve
Runner main 4377987a08c924ebee823f5dee97614a9c4c423b Release identity120;205 pin not merged
Embedded main 13ae9f5c6d78f2b0bb57e7160629f716fdb2b012 Both runner tuples122/manager198
Runner122 served artifact JAR SHA256 547ee83616f856302d2b7d92bb9ff308b27f9278757a3d75f3e747d3c445fc10 Published,198/38/bootstrap16
Manager205 served artifact JAR SHA256 35533eeb95361dc6704942de656fb9b6ba1fa295330b128da842ed8ee3b8db27 Published, scaling5
Live coordinator route url:buildtest:, image filename302856bf-deploy6 Running byte identity not established

Next steps

  1. Re-verify current state first, including claims. The primary gate is BLOCKED_INFRA: required remote check fails before execution. Have the provisioning/service owner trace the request-write/response-EOF mechanism and reconcile any potentially created droplet; do not repeat the ambiguous create or requeue a failed check hoping for green. This lane's transport evidence does not identify the upstream code defect, and no speculative fix was made.
  2. Diagnose and reproduce the retained first-attempt process-tree CPU accounting mismatch through the public runner path. Fix upstream at its actual owner if confirmed; no timeout increase, iteration reduction, assertion weakening, test deletion, retry or capacity explanation. Once infrastructure is healthy, run the required exact-head full gate and wait to terminal with bounded build-watchman plain mode. Do not mark the pin proposal ready while required remote is red.
  3. Supervisor final review of the two separate review findings; keep current pin proposal and close the older one as superseded if satisfied. Only supervisor may merge/enqueue. If main/head changes, rebase, repeat the applicable gate and review the changed exact head.
  4. Reconcile the published122 release-identity branch with main. Future205 adoption requires a new unused runner release, authorized publication, Embedded version/hash pin and validation. No205 fleet artifact was produced here.
  5. Retain original focused admission test-design decision and cancellation/close validation. The cancellation proposal must be rebased and its failing-first public reproducer and exact-head checks verified by its existing owner; the two original fleet main gates remain outstanding.
  6. Operator-only production decision: inspect runtime artifact digest/identity first. If the fleet still needs the approved Embedded49/runner122/manager198 source build, approve the conditional deployment recipe at https://github.com/CodexCoder21Organization/BuildTestRunner/blob/wip/L26-tier-policy-review/investigation/L26/operator-recipe.md . Target is url:buildtest: on https://api.nursery.wasmserver.com . Confirm all old writer processes fully exited per Embedded README Deployment when changing the publication-lock protocol. The published server972/Embedded43 coordinate is not the proposed49 build. Do not redeploy merely because the filename is inconclusive.

Operational knowledge

  • On this host export PATH=/code/ws/bin:$PATH. Use cs or java -jar /code/ws/bin/coursier.jar; /usr/local/bin/coursier is wrong architecture. If a launcher cannot accept it, add a clone-local executable shim for that script only.
  • The provisioning service is currently unusable for these gates. No new remote run or check rerequest was submitted. Full local builds/suites are prohibited on the shared6GiB container. Preserve the retained exact-head prior XML/logs; test execution by this continuation was0.
  • Use build-watchman0.0.21 only for a gate being awaited, as a tracked bounded process; never --to-merged, never a hand-written indefinite polling loop. Poll GitHub no more than once per60seconds.
  • Evidence/reviews: https://github.com/CodexCoder21Organization/BuildTestRunner/blob/wip/L26-tier-policy-review/investigation/L26/findings.md ; transport log: https://github.com/CodexCoder21Organization/BuildTestRunner/blob/wip/L26-tier-policy-review/investigation/L26/log495cc5ef.txt ; imported prior findings retain the full first-attempt stack. Initial prior-lane evidence was separated explicitly so its publication/testing actions are not attributed to this continuation.
  • No separate report-challenge CLI was run: it automatically enqueues/merges, which this lane forbids. Infrastructure details remain durable in the evidence branch and this handoff.
  • Previous lane labels in evidence are historical investigation identifiers only, not source concepts or runtime settings. Keep them out of product code and PR prose.