← Handoffs

Repository · handoffs

2026-10-02 · Finish the kotlin build ci rerun admission first and completion gateway cancellation pull requests decide the second round review findings re run the gates merge both

View on GitHub ↗

id: hf-2026-10-02-finish-the-kotlin-build-ci-rerun-admission-first-and-completion-gateway-cancellation-pull-requests-decide-the-second-round-review-findings-re-run-the-gates-merge-both url: url://handoff/handoffs/hf-2026-10-02-finish-the-kotlin-build-ci-rerun-admission-first-and-completion-gateway-cancellation-pull-requests-decide-the-second-round-review-findings-re-run-the-gates-merge-both title: Finish invariant and observer focused/CI gates summary: Observer candidate passes guards2/2 and strict mixed5/5; focused remote health RPC failed. Shared PR source unchanged by L3p; production work remains below. No merge. created: 2026-10-02T21:40:09.715Z completed: null blocked-reason: Observer fix passes strict acceptance; 132-selector gate could not start because buildtest health RPC timed out. Focused and candidate-head CI gates remain beyond the lane hard stop. dependencies:

RE-VERIFY — L3p final checkpoint, 2026-10-05 14:11 UTC. Observer correctness and strict runtime acceptance are proven. Focused verification and candidate-head CI are not. The handoff remains open; preserve L3o's independent production state below. No merge, enqueue, deploy, publication, completion or claim release.

L3p proven test candidate and remaining gates

Candidate https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-jimfs-candidate-20261005 is pushed at ba623dcc440911450c9e145f9116438b04c5cf22. Its diff from the shared PR is exactly three test files. It uses the correct single-worker park-blocker observer and Jimfs1.3.2 through Okio's public NIO adapter for tracker storage. Every tracker read/publication,100rounds,44fresh+10recovery callers, original FIFO/class assertion and300s test cap remains. Real download/data files remain.

  • Old-predicate preparation scenario failed0/1 at the intended assertion; correct candidate guards pass2/2.
  • **Five consecutive strict mixed first attempts pass at the exact candidate:**126490,102713,109414,93468,114807ms, each below150000ms. Fresh test JVM per invocation, one attempt, no JFR during acceptance.
  • Three-per-observer sampled comparison completed at identical c71 production: OLD=[cap300s,161965ms,cap300s]; PARK=[194714ms,cap300s,294106ms]. Capped outcomes are lower bounds. Both predicates vary by more than100s; no repeatable extra50s park-predicate cost was established. Native samples center on tracker filesystem work. The existing byte-at-a-time fake fixture also capped on its first unprofiled attempt; the bulk-copy fixture then passed all5required timings. This does not give a precise causal overhead estimate.
  • 132-selector focused gate has no verdict. One exact-source remote attempt exited1 after38.11s, before tests or a run ID: [FAILED] Build service RPC 'health' to url://buildtest/ failed with TIMEOUT: Request timed out after 30000 ms. No rerun and no remote-run existence claim.

PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/296 was checked at14:07UTC: OPEN/CLEAN, headc71d60566003ae73dccd558a092eac3546080b1e, old-head bld-build SUCCESS. That CI does not cover this candidate. This lane did not promote source or patch the PR description without its focused gate. The named kotlin.build(remote) check conflicts with the repository's actual required bld-build policy; no status or protection change was invented.

Resume at the132-selector gate using the manifest in the evidence branch, remote when health works or serialized local execution with adequate time. Then rebase/integrate against current main and live PR, re-verify strict timings if the tested source changes, push test changes with an exact lease, REST-PATCH the description and get required CI green. The110-minute lane bound prevents those remaining gates here; this is a partial salvage result, not completion or a request to relax acceptance.

Evidence: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-evidence-20261005 at 54711aa8cdcfeeddc790639bb8bfe1a400bef64e, under handoff-artifacts/l3p/. It contains FINAL findings, full stacks, all timings, original132manifest, JFR data/summarizer, and one-attempt public-workspace runners. Workarounds and my corrected harness/reporting mistakes are recorded explicitly there.

L3p durable branch Remote SHA
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-evidence-20261005 54711aa8cdcfeeddc790639bb8bfe1a400bef64e
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-inmemory-candidate-20261005 5018f127d307576a8848246dba246dcceb0be16b
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-jimfs-candidate-20261005 ba623dcc440911450c9e145f9116438b04c5cf22
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-observer-failfirst-20261005 86c284fdaeca383c58c2222c704e171e8f03cb6f
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-old-observer-measurement-20261005 033ac35294a523588bc22551b89e5f7062dd3732
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-old-sampling-only-20261005 1fd36f987af08fa96bb2edf407fae6f450f02677
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-park-observer-measurement-20261005 7dd5a80dea1c89f6d0207b1f9404362d46d4684f
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-park-sampling-only-20261005 5f261e26a49b20f00ec98daf8bb9c9e3232ab104

RE-VERIFY — 2026-10-05 14:10 UTC. This is an in-flight snapshot. Re-read live PR and branch heads before continuing. Do not merge, enqueue, deploy, publish artifacts, complete this handoff, or release the orchestrator's claim.

Mission and live state

Finish rerun admission before replacement, completion-gateway cancellation, and tracker/receipt correctness under concurrency. The owner is fable-drain-20261004. PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/296 was last verified OPEN/CLEAN at c71d60566003ae73dccd558a092eac3546080b1e with bld-build SUCCESS. Main e3f692ba148d2d46b74cb59dd395f85dd20e230d was fetched/rebased before the latest targeted build. The cancellation PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/297 was already merged at 401d11da5a6ac1caad2c6fa3f0d23169681897df (GitHub re-verified at13:34UTC).

L3o verified invariant work

Candidate https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-invariants-20261005 atb7f42b9731dc355cfe2d2f08516e34904d813be7 passed146/146 focused selectors, with146 unique verdicts and no reported internal retries. This includes the original132 selectors,11new regressions and3existing completion-cancellation cases. It is not yet on the PR.

  • Repository allowance evaluation and slot commitment share storage ownership. Sixteen genuine distinct-head callers cannot spend the same final slot. Nested callback reads see the reserved slot.
  • Registration failure after durable tracker publication finishes the placeholder so the watchdog can recover it. Failure before publication releases unused allowance.
  • Receipt existence/read sequences share the pruning guard. Active comments retain ownership across callbacks, tracked replacement receipts survive until cleanup, and expired active comment decisions renew on completion.
  • Successful sweep completions write terminal ownership so receipts eventually expire. Ambiguous completions retain registration ownership.
  • A real child-JVM peer-write test passes on c71 and refutes the claimed missing peer-cache mutation for that scenario. No unsupported cache rewrite was added.

Independent review found one added cleanup path: expired-receipt renewal failure skips scheduled recovery of an already-published replacement. Public test testRerunRegistrationRecoverySurvivesReceiptReleaseFailure fails0/1 on2ec exactly at the missing scheduled worker after verifying both I/O failures and durable publication. Test branch0de9421a7bcb590e30bb44c7ea4644efc02defc7 is in the table below. The final candidate puts recovery in the release cleanup finally. The unchanged regression then passed1/1 and is included in the final146/146 focused pass. Complete failing and passing output is in the evidence branch.

Remaining L3o gates and constraints

The final candidate b7f42b9731dc355cfe2d2f08516e34904d813be7 passed146/146 focused selectors. Full-suite acceptance is BLOCKED: run https://buildtest.kotlin.build/run?id=1598acd2 contains1162 tests and its gitContext matches b7. Last successful direct read13:58UTC showed PENDING,0/1162. The client exited1 after getBuildRun timed out at the existing30000ms limit; its cancellation request lost the current RPC stream and replacement health request also timed out. A separate public getBuildRun read failed too. Remote cancellation is UNCONFIRMED. No full-suite test verdict exists and no replacement run was submitted. Reconcile this run before another submission; the observations do not establish a transport root cause.

The candidate is safely pushed on WIP, but live PR https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/296 remains OPEN at c71d60566003ae73dccd558a092eac3546080b1e with old-head bld-build SUCCESS. That CI does not validate b7. After the exact-source full suite passes with zero internal retries, integrate accepted observer changes and current main/live PR, rerun affected gates if the source changes, force-with-lease push and obtain green required CI. The PR description already opens with WHY and accurately describes the current head, candidate and missing gates via REST PATCH.

No whole local suite, mixed-observer/FIFO/count edit, iteration/deadline reduction, excluded-repository edit, merge, enqueue, deploy or publication. All local clients/watchers were stopped and owned-process absence checked; this does not establish remote cancellation. Final findings/evidence are durable, and a BLOCKED report is being uploaded. The orchestrator must resolve the check-policy mismatch below and arrange continuation beyond this lane's150-minute stop.

The lane template requires a kotlin.build(remote) GitHub check, but this repository README and live rules require only bld-build under its runner-only policy. The orchestrator must resolve that acceptance mismatch; no check is fabricated and no policy bypass is authorized.

Durable L3o branches

Branch Verified remote SHA
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-evidence-20261005 15973702f2983b618e4a8271c31f646ca8ec8057
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-invariants-20261005 b7f42b9731dc355cfe2d2f08516e34904d813be7
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-last-slot-reproducer-20261005 5e2cc467da464f15dc40f5b87aa588fd0e9b693c
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-nested-failfirst-20261005 179b121b900d5b84876281d82ad77bec81fee37e
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-peer-cache-20261005 39bb7d08f0fdc512a28627979b3a569fb827276b
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-receipt-failfirst-20261005 4c88381412842a45bfc4552a1bc511902de8963a
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-registration 1129e25b720cd0c97f073d42da229ba8276ccdf6
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-release-recovery-20261005 0de9421a7bcb590e30bb44c7ea4644efc02defc7
https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3o-sweep-terminal-20261005 fd4cae5affa3670144675fe2a8bab0fc32ce2ed2

The evidence branch has advanced to15973702f2983b618e4a8271c31f646ca8ec8057. Its diagnostics/L3o directory holds exact selector manifests, full failing and passing outputs, invariant table and incremental findings. Local findings: /tmp/claude-1000/-code/5d1ba4b3-2f5d-4da2-95e5-39da42464dfb/scratchpad/out/r-L3o-findings.md.

Observer lane snapshot, preserved from the last readable shared body

RE-VERIFY — L3p observer progress, 2026-10-05 13:02 UTC. The live shared PR remains at c71d60566003ae73dccd558a092eac3546080b1e. No observer fix has been pushed there. Production invariant work below remains owned by L3o and is preserved.

The old WAITING-only observer failed the public preparation scenario at its intended assertion (0/1 pass, 1748ms). Source: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-observer-failfirst-20261005 at86c284fdaeca383c58c2222c704e171e8f03cb6f. Evidence and full stack: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-evidence-20261005 at a7a4a33a (verify current remote SHA before reuse).

The initial matched profiling pair was OLD233047ms/PARK234565ms, followed by PARK134473ms at identical source. A later old run reached the cap and the CLI started another attempt; it was stopped and is not accepted evidence. A revised six-run comparison uses sampled execution/native stacks only and the published workspace API with exactly one attempt and one test worker. Both variants retain c71 production and all caller/iteration/assertion/deadline settings.

Candidate test source is checkpointed, not yet validated: https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/l3p-inmemory-candidate-20261005 at b6b60fb4f5aeed94131be4562a53ebc78b2aac9a. It combines the park-blocker observer with the existing synchronized in-memory tracker filesystem. All tracker reads/publications and54callers x100rounds remain; real download/data files remain. Native samples in both observers concentrate on tracker snapshot publication and persistence locks; the candidate must prove the cost reduction before adoption.

L3p remaining: finish3old/3park comparison; validate the candidate and correct-observer scenario; five consecutive mixed first attempts below150s at final source;132-selector focused gate; rebase onto live shared PR, push test files only, required CI and PR-description update. No merge/enqueue/deploy/publication/claim release/completion. Hard stop14:14UTC.

Full prior history

The full shared handoff body was preserved before this rewrite at https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/2f9e08cb23afd638b0a5e2d15fe9381b2091e91f/diagnostics/L3o/handoff-full-before-compaction.txt . Read it for the earlier review rounds, runtime/observer evidence, historical branches, root-cause hypotheses and original acceptance details. This rewrite removes duplicated history from the live body without discarding it.

Reporting friction

The13:16 update exited143 without acknowledgement; the13:26 update reached its60-second command bound. Subsequent successful reads confirmed neither body update landed. One verification read failed with an ambiguous getHandoffFile transport outcome (stream ended after0of4bytes); later reads succeeded. RUNNING reports were acknowledged separately. This smaller current-state body preserves the full prior history remotely; do not infer a transport root cause from the size change alone.

Final L3o evidence

Final report, all baseline/fixed output and exact selector list: https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/15973702f2983b618e4a8271c31f646ca8ec8057/diagnostics/L3o/r-L3o-findings.md . Final full-client failure: https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/15973702f2983b618e4a8271c31f646ca8ec8057/diagnostics/L3o/full-final-1162.log . Independent status failure full stack: https://github.com/CodexCoder21Organization/kotlin-build-ci/blob/15973702f2983b618e4a8271c31f646ca8ec8057/diagnostics/L3o/final-run-direct-1402.json .