Finish four watchdog gates and remaining durable-admission fixes

RE-VERIFY: 2026-10-09T05:26:28.082832+00:00 CHECKPOINT at the lane time box. Re-fetch main, PR states and this body before acting. Historical conclusions are hypotheses. The supervisor owns every merge, enqueue, deployment, publication and handoff completion.

Mission

The original request is near100% BuildTest droplet utilization with both blocking durable-admission defects fixed. Preserve the approved conservative migration, completed-static cleared-field/detail-history policy and settled fixture rulings. Immediate conclusive no-create return followed by FIFO reacquisition is approved. Run10 approves a separate post-proof unit snapshot while preserving every inherited assertion, including exactly one total snapshot in the interrupted copied-boundary fixture.

Approved fixture decision and run11 checkpoint

The supervisor explicitly APPROVED establishing a complete fleet observation through public operations while preserving every inherited assertion. The alternative nullable observer assertion is rejected. The earlier separate destruction/post-proof snapshot ruling remains approved. Do not ask these questions again.

The fresh run11 candidate is cf95ec2c53d934525fa570fbfd5ac3a6de4330bb on main f87fd2d969d2c78b3a17104f7ca65583f5ca012a. Approved setup submits a real held capacity waiter, awaits both its wait boundary and the existing completed-fleet-observation callback, checks a non-null fleet snapshot, cancels the waiter, and requires zero provider creates plus unchanged original admission state, status and destruction history. All24 inherited check/assert calls are unchanged.

The corrected fixture compiled and failed3/3 automatic attempts at the original missing-unit assertion on unrepaired-watchdog baseline5e4de22d92e6d87ffdd3cdeb419ce77be64e6219 (mainb865c4a8). All preceding setup and original proof/status/request assertions passed. Final scenario count0passed/1failed; this is the expected old-code failure. Full XML, console and three traces are pushed in run11 evidence under baseline-proof.md and watchdog-baseline4.. Baseline execution started04:02:17UTC and completed within its1800-second bound. Candidate cf95ec2c compiled and passed this same regression on its first attempt: 1 passed, 0 failed, no retry. Full output is saved in candidate-proof.md and watchdog-candidate3.. The later inherited assertions for exact snapshot count, public fleet re-entry, history and copied-record restart all passed. The baseline/candidate fixture bytes are identical. No source PR push.

Before the latest main rebase the only baseline-to-candidate difference was the34-line watchdog production delta. The newer main changes bound archive test extraction and validate runner timestamps; the relevant watchdog/helper changes remain intact. Re-verify and preserve those changes before further builds. Two of six inherited gates now pass on their first attempt at the same candidate head: testWatchdogThenReaperRetiresEveryConfirmedAbsentIdInUnit and watchdogAbsenceWriteFailureKeepsFollowerQueuedAcrossRestart. Candidate total is 3 passed, 0 failed, all first attempts. Four inherited gates, remaining capacity/retry experiments, whole-PR reviews and exact-head full suite remain pending.

The historical run10 result remains 3/3 fixture failures caused by the documented nullable fleet result before observation; full evidence is retained in the run10 branch. That question is now settled, not an outstanding operator decision.

Chain of findings

1. The approved inherited wake ordering correction is integrated. Identical corrected fixture failed main441a2fcb0/1, then passed3bd7f6b5 1/1 first attempt. Both follower replies are held until a real prepared two-slot wait; every old workload/capacity/count/FIFO/timeout assertion remains. A new frozen-clock assertion proves grant before the unchanged deadline. README1120–1122 already states immediate release/FIFO.
2. Legacy migration excludes permanently released shards only from reconstructed missing demand. Historical baseline0/3, original fixed1/1, fresh source630fb14d fixed1/1 first attempt across two actual service starts, retaining exact primary/request/unit identity. This correction and public regression are carried into current source; those legacy passes are historical after integration.
3. The corrected NEW retry regression failed3/3 automatic baseline attempts on630fb14d plus only an observer seam: the released follower persisted a fresh queued request and waited in admission rather than stopping. The same fixture passed1/1 first attempt against the guard and1/1 at current integrated source1efee264c5b99c401b1d7ab3f544b6675d5aa4ef. The guard checks saved and actual mutation state under the run disk lock, rejects fresh demand from a released follower, and preserves sibling requests. Existing deadline stop text, exact primary ownership, empty follower assignment/history and no extra provider/grant/demand counts are asserted.
4. Current source integrates main's per-worker provisioning-budget behavior. The first integrated compile exposed one missing retained-recovery worker argument; corrected before push. Main's e2eFailedDeletionFinishOrdersSiblingRequeue then passed1/1 first attempt at1efee264c5b99c401b1d7ab3f544b6675d5aa4ef. The single-branch clone fetch refspec excludes main; explicit git fetch origin main:refs/remotes/origin/main is required. Source is based on main d4d7d7185798086bb8fb24103caa84411bfeffc2; six integration conflicts were resolved preserving both main's worker/lock boundaries and the PR's retained replacement state.
5. NEW public watchdog baseline compiled and failed3/3 automatic attempts at1efee264c5b99c401b1d7ab3f544b6675d5aa4ef, outside the PR. It proves exact intact two-ID membership after startup, then writes alias absence, invokes the real watchdog, and verifies committed absence/live status/unchanged requests. The wholly absent unit remains alongside an independent positive unit. Production updates destroyed history/proof and older derived accounting but never retires the persisted admission unit. No watchdog fix was made because of the inherited contract question below.
6. Historical CI associated with630fb14d: https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37319511467 — all four artifacts read;3751 scenarios,3733 first-attempt passes,3743 final passes,8 failures,34 failed attempts,10 retry-only final passes. Its workflow checks out github.sha (PR merge ref), so these are associated-head artifacts, not exact local-source passes. Seven earlier failures persist; testReleasedUncreatedSlotLateDeletionKeepsPrimaryReservation also failed. All full traces/XML are preserved. Earlier3bd CI3742/3749 with7 failures and920 CI3731 first/3737 final of3745 with8 failures are historical. No manual CI rerun was requested.
7. Historical run9 admission-scoped inventory:710 selectors in708 files,33 helpers excluded;2 first-attempt passes at the then-current head (retry and sibling ordering),708 unexecuted. This is not the entire suite; run10 current source has about3962 root test scripts. The attempted unchanged unknown-owner selector exited75 at memory gate before launching a compile/test JVM: zero executed tests. Do not turn it into a baseline verdict or call remaining cases host-load flakes. Full suite, other ownership/restart/budget neighbors, retry-only classification and whole-PR independent reviews remain open.

Historical run10 findings

1. Read every failed attempt before implementation. Source-associated Actions run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37325503186 contains 3762 scenarios: 3744 first-attempt passes, 10 retry-only passes, 8 final failures. Four shards and dependent bld-build failed. All eight final-failure fixtures are byte-identical to the saved main comparison and current candidate; main c918 passed all eight first attempt in https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/37858390400 . Two cache retries have matching first-attempt failures on that main. Causal classification of five other retry-only mechanisms remains open after the three landed repairs below.
2. Candidate source is 9c0d2b90e77fc7a1212a6d7a890750ec61230bf0 on wip/L44-run10-watchdog-fix, based on main 09770c9c. The watchdog commits destruction/proof first, then retires fully absent units in a separate snapshot outside admissionLock. Unknown aliases remain intact; interrupted or closing callers do not start the separate operation. No inherited assertion changed. This is a FAILED candidate test result, not a verified fix.
3. Historical unchanged new-test baseline: 3/3 failures at the missing unit-removal assertion on 1efee264. Relevant watchdog/helper bytes match the rebased baseline. The final candidate invocation compiled successfully and failed 3/3 at the fixture problem above: 0/1 scenario passed. The other six named gates remain unexecuted on this candidate. Earlier bounded commands ended during compilation without executing a test; they are not failures or passes. No test/project timeout was changed and no CI rerun was requested.
4. The last verified remote main is 0be27701e9c3e236afddd7ec15bd2d080591e6b0. It advanced while the exact-head selector was queued/running; source was deliberately not changed underneath that command. Its new service hunks concern duration statistics and archive project identity; the scheduler also changes repeated memory-skip logging. The later completion-timestamp change also leaves the nullable fleet contract unchanged; both getFleetStatus implementations explicitly return null before any complete observation (their later reservation filtering differs). They do not implement durable admission retirement. Rebase after the fixture ruling before another build or PR update.
5. Remaining capacity hypothesis: a conclusive no-create path lowers the in-memory reservation but leaves a persisted unit which later deletion recount can restore. Current public reproduction is required before any fix. The late-refusal/unknown ownership cases overlap https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1286, which remains another actor's OPEN draft at d42d54ac with five failed checks. Its accounting changes do not contain this durable-unit cleanup. Do not touch its branch.
6. Separate narrow test/code reviews are preserved, including their correction after the actual result. The initial review missed the missing fleet observation; it is not a full approval. The repeat-confirmation wake question remains unproved, with a public experiment written down and no speculative patch. Whole-PR review and all remaining gates are open.
7. The owned PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1130 remains OPEN at 1efee264c5b99c401b1d7ab3f544b6675d5aa4ef with all five checks FAILURE. No PR branch push this invocation. All 41 inherited branches were rechecked and retained. Both new WIP branches preserve source, test, CI/main evidence and the required fixture decision. No remote suite, merge, enqueue, deploy, publication or handoff completion was performed.

Newer main repairs to preserve

Three retry-only fixtures now match main exactly, and all three passed first attempt in the saved main comparison. Their merge commits are verified ancestors of the candidate. Preserve them; do not duplicate their implementation. Candidate integration still needs verification.

• Duplicate finalize: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1298 merged as https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/5f67f3df8026fdfbee73c611001795142d855504 . The fixture holds resumed workers and waits for independent backfill before exact directory comparisons.
• Stalled creation: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1291 merged as https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/519bb78520bb3838889d3c8c95cfaf60c63efc1d . Manual time advances follow registered cleanup sleeps, preserving original assertions and bounds.
• Startup heap: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1300 merged as https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/7cb3a6330d3b097eda0fed0d334b2606011574bc . The repair includes bounded cold/restarted result reads as well as shared fixture encoding; earlier notes describing only fixture costs were incomplete. Original volumes, heap and time bounds remain. Five other retry-only mechanisms remain unresolved; see ci-classification.md and landed-retry-findings.md. Two cache retries have matching main failures. All eight final-failure fixtures are byte-identical across old PR, main comparison and candidate.

Relevant PRs / refs

• OPEN source PR: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1130 — live1efee264c5b99c401b1d7ab3f544b6675d5aa4ef; all four shards and bld-build FAILURE at final run11 verification. Reason-first description includes honest counts and unexecuted gates. No merge approval.
• Already merged: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1176 — live historical head c7bfda8df75bf95d6c702aeb865e2505b119207b, all5 checks green; merge https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/8d8e13a56a2bc99da7407008d5fd73e32cfb9ade is an ancestor of current main. No action needed.

All43 historical satellite heads below, including the two run10 branches, were reread with ls-remote around 05:08 UTC on 2026-10-09; their heads were unchanged. KEEP every satellite until content comparison; no PR/branch was closed. Proven legacy/retry/static scenarios are carried into source or retained as evidence; unexecuted drafts are preserved without ready claims. Evidence branches never belong in the implementation diff.

| Other actor admission draft | d42d54ace6dc4f2ff29d6c72e9f322a020fb698c | OPEN; five FAILURE checks; overlaps accounting. Do not touch. | | Merged repair | 08a516c0a87d64872d7a86ac5923b057f069dc50 | MERGED; five SUCCESS checks; ancestor of main. Preserve; no lane action. | | Merged repair | e7c5d51bf80bb3140ac73598fd883c11bdb2bdc9 | MERGED; five SUCCESS checks; ancestor of main. Preserve; no lane action. | | Merged repair | f45b98eb6b6e54130879648ed66fb8b218666428 | MERGED; five SUCCESS checks; ancestor of main. Preserve; no lane action. | | wip/L44-run10-watchdog-fix | 9c0d2b90e77fc7a1212a6d7a890750ec61230bf0 | Candidate and unchanged reproducer: compiled, then 0/1 scenario passed with 3/3 fixture failures. KEEP; not ready to merge. | | wip/L44-run10-evidence | f12e0495b44e0191ae8526f7733cdbc9ecf48c91 | Current CI classification, raw failed attempts, main comparison, overlap record and complete rebased source patch. KEEP; never merge evidence. |

| wip/L44-run11-watchdog-fix | cf95ec2c53d934525fa570fbfd5ac3a6de4330bb | Rebased candidate plus approved fixture setup; primary regression and two inherited gates PASS first attempt3/3 total. Four inherited gates and remaining fixes still pending; KEEP, not ready. | | wip/L44-run11-watchdog-baseline | 5e4de22d92e6d87ffdd3cdeb419ce77be64e6219 | Same approved fixture on unrepaired watchdog, based on main b865c4a8. Expected missing-unit failure3/3; KEEP baseline proof. |

| wip/L44-run11-evidence | 4e0d3e7cbb2178853bfb27978220445a3508ed2b | Expected baseline3/3 failures, candidate first-attempt3/3 passes, full logs/XML, separate reviews and remaining gates; KEEP, never merge evidence. |

Next steps

1. Resume from candidate cf95ec2c53d934525fa570fbfd5ac3a6de4330bb and evidence 4e0d3e7cbb2178853bfb27978220445a3508ed2b. Fetch/rebase current main before the next build, preserving the three main repairs. Run the four remaining inherited selectors separately: watchdogDynamicMissingRetirementReconstructsPersistedBoundary; testReleasedUncreatedSlotsWakeQueuedRun; testLegacyReleasedSlotsDoNotBecomeAdmissionDemand; testReleasedShardCannotReenterFreshAdmissionAfterBackoff. The approved setup, old-code3/3 failure proof, candidate primary first pass and two inherited first passes are complete. Do not reopen settled fixture rulings.
2. Execute or refute the saved repeated-confirmation wake experiment and the newly identified claim-association restart boundary. The latter concerns a valid created-droplet snapshot before its ID is copied into its matching claim-bound unit; it is a source-review concern, not a reproduced defect. See repeated-watchdog-deterministic-boundary.md and claim-association-boundary-review.md in run11 evidence. Never patch from inference alone.
3. Reproduce remaining capacity failures and five unresolved retry-only mechanisms against current source. Preserve the three landed main repairs above. Record overlapping ownership work for the supervisor; leave the other actor's accounting draft untouched and preserve every inherited assertion.
4. Complete separate whole-PR test and code reviews, one exact-head remote full suite if admitted, and all required green checks. Before a PR update, check OPEN/overlap, fetch/rebase current main, run required gates, push with an explicit lease and write an honest WHY-first description. Supervisor alone reviews and merges/enqueues/deploys/publishes/completes.

Operational knowledge

Every shell prepends /code/ws/bin. Fresh source checkouts stay inside the lane workspace. Local JVMs use jvm-slot, one selector per acquisition, with memory.current below 4.5 GiB before launch. Put the 1800-second execution bound INSIDE admission, so queue time does not consume it. No full local suite or large fatjar build. Project/test bounds are unchanged. Cancelled pre-admission commands executed zero tests; do not count them as failures or passes. Baseline has three expected missing-unit failures; candidate has three first-attempt scenario passes.

The selected workspace uses real source/support and one unchanged selector per admission. Source scripts use verified JVM coursier; global /usr/local/bin/coursier is wrong architecture. Explicitly fetch main if the clone fetchspec follows only the original PR ref. Check OPEN/overlap before PR push; use an explicit expected lease when no WIP tracking ref exists. No remote run was submitted this invocation. The source WIP and baseline WIP are pinned to exact tested heads; the original PR is unchanged.

All owned test commands finished naturally before finalization. No watcher is running. Handoff report/update may fail in transport; use at most three bounded attempts and preserve identical text on disk. This invocation releases its claim at CHECKPOINT; the next invocation must claim and read the live body. Never complete this handoff from the lane.

Run11 final live verification — 2026-10-09 around 05:23 UTC

• https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1130: OPEN at 1efee264c5b99c401b1d7ab3f544b6675d5aa4ef, five FAILURE checks. No source branch update by this lane.
• https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1286: OPEN draft at d42d54ace6dc4f2ff29d6c72e9f322a020fb698c, five FAILURE checks. Another actor owns it; untouched. Its no-create/terminal accounting overlaps the remaining capacity work.
• https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1350: OPEN at ff7739ca6285a2489a4e27636a77c8a13c0f5bc0, five SUCCESS checks. Another actor owns it; untouched. Its atomic watchdog state/spool writes concern an adjacent persistence boundary, not this admission-unit retirement.
• Required aggregate check is bld-build in current main rules. No CI rerun, source PR push, merge, enqueue, deployment, publication or handoff completion occurred.
• Separate test-comprehensiveness and adversarial review passes are saved in reviews-final.md. They support the narrow verified regression but do not approve the whole PR. Two unexecuted public experiments, four inherited gates, remaining capacity/retry classifications, whole-PR review and exact-head full validation are still required.

Run11 admission note

Opportunistic attempts to switch an unstarted command to a free second slot did not gain admission and discarded prior queue positions. Retain the normal queue entry; do not reuse those probe scripts. A prepared workspace may be reused only after its preceding selector has fully ended, with a fresh JVM-slot acquisition for each next selector. The candidate eventually compiled and passed after normal queued admission.