Obtain the final required check and review the superseded-kill waiter PR

Written 2026-09-27 03:26 UTC. RE-VERIFY: GitHub heads/checks and buildtest outcomes are snapshots. Only the orchestrator merges or enqueues. No production changes authorized.

Mission and remaining work

The implementation and strengthened public-API tests are now locally verified and pushed to https://github.com/CodexCoder21Organization/ContainerNursery/pull/629, head 69318c049acfa2c0820af225a3e16528dbf50b54, base main. Required run https://buildtest.kotlin.build/run?id=d1ad8aa9 FAILED before tests during allocation: DropletServiceCallStalledException: Droplet service call listDroplets stalled for 30000ms during findDropletsByNamePrefix. It has dropletId=0 and 0 passed / 0 failed tests. Informational remote run https://buildtest.kotlin.build/run?id=7e7cc611 also failed before tests because it could not reconnect to digitalocean-droplets. Restore that shared-service path through its existing owner, then obtain a fresh required result and final orchestrator review. Do not blindly rerun or change this tested waiter fix to hide an allocation failure. Do not treat the original-head green check as validation of this head.

The separate Gradle restart-adoption startup-hook defect is assigned to W33 by the orchestrator, with W25 source at https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/adoption-hook-startup-W25 . Do not duplicate that work here.

What was found and done

Original symptom: testGetOrCreateContainer_slowStartupDeadlineReplacesTimedOutGeneration failed three attempts in https://buildtest.kotlin.build/run?id=69ef907a and in https://buildtest.kotlin.build/run?id=85e62fda / https://buildtest.kotlin.build/run?id=85c85245 while unrelated changes were being validated. A replacement caller was refused because cleanup appeared unconfirmed.

Mechanism: start settlement advances requiredKillEpoch after the first quarantine kill begins. Main completes the old attempt false, then starts the required successor. A waiting caller consumes that intermediate false as terminal failure. The fix leaves a normally returned obsolete attempt pending, releases its cleanup worker capacity, starts or joins the successor, and forwards the successor's terminal outcome. Failure remains false; caller cancellation does not cancel the independent cleanup deferred.

W29 found the implementation already present in the original PR, strengthened both regressions to hold the successor open and assert the caller remained pending, and demonstrated 0/2 passing on main at A superseded attempt must not finish the caller before its successor. Those archived traces were retrieved and checked by W39. W39 ran the exact seven-test command (7 passed, 0 failed), then 52 quarantine/kill/settlement/shutdown/management neighbors (52 passed, 0 failed), for 53 distinct tests and 59 successful executions. Both source review passes found no remaining concrete issue at final head 69318c049acfa2c0820af225a3e16528dbf50b54.

W39 fetched and rebased immediately before updating the PR; main remained 40f3f7501a153b13ac1a763464e4a0f72515253c and source HEAD was unchanged. GraphQL verified OPEN, unqueued, original head 6ce62319723602fe827acdb9f78fd63027855a20. An explicit old-SHA force-with-lease updated only that head to the validated source. The why-first description includes baseline evidence, current counts, and the separate W33 issue. No new production implementation change was needed. Other recent ContainerNursery actors are on distinct branches, and their work was preserved.

Branches and unmerged state

No Maven version changed. Neither W29 nor W39 published or deployed, and this handoff requires neither. All product changes are in the open PR, not main. Source tree is clean; no stashes or unpushed source changes.

Validation commands and evidence

Preferred remote submission: scripts/test.bash --remote --test testQuarantineKill_waiterFollowsSupersededAttempt --test testQuarantineKill_waiterGetsFailedSuccessorOutcome --test testGetOrCreateContainer_slowStartupDeadlineReplacesTimedOutGeneration --test testLateStartRequiresKillAfterSettlement --test testLateFailedStartRequiresKillAfterSettlement --test testShutdown_reportsUnconfirmedQuarantineSurvivors --test testGetOrCreateContainer_startCancellationStillCleansContainer --log ../../seven-tests.log.

That run https://buildtest.kotlin.build/run?id=4380b0f1 failed before tests during allocation: Could not verify droplet cleanup before requeueing runId=4380b0f1 during allocation failure: SandboxException: Cannot invoke method on closed instance proxy: dropletserviceserver/DropletImpl.getName()Ljava/lang/String;. The run was failed without re-entering admission. This is zero test executions, not a failing regression. W39 did not blindly requeue it.

Local fallback: same seven selectors, replace --remote with --local, wrap the command in /tmp/claude-1000/-code/1f0f7d17-dc24-4d3a-a8ff-48753bfab06a/scratchpad/cx/jvm-slot.sh, and write --log ../../seven-local-tests.log. Result 7/7 pass. Neighbor exact command and selectors are preserved in handoff-artifacts/W39/neighbor-command.txt and neighbor-selectors.txt on the W39 evidence branch; result 52/52 pass in neighbors-tests.log. No whole local suite was run on the shared host.

Build-watchman must use --interval-seconds 300 to preserve the shared GitHub budget. Never use --to-merged in this worker role. gh pr edit on the installed old gh fails through classic project fields; a structured REST PATCH with a JSON body successfully updates descriptions. Handoff CLI update returned Stream closed once and succeeded on the second attempt. report-challenge automatically enqueues/merges and was not invoked under the assignment's absolute prohibition; the orchestrator may record the allocation/proxy issue centrally.

Invariants and review scope

• Only a normal kill at the final settlement epoch confirms disposal.
• A normally returned superseded attempt remains pending for its original waiters; they receive the successor outcome within their existing wait bound.
• Terminal failed cleanup retains quarantine and refuses replacement with the full diagnostic.
• Old worker capacity is released before successor dispatch; forwarding callbacks consume no waiting worker.
• Caller cancellation leaves cleanup ownership intact.

W39 review A: both tests distinguish premature false from terminal false; executor barrier and undispatched Unconfined caller force the ordering; gates and owned executor close in finally. Success checks identity, creations and kills; failure checks exact message/no replacement. Review B: lock/epoch transitions, successor completion before callback registration, a concurrent actor starting the successor, dispatch failure and cancellation all preserve the contract. No timeout, iteration, interface, dependency or retry-policy changes. README accurately states the user-visible contract. Detailed separate passes are in handoff-artifacts/W39/review.md.

Diff for final review

Four files: ContainerNursery.kt forwards superseded normal-kill outcomes (~35 lines); two test scripts force successful/failed successor ordering; README adds four contract lines. Principal risk is lifecycle callback ordering, covered by the fail-first pair and 52 neighbors. Required CI on this head failed before tests due to allocation-service failure. This is a CHECKPOINT, not a green final-review recommendation.

Final W39 state

CHECKPOINT. GraphQL at 03:26 UTC: OPEN/BLOCKED, main-targeted, no queue entry, unchanged validated head. Gradle SUCCESS at https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/36290675800/job/108540015848 and bld-build-release SUCCESS; both remote checks FAILURE before tests. No watcher or local test process remains. Claim held for orchestrator. Product source and full proof artifacts are pushed. The only remaining required engineering gate is a green remote run after the shared allocation service path is restored; do not merge before that.

Final W39 report: https://github.com/CodexCoder21Organization/ContainerNursery/blob/7795225eff9ec396a3d6f6bda9cfa1719cb79cbb/handoff-artifacts/W39/report.md . All source and evidence heads reverified after push.