Finish ContainerNursery full-suite verification and final review

Written: 2026-10-27 07:08:16 UTC by fable-W1-1027.

RE-VERIFY: This is a write-time snapshot. Recheck PR state/heads and required checks, every branch via git ls-remote, and native FIRST-attempt histories in final suite XML. Four targeted passes are not a full-suite result. No merge or production action was performed by W1.

Design decisions in force

Owner requirements carried from the original handoff, 2026-10-26:

• "there must be no blocking machine-wide test lock - run full suites remotely and in parallel." (ASCII transcription of the original punctuation.)
• "Any first-attempt failure is a finding: reproduce and fix at its cause with a fail-first test."
• "No reflection, no mocks, no timeout increases, no iteration cuts, no weakening or deleting assertions. Use the injected Clock."
• "No notes files in repos."
• "Never merge or enqueue."

Current worker brief, 2026-10-27: "Use your best judgement, do not ask blocking questions." "Take no production action at all - except reading logs, metrics, and status endpoints, which you may do freely."

The worker may prepare PRs but may not merge, enqueue, approve, deploy, restart services or publish unmerged artifacts. The orchestrator owns all remote waits and landing. The brief permits targeted local fallback when BuildTest cannot execute. No full local suite or local test-lock queue was used.

The 2026-09-16 main-only PR rule remains in force: "pull requests only ever target main; never open a PR onto a feature branch -- fold side work into the PR branch directly." The 2026-10-27 orchestrator ruling limits this invocation to folding the existing three fixture commits into the original PR branch, checking four targeted scenarios once, posting evidence and resolving the side PR. The orchestrator owns L205 integration, all remote waits, review and landing.

Latest state and what remains

The original PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/743 is OPEN on main at e025b671833647634c873549f1825710216966ed, normally pushed and confirmed by git ls-remote. The other session advanced its branch to 927943b8d during W1's first run; rebase preserved all of that work and equivalent versions of the three corrections. W1 added only the missing receipt-exists assertion. The resulting combined head passed 4/4 FIRST attempts, four native attempts and zero retry histories; tracker observed 28 identities and zero survivors.

Side PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/757 is CLOSED, with its reason and replacement linked in the closing comment below. It was not merged. W1 did not merge, enqueue, approve or deploy anything. Worker outcome: NEEDS_MERGE_THEN_CONTINUE under the orchestrator ruling, because final full-suite evidence and review/landing remain. The claim remains fable-W1-1027 for the orchestrator as the worker brief requires.

The side PR was created against a feature branch, contrary to the owner's rule. Its useful corrections are preserved on the original branch, whose PR targets main. A concurrent push already supplied equivalent versions of all three fixes. Rebase preserved those incoming versions, including the complete raw-record error/list assertions, exact three-read count, isolated process enumeration, checked directory removal and Process wait. Only the side PR's explicit receipt-preservation assertion needed an additional commit. No default-branch or production mutation is included in this fold.

The original branch now includes the L205 checkpoint. Before merge: review the deliberate missing-owner expectation correction, obtain final combined-head full-suite FIRST-attempt XML and required green checks, and finish the orchestrator's final review. Then land the original PR and complete this existing handoff. No Maven publish or deploy is prepared or required by this verification effort.

The old compiler-timeout premise is stale on the current path: the recovery test already invokes the in-process JDK compiler with -proc:none and its unchanged ten-second bound, verified directly in its source. No historical external-compiler mechanism is claimed from passing current-code tests.

Relevant PRs / refs

Fresh validation and durable evidence

One-shot build-watchman 0.0.21 assessment at approximately 07:08 UTC: required bld-build PENDING; non-required bld-build-release PENDING; no buildtest run URL yet and progress not measurable in this poll. RESULT PENDING, exit 0. This is not a green CI verdict. The one-shot process exited; no CI watcher remains.

Fresh validation at e025b671833647634c873549f1825710216966ed: 4/4 FIRST attempts passed, 4 native attempts, zero retry histories, exit 0. The process tracker observed 28 identities and zero survivors. Whole-test durations were legacy 1892ms, pending signal 2412ms, saved-file sweep 9685ms and recovery 10446ms. These are targeted results, not a full-suite verdict.

The earlier selection at d5ff575dd also passed 4/4 FIRST attempts, 4 native attempts and zero histories; its tracker observed 23 identities and zero survivors. That result was superseded for integration when the original branch moved. The required rerun above is on the resulting combined head.

Command (after fresh fetch/rebase):

scripts/test.bash --local --test testNurseryPidFile_recoveryStopsGroupAfterRootExit --test testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof --test testPidFileConditionalRemovalPreservesNewOwner --test testNurseryPidFile_pendingSignalMustKeepOwnership --log <outside-repo.xml>

<details><summary>Resulting combined-head runner XML</summary>

<tests>
  <buildrule name="containernursery.buildMaven()" success="true" time="641468" output-files="199" output-bytes="123279403" executions="2"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testPidFileConditionalRemovalPreservesNewOwner.kts" name="containernursery.testPidFileConditionalRemovalPreservesNewOwner" success="true" time="1892"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_pendingSignalMustKeepOwnership.kts" name="containernursery.testNurseryPidFile_pendingSignalMustKeepOwnership" success="true" time="2412"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof.kts" name="containernursery.testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof" success="true" time="9685"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_recoveryStopsGroupAfterRootExit.kts" name="containernursery.testNurseryPidFile_recoveryStopsGroupAfterRootExit" success="true" time="10446"/>
</tests>

</details>

<details><summary>Earlier pre-concurrent-update runner XML</summary>

<tests>
  <buildrule name="containernursery.buildMaven()" success="true" time="588062" output-files="199" output-bytes="123275394" executions="2"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testPidFileConditionalRemovalPreservesNewOwner.kts" name="containernursery.testPidFileConditionalRemovalPreservesNewOwner" success="true" time="3997"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_pendingSignalMustKeepOwnership.kts" name="containernursery.testNurseryPidFile_pendingSignalMustKeepOwnership" success="true" time="4183"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof.kts" name="containernursery.testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof" success="true" time="11500"/>
  <test file="/code/ws/W1/workspace/ContainerNursery-fold-0635/tests/testNurseryPidFile_recoveryStopsGroupAfterRootExit.kts" name="containernursery.testNurseryPidFile_recoveryStopsGroupAfterRootExit" success="true" time="11605"/>
</tests>

</details>

The PR comment https://github.com/CodexCoder21Organization/ContainerNursery/pull/743#issuecomment-6435919482 contains the new XML and counts. The closed side PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/757 retains full historical baseline XML/stacks, corrected XML, legacy revert-control XML/stacks, real public process probe source/output and earlier independent reviews. Its closing explanation is https://github.com/CodexCoder21Organization/ContainerNursery/pull/757#issuecomment-6435920132 . No notes or test artifacts are committed in the source repository.

The earlier required CI findings remain accessible at https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40452850199 (bld-test-results artifact 12712451938). That is a baseline failure, not the current head's verdict. The orchestrator must inspect FIRST attempts rather than only final success counts on the combined head.

Next steps

1. Recheck original PR https://github.com/CodexCoder21Organization/ContainerNursery/pull/743 and all branch heads before editing. L205 integration and fixture reconciliation were completed in this invocation; preserve those changes.
2. Review the deliberate expectation change https://github.com/CodexCoder21Organization/ContainerNursery/commit/a4d6b3eb571df0ba713752168c45fe14ebab8925, and obtain the final combined-head required CI XML and full remote-suite FIRST-attempt result once BuildTest is healthy. Use scripts/test.bash --remote --test . --log <outside-repo.xml>. Do not broaden this worker's four-test gate into repeated runs or requeue unchanged red CI.
3. The orchestrator handles remote waits and landing. After the original PR is merged and final verification is complete, report DONE and complete this same handoff. No publish, pin bump or production deploy follows from W1's corrections.

Operational notes

• Successful claim and reports use fable-W1-1027. Leave this claim in place for the orchestrator, as the worker brief explicitly requires. Waiting for review, remote CI or landing is not a worker BLOCKED outcome.
• The remote preflight in this invocation failed before admission: Build service RPC health returned STARTING: recovering local state (elapsedMs=6073397), expected exactly OK. No run ID/XML or remote pass exists. The permitted targeted local fallback ran; no full local suite or test-lock queue was used.
• The initial handoff claim command failed while fetching bytecode (read 262102 of 2813637 bytes, followed by shared fallback deadline expiry). A bounded retry succeeded, as did canonical get and report. No persistent handoff service failure is asserted.
• report-challenge's helper automatically enqueues/merges. It was not invoked because this worker is forbidden to merge or enqueue. The exact failures and workaround are retained here; the active infrastructure owner retains responsibility for a durable fix.
• CLI 0.0.9 update/get/claims do not support --agent; mutation/report commands that accept it used the required name. Update replaces fields supplied; no mirror repository was edited.
• Nothing from this invocation is deployed or published outside GitHub. This follows from the executed command inventory: no deploy/restart/upload/publish commands were run. No production state was changed.
• Final checkout/worktree/stash/local-only-commit and owned-process audits: Clean checkout, no stashes or local-only branch commits, one worktree, exact remote-head match; both targeted commands and trackers finished, with zero tracked survivors. No long CI watcher or scheduled report timer was started. One-shot CI assessment finished PENDING, exit 0; no owned watcher remains..

Earlier evidence (snapshot from 06:25 UTC, superseded by latest state above)

1. Initial claims showed no active owner; W1 claimed only this handoff. A fresh checkout was created at workspace/ContainerNursery-own-0555, and root instructions, README, triage guidance, Testing Architecture, Engineering Philosophy and Code Review guidance were read. Earlier local W1 notes concerned another handoff and were excluded.
2. Original PR remains OPEN at 3f0292c012ee2acb6e0a26fec9a1d1ae5de93cf7. Required CI https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40452850199 has 1485 tests: 1482 FIRST passes / 3 FIRST failures, 1483 final passes / 2 final failures, 1490 native attempts. Release CI succeeded. XML artifact id 12712451938 was downloaded and inspected, including every attempt-history record. Recovery passed first attempt in 15131ms overall; this is not its compiler duration.
3. FIRST failures were testNurseryPidFile_tmpSweepReadFailureMustPreserveCleanupProof (all 3 attempts: expected AccessDeniedException but read completed), testPidFileConditionalRemovalPreservesNewOwner (all 3 attempts: explicit missing-registration-owner error), and testNurseryPidFile_pendingSignalMustKeepOwnership (first fails at child.isAlive, second passes). Full original stacks and final counts were collected by a read-only delegate; no CI rerun occurred.
4. A targeted remote baseline was rejected before admission: Build service RPC 'health' to url://buildtest/ returned invalid health value "STARTING: recovering local state (elapsedMs=3761389)." (runtime class java.lang.String); expected exactly "OK". No run ID or XML was created. The original requested remote full-suite result therefore remains absent; this is not a passing run. The allowed local fallback ran only four selected scenarios, one local test command at a time.
5. Unchanged local baseline: 3/4 FIRST passes; legacy removal fails all 3 attempts. Recovery PASS 8792ms, sweep PASS 9710ms, pending signal PASS 2789ms. Its audit observed 25 owned process identities and zero survivors. The local uid 1000 account really denies a chmod-000 read; current CI's root container does not, so the old error trigger does not exercise the intended fault everywhere.
6. https://github.com/CodexCoder21Organization/ContainerNursery/pull/757 contains only three test files, in three isolated commits. The sweep now faults only source() of its exact saved ownership file through the existing public FileSystem input. Real files, process tree, collection receipt, metadata, writes and cleanup remain real. It verifies the complete error, proves the sweep reached the fault, verifies the receipt survived and retains the original final-record-removal assertion.
7. The legacy conditional-removal test now asserts the exact existing missing-owner error and equality of the complete public ChildProcessInfo list. All original predecessor, replacement, count and unconditional-removal assertions remain. This intentionally corrects an old silent-success assumption to match the current unknown-ownership contract. The PR's first line says do not merge: awaiting approval of https://github.com/CodexCoder21Organization/ContainerNursery/commit/a4d6b3eb571df0ba713752168c45fe14ebab8925 so the expectation correction is reviewed explicitly.
8. The pending-signal fixture collects its own Process before its existing isAlive assertion, after worker join and exact signal-count verification. Rejected successor registration, its complete error, retained records and child-alive-while-held checks remain before release. JDK source shows Process.isAlive reads a flag published asynchronously by its reaper after OS exit. A narrow public SystemProcessChecker probe made 100 genuine cleanup calls with four concurrent callers: 3 observed cleanup=true / Java Process.isAlive=true / OS-executing=false, and every bounded waitFor collected its child. This is a measured observation gap, not a claimed >=50% old-test failure rate. No production cleanup behavior changed.
9. Legacy correction alone passed 1/1 FIRST attempt, zero histories (1259ms). Final requested targeted run at d5ff575dd16ac4aec625e469929c75ead7f856f0 passed 4/4 FIRST attempts, 4 native attempts, zero histories: recovery 8999ms, sweep 10432ms, legacy 2117ms, pending signal 2473ms. Final audit observed 15 identities and zero survivors. Fresh fetch/rebase changed neither tree nor tested head.
10. The requested legacy revert control restored exact original head 3f0292c0 temporarily. The old legacy test failed all 3 attempts again with the same missing-owner exception (1412/1435/1531ms). W1 restored its own branch afterward; no control mutation remains. No extra full-suite or repeated wider verification rounds were started after the targeted bar and required control.
11. Independent test-completeness and code-quality passes found no new defect or weaker assertion in the three-file diff. The same-pattern search noted two unchanged sibling scenarios, successorCannotClaimDuringSignal and successorJvmCannotClaimDuringSignal, as potential Java exit-observation candidates; their failures were not reproduced and no measured rate is claimed. They perform additional persisted cleanup before their worker returns. The report does not declare a repository-wide flake fix or a clean full suite. Existing initial-unknown-read and Clock neighbors already wait for their Process.
12. W1 discovered another session's fresh wip/hd22-L205-checkpoint branch, which continues original-PR production work. W1 did not overwrite, rebase, push or revert that branch or the original PR branch. The new small PR targets the original branch, avoiding a duplicate full production diff. Its current check rollup is empty; final integration and full CI belong to the orchestrator.