Handoff: Finish remote CI verification for ContainerNursery descendant cleanup

Written 2026-10-25 UTC.

RE-VERIFY: This is a write-time snapshot. Recheck both PRs with gh pr view <full-PR-URL> --json state,headRefOid,statusCheckRollup,mergedAt, read the current failed remote check output, and verify BuildTest publication health before requesting another upload. Never infer current service state from the older disk-full message alone.

Design decisions in force

User instructions from 2026-10-25:

• "Do NOT merge or enqueue."
• "Do the work yourself; do not start other model sessions."
• "Any wait loop needs a hard deadline and must be killed before you finish."
• "Do not commit notes or report files into the repository."
• "Kill only processes you can prove came from your own runs (check cwd under your checkout), by PID."
• "Never weaken assertions, raise timeouts or cut iterations. No reflection or mocks."
• "If production leaks children, that is a real product bug and the fix belongs in main code, with the test asserting the absence of surviving descendants."

No merge, queue operation, production deployment, service restart, production deletion or unchanged CI retry was performed.

Remaining work

The code and requested local verification are complete. Required bld-build has finished SUCCESS, but full CI verification still has the legacy BuildTest submission refusal plus one first-attempt HTTPS scenario failure (details below). Finish CI verification for https://github.com/CodexCoder21Organization/ContainerNursery/pull/743, keeping it open and outside the merge queue. The old non-required kotlin.build (remote) check cannot execute this code because BuildTest refuses upload-session creation. Preserve that separate infrastructure result; do not describe it as a test failure or bypass it. Required bld-build state is recorded in the fresh status section below.

The user was asked asynchronously whether this lane should finish at a green required check with the separate BuildTest blocker reported, or expand into BuildTest recovery. No answer had arrived when this body was prepared; the lane remained scoped to ContainerNursery.

What was found and done

1. Fresh clone of main into workspace/ContainerNursery; read README.md, AGENTS.md, Testing Architecture, Engineering Philosophy and the linked cleanup-ownership PR. Branch fix/hd22-L80-no-orphan-sleep-helpers.
2. Before the first candidate, no running process had this checkout as cwd. Three unchanged tests passed but left running helpers with PPID 1 and that cwd:
   • testZombieCleanup_systemProcessCheckerKillProcessVerifiesDeath: PIDs 86820 and 86883, sleep 600.
   • testLogFileTailForwarder_reopensAfterCopyTruncate: PID 87540, sleep 60.
   • testWatchdogRequiresConsecutiveProbeFailures: PID 94459, /bin/sleep 1. Other thirteen selected sleep candidates left no orphaned helpers. The four baseline leftover PIDs were stopped only after comparing cwd and kernel process start ticks with the recorded snapshot.
3. The production issue was confirmed. The checker and JAR cleanup signaled and verified only the root PID; a shell or JVM could exit while its helpers continued under PID 1. New public-API regressions force descendants to exist before disposal, retain their exact handles, and observe real OS execution state. All corresponding original paths failed on every native baseline attempt (three attempts each). The checker body exercises ten root/child/grandchild trees and an unrelated sibling. Real JAR scenarios cover kill, graceful shutdown, and PID-only adoption, with a child shell and two sleep grandchildren. A fifth scenario covers interrupted checker cleanup.
4. Main code now captures descendant handles before signaling a parent, stops/verifies the captured processes, and preserves caller interruption. JAR containers retain pending descendant ownership and liveness until verified cleanup, before releasing port/temp/launch resources. Three fixtures now use a direct sleeper, or exec after setting ignored TERM, with bounded collection.
5. A local neighbor caught an initial compatibility error: custom real-process launcher wrappers implement pid() but not toHandle(). The implementation was corrected to use the existing PID contract. All three wrappers were enumerated before resuming. Test and code reviews were performed as separate self-review passes; no other model session was started.
6. Final repeated verification: ten rounds of the five new regressions, 50/50 first-attempt passes, plus 11 neighboring checks in round zero, for 61/61 passes and 61 native attempts. Every post-round process snapshot was empty. The existing launcher compatibility check passed separately.
7. The single requested full suite ran as flock /tmp/claude-501/hd22-suite.lock scripts/test.bash --local --test . --log <outside-repository XML>, after fetch/rebase. It exited 0 at 08:06:56 UTC: 1,085/1,085 first-attempt passes, zero failures, 1,085 native attempts, zero retry histories. Before and immediate after snapshots were both empty. Changed-file hashes were frozen before the run and unchanged afterward. The tested source is commit 58a74f696c0d6bbdcfd47f64295516d3f315f233; the pre-PR rebase did not change its tree.
8. The full suite emitted two previously recorded TerminalMemoryWarningEffect warnings about output following the terminal memory record. They did not fail tests. Their recurrence, exact records, first stack and missing scenario attribution were appended to the existing challenge in https://github.com/CodexCoder21Organization/PlanRepository/pull/11236. That documentation PR was green in the watch-build assessment and remains unmerged.

Evidence that survives the workspace

https://filedrop.wasmserver.com/drop/2d4d630e75

Raw bundle: https://filedrop.wasmserver.com/raw/2d4d630e75/l80-verification.zip

The bundle expires after one year. It contains 39 selected XML/snapshot/infrastructure evidence files plus verification.json with hashes, source commit and totals. It includes the full suite XML, its before/after snapshots, all three confirmed baseline orphan snapshots, five failing production regression XMLs, all ten final repeated-run XMLs/after snapshots, the compatibility result, full failed remote-check output and the read-only host filesystem observations. No credentials, private keys, build binaries or dependency caches were uploaded. Code and tests are on the remote Git branches below; notes and reports were not committed to ContainerNursery.

Relevant PRs and remote branches

Both worktrees were clean, with no stashes, and their exact remote head SHAs were verified using git ls-remote. PlanRepository was a shallow single-branch clone: its remote-tracking refspec initially omitted the new branch, so git log --not --remotes listed the pushed commit even though ls-remote confirmed the exact SHA. That was metadata, not unpublished work.

Deployed or published but not merged

None. This lane launched only local test children and read-only remote HardwareControlFabric ping and df commands. The release workflow only builds/verifies on PR events; its draft-release publication steps require a main push. No production route, JAR, process supervisor, Maven release or application was changed.

Separate BuildTest publication blocker

The remote check https://github.com/CodexCoder21Organization/ContainerNursery/check-runs/121114835568 failed before any code upload: zero created sessions and zero sent chunks. The summary says createUploadSession failed after five internal attempts because:

RPC request 'createUploadSessionStructured' failed: INTERNAL_ERROR - RPC error: INTERNAL_ERROR - Execution-duration publication failed for '/root/buildtest-data' while publishing 2 queued events for runs [b95d4f17]: java.io.IOException: No space left on device. New publications are refused.

This is already recorded at:

• https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-25-0513-buildtest-host-data-disk-full-kotlin-build-remote-fails.md
• https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-24-0317-kotlin-build-remote-is-red-on-every-urlresolver-main-merge.md

Read-only HardwareControlFabric probes at about 08:20 UTC succeeded. /dev/vda1 currently showed 310G total, 295G used, 16G free, 96% used; inode use was 4%, with 39,758,995 free. Thus current free blocks/inodes do not explain the complete retained refusal. The unchanged b95d4f17 error supports a retained publisher failure as a hypothesis; this lane did not inspect the implementation or prove that mechanism. No data was deleted and no service was restarted.

Existing broader BuildTest recovery work is tracked in the Handoff service, including url://handoff/handoffs/hf-2026-10-22-land-and-deploy-the-tinysite-memory-leak-fix-and-the-buildtest-disk-full-recovery-fix-once-buildtest-accepts-builds-again. Do not assume that older record's deployment state is current; re-verify it and coordinate with its owner before changing shared production state.

Next steps

1. Re-verify both PRs and their exact heads. The source change and local suite are already complete; do not repeat the full local suite unchanged.
2. Required Actions is already green; read its saved first-attempt evidence before treating CI as clean. The one first-attempt HTTPS failure remains unexplained, on the initial request before replacement/stop. Establish a consistent local reproducer before any proposed fix; keep its existing timeout and assertions. Never rerun unchanged CI to investigate it.
3. Obtain the BuildTest owner's recovery of upload/publication admission. Preserve journal evidence, diagnose the actual retained failure, and fix it at its owning layer. Do not delete run data or restart/deploy services merely to clear this check.
4. Only after verified recovery, request one fresh legacy remote check and confirm that a real upload/session/run is created. An unchanged retry while admission still refuses is not useful.
5. Refresh the PR description/status with the actual outcomes. Keep both PRs open: no merge, enqueue or production deploy is authorized by this task.

Reusable operational knowledge

• A successful test exit is not proof that helpers were collected. Snapshot PID, PPID, cwd and args outside the test runner, and retain exact descendant identities before the parent can disappear.
• Do not add a new Process-wrapper requirement accidentally. Existing launchers can wrap real processes and forward pid() without implementing toHandle().
• Successful local XMLs omit attempt histories; count one native attempt for such a test. Failed or retried cases use root-level attempt-history nodes. Do not report only final pass totals.
• The required-only watch-build view did not report the legacy failed check; inspect the full check rollup when classifying CI, and disclose the infrastructure refusal separately.

Monitoring and tooling observations

Both required-check watchers have exited. The original watcher reached its own 30-minute idle limit while Actions was still executing; the resumed watcher used the default 120-minute idle window, kept a hard outer deadline, and exited GREEN after required completion. No CI job was cancelled or rerun.

If gh pr edit --body-file fails with the old GraphQL repository.pullRequest.projectCards deprecation error, use the REST pulls PATCH endpoint with a JSON body file. This was used without changing source or CI.

Required Actions result and newly observed first-attempt failure

The required bld-build completed SUCCESS at about 08:55 UTC, https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40175473284 . Its uploaded bld-test-results/test_log_file.xml was downloaded and parsed rather than trusting only the green check. It contains 1,085 final passes, 1,084 first-attempt passes, one first-attempt failure, and 1,086 native attempts. All five new descendant regressions passed on their first attempts. No workflow or manual test rerun was requested.

testInheritedHttpsReplacementChecksSuccessorConfiguration failed first with java.net.SocketTimeoutException: Read timed out in InheritedHttpsListenerScenarios.readLine:861, via openTls:838, request:789/786/784, replacementRestore:554, main:126. The assertion surfaced from the Kotlin test at line28. The complete 83-line first-attempt stack is preserved in the companion evidence bundle below. The runner automatically repeated the scenario once and it passed.

The failing call is the initial predecessor request, immediately after waitReady, before any replacement or stop call. waitReady checks only the front HTTPS listener log; backend creation is triggered by this request. Logs show front listener started 08:32:17.416, route creation 08:32:18.537, container start 08:32:19.806, app registered 08:32:22.323. openTls sets the existing five-second client read timeout. These observations suggest startup/readiness ordering is worth examining, but do not prove why the response missed its bound. No timeout, assertion, source or fixture was changed on that theory.

Ten targeted bounded local repetitions of exactly this public scenario all passed on their first native attempts, and all after-process snapshots were empty. Results are preserved as inherited-N.xml plus before/after process snapshots. Do not treat passing repetitions as proof of a cause or a fix. The required local full-suite evidence remains 1,085/1,085 first-attempt passes on the unchanged final source commit.

Final verification snapshot at 2026-10-25 09:07 UTC

The full CI XML, complete initial HTTPS stack and console output, ten local first-attempt XMLs with before/after snapshots, and final empty process snapshot are preserved remotely via FiledropCli: https://filedrop.wasmserver.com/drop/f435dbff76 ; raw https://filedrop.wasmserver.com/raw/f435dbff76/l80-ci-followup.zip . One-year expiry. The companion zip includes hashes and the tested source commit in verification.json. These files supplement, and do not replace, the original local-suite/baseline evidence bundle above.

All ten local follow-ups passed, no cause or fix is claimed for the first-attempt HTTPS CI failure. Both required-check watchers, the bounded repetition driver, and evidence uploads have exited. No source changes were made after the exact locally full-suite-tested head. All work is pushed; no partial source exists only in scratch. ContainerNursery PR remains OPEN/unmerged at 58a74f696c0d6bbdcfd47f64295516d3f315f233; required bld-build and release checks SUCCESS; legacy kotlin.build(remote) FAILURE before upload; analytics kompile-remote-build still IN_PROGRESS. PlanRepository PR remains OPEN/unmerged at 8fcc89029892810196f1bd7a0d23fad084b1b63e, both checks SUCCESS.

Requested brief audit

• Fresh main clone, instructions, named branch: done.
• Identify leaking candidates individually with process snapshots: done, three confirmed; thirteen other direct-sleep candidates did not leak.
• Distinguish production leak through public real-process/JAR APIs: done, five fail-first scenarios, baseline XML preserved.
• Fix main code and fixtures without weaker assertions/timeouts/iterations: done; five scenarios pass and existing ownership/launcher neighbors pass.
• One locked full local suite and exact first-attempt counts, no new checkout orphans: done, 1085/1085 first attempts, snapshots empty.
• Milestone pushes, latest-main rebase, open PR with exact motivation: done; no merge/enqueue.
• Complete CI verification: required check green but full CI remains blocked by legacy upload admission; the separately observed first-attempt HTTPS failure also remains unexplained. Do not call all checks or all CI first attempts clean.