Handoff: finish ContainerNursery launch-pin ownership, run the full local suite, and open the PR

Updated 2026-10-27 11:25 UTC. Earlier lane results below remain historical; the continuation snapshot is authoritative for pending execution.

RE-VERIFY: This is a write-time snapshot. Recheck main and the three branch heads with git ls-remote, inspect out/findings.md and out/evidence.md on the main work branch, and query gh pr list --head for each branch before doing anything. No production state was queried or changed in this effort. Do not infer present service health from the historical outage diagnosis.

Continuation direction, 2026-10-27

The owner explicitly superseded the earlier scope boundary: “You are explicitly authorized to finish this whole fix, including the read-error adoption case and any closely related ownership paths the tests reveal. Do not stop at the scope boundary again.”

“Stop only for things that need the owner: a production change, a deploy, or weakening a test.”

Current acceptance: every new/changed test twice, then one full suite on exact final head; use local execution only when load is under 16, otherwise a GitHub runner. First-attempt passes only. Rebase current main, open the PR with outage motivation, evidence and counts; watch required CI with build-watchman 0.0.21 tracked by PID without --to-merged. Never merge, enqueue or deploy. Record PR URL in this handoff and out/final.md.

Current continuation source: https://github.com/CodexCoder21Organization/ContainerNursery/commit/52defb93952fd0a054f78218f60428cad8f7aa35 . It is preserved as the parent of the temporary runner harness on https://github.com/CodexCoder21Organization/ContainerNursery/tree/wip/cnPin-validation-2026-10-27 (remote head fc548eadd4f14dde9315595cee6dbc730bc9d598, verified ls-remote11:23UTC). The normal feature ref remains9840be3c5 until the source snapshot passes the requested final validation. No PR yet.

Continuation snapshot at11:25UTC

• Target-path ownership is required before adoption publication. Storage I/O, malformed target owner, missing record, invalid start identity and valid-but-different owner reject adoption and preserve the pin. Unrelated malformed JSON remains reported/skipped. Live publisher PID/start protects the pre-spawn interval; publisher exit permits exact-child discovery.
• Red batches: target unreadable/malformed/missing0/3 first-attempt passes; compiler-work/pre-spawn0/2; valid-PID/bad-start0/1. Initial fixed batch9/9 first-attempt passes. Independent code review found restoration before DetachedCreation registration. The corrected public shutdown test failed all3 attempts (first16,912ms); after moving restoration after cleanup ownership registration, it passed first (21,668ms).
• The later selected-nine batch was7/9 FIRST attempts. Primary timed out at the unchanged30-second limit on attempts1/2 (32,606/33,335ms including cleanup), then passed automatic attempt3 (28,387ms). That remains a FAILURE. Publisher reclamation also failed all3 because the test issued a second upload without triggering a second startup recovery. Its fixture now constructs a fresh real nursery after exact child exit before its second fence.
• Compiler count baseline failed expected0/actual1 in the timed primary body. Preparation now owns all Java compiles; the timed workload retains3 Nursery JVMs,1 child,2 replacements and3 upload fences. No timeout, iteration, input or assertion was reduced. The later primary limit failures remain evidence; no host-load explanation is claimed as their root cause.
• Coverage review added independent valid live wrong-owner rejection and final pin reclamation following blocked shutdown. Failed adoption intentionally preserves its durable pin; the latter assertion now starts another real nursery and observes deletion of the exact record through a forwarding native filesystem before asserting the pin is gone/source remains. It does not mistake an upload fence or a closed failed owner for a new recovery trigger.
• Current explicitly authorized GitHub-runner validation: https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40495665423 . Exact source52defb939; old JarReadPinStore baseline for the new mismatch scenario, then all13 discovered launch-pin scenarios twice, then full suite once. XML first-attempt gates reject retry-only passes. Tracked local watch PID3170921; bounded150m, terminal outcomes observed through gh run watch. Local load rose above16, so no further local builds began.
• Previous runner preparation https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/40495051616 was deliberately cancelled after static review corrected an unrun immediate-cleanup expectation. No verdict from that cancelled preparation is claimed. Harness is temporary and excluded from the final feature PR; no production, merge, enqueue or publish operation occurred.
• Test/code reviews and causal stacks are in out/test-review.md, out/code-review.md and out/continuation-evidence.md on source52defb939. Latest root findings are summarized here. Full suite/PR/CI remain pending. Main ce4d894a9 reverified11:23UTC; runner preflights require it remains an ancestor and reject an automatic rebase changing the tested source SHA.

Design decisions in force

User direction dated 2026-10-27, still binding:

• “ContainerNursery: a ContainerNursery restart deletes the launch path of a child it adopts — reproduce test-first, fix, open a PR (never merge, never deploy)”
• “Work on a branch from current main.”
• “Write an end-to-end test through the public API, with real processes and no mocks.”
• “Repeat the replacement boundary at least twice. Once the child is confirmed exited, assert the pin is reclaimed.”
• “Base the assertions on observed ownership and files, not elapsed-time thresholds.”
• “Persist launch-pin ownership tied to the child PID plus its process start identity, and transfer that ownership on handoff and adoption.”
• “Ordinary inspection-pin cleanup stays as it is.”
• “A pass that only happens on retry counts as a failure.”
• “Do not merge, enqueue or deploy. Note in the PR that deploying it needs the owner's approval.”
• “If a phase blows up in scope, push what you have, write down exactly where it stands, and stop rather than drifting.”
• “Never raise a timeout, cut iteration counts, weaken assertions, or skip tests.”
• “Run df -h / before every build. If less than 3 GB is free, stop and write that down in your findings instead of building.”
• “Use ~/bin/coursier, never /usr/local/bin/coursier.”
• “Before pushing, run the repository's full local test suite once on the exact head, and report pass/fail counts.”

This checkpoint uses the explicit scope-stop exception to preserve unfinished work, not to claim a passing push. The first red commit was also explicitly required before the fix/full suite. No timeout, iteration count or assertion was loosened.

Mission and why

The user supplied a detailed diagnosis of the 2026-10-27 buildtest outage. ContainerNursery restarted at 05:44:28 UTC while its buildtest child (started 04:55:59) survived and was adopted. The child's executable was a .jar-history-launch-UUID.pin hardlink. Startup recovery removed that pathname although the JVM retained an open descriptor. Kotlin's embedded compiler later reopened its own code-source pathname using ZipFile, could not find extension resources, and repeatedly failed initialization with retained application state and repeated full GC. This task repairs ContainerNursery's executable-path lifetime only; the separate Kotlin compiler initialization leak is not part of this change.

A fresh ContainerNursery clone began at main ce4d894a93e705689ac47e82fdf12940568438b6. Deployed source 61037c23013bc1c928f7d0b65da49d2ac21d65a0 is its ancestor, two commits behind. main was unchanged when rechecked around 10:00 UTC.

What was found and done

1. Read user diagnosis, fresh clone AGENTS.md and README, relevant RouterReplacement.md, all authoritative testing/philosophy guidance. Links: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md . Ownership invariants and step plan precede edits in out/findings.md.

2. Proven root cause: JarReadPinStore.begin persisted only a pin pathname on old code; a replacement store's activeRecords was empty, so recover deleted it. adoptExisting restored child process state without restoring the launch capture lease. The public regression uses three real nursery JVMs, two actual SIGTERM handoffs/adoptions of one real small JAR child, stable PID/start/path observations, and a child-side fresh ZipFile open per request. A synchronous public /upload/jar is a startup-history writer fence. Final ordinary shutdown confirms child exit before asserting the pin is gone.

3. Baseline regression failed on first attempt, 19.833 s: same child PID 3028728/start 2026-10-27T09:13:05.570Z returned fresh ZipFile NoSuchFileException after first replacement. Red test was the first pushed commit: https://github.com/CodexCoder21Organization/ContainerNursery/commit/7a775a864 . The runner also automatically failed attempts 2 and 3; no manual requeue is evidence. Raw logs are not the durable record; out/evidence.md carries first-attempt exception stacks and per-attempt counts.

4. Partial implementation saves launch_owner {pid,start_instant}; bind uses a retained capture outside the child's process lock; finish/recover preserve exact live owners; legacy path-only/null-owner records discover exact live -jar matches and save ownership; adoption restores the same pathname lease through the history writer before child publication. Detach releases the old manager lease while durable child ownership keeps the path. Linux zombies/exited states are not executing owners. Unknown/unreadable identity preserves and reports instead of deleting.

5. Independent review found (a) an unrelated malformed JSON record aborting adoption and disposing the healthy child, and (b) child spawned but process handle held before durable owner write, where null-owner recovery could delete its launch path. Both had actual first-attempt reds before their fixes. The latter public launcher test starts a real Process and pauses returning it. Ordinary inspection cleanup remains outside launch-name branches; ephemeral history uses the same store recovery and therefore gets the same owner guard. Activation-history files are separate hardlinks, not the actual launch pathname; no same-path deletion was established there.

6. The first fixed primary run FAILED at the unchanged default 30-second test method limit while third nursery startup was pending. Do not label that first fixed trial green. Moved Java fixture compilation to independent test-only build rules, following the repository buildOversizedStdoutFixture pattern; fixture namespaces/inputs remain private to each scenario. No runtime dependency or published artifact changed. Revised primary was re-run on original production code and still failed missing-pin in 9.938 s before accepting the setup correction.

7. Current partial patch targeted batch: 5 passed / 1 failed on FIRST attempts:

   • testJarLaunchPinSurvivesNurseryReplacement: PASS 21.663 s, both replacements and final reclaim.
   • testJarLaunchPinAdoptionSkipsUnrelatedInvalidRecord: PASS 14.841 s.
   • testJarLaunchPinAdoptsLegacyOwnership: PASS 15.967 s; also checks stale same PID with wrong start identity is reclaimed.
   • testJarLaunchPinRecoveryWithoutAdoption: PASS 20.680 s; live child readable, exact exit then later recovery removes pin, configured source bytes unchanged.
   • testJarLaunchPinRecoveryDuringProcessPublication: PASS 9.972 s.
   • testJarLaunchPinAdoptionRejectsUnreadableOwner: FAIL 14.011 s; exact valid owner-record read gets one injected IOException after startup fence, but adoption publishes child without restoring a lease. Attempts 2/3 also FAIL 16.151/14.482 s. The test uses ForwardingFileSystem over actual SYSTEM only to inject that precise read error; processes, nursery/admin and ownership are real.

8. Remaining proven bug: adoptProcess catches Exception around native reads, parsing, legacy migration writes, OS liveness and record claim; it reports/skips a valid target failure, returns null, and restoreLaunchPin treats null as successful no-op. Narrowing the catch blindly to pure parsing is not sufficient: a malformed owner field on the target record could still be skipped. A robust next design may identify requested child's exact -jar path first and require restoration for generated launch paths, while keeping direct-source older children supported. Write a failing malformed-target/missing-target scenario as needed before that change; preserve unrelated malformed-record behavior.

9. Further static review proposed a pre-spawn gap (pin created but ProcessBuilder.start not called, no child to discover). A deterministic public-launcher pre-start test was added. It PASSED current patch on its first attempt, 25.942 s. Thus this hypothesis is NOT established by reproduction. Discovery preserves ambiguous records if any relevant process is unreadable; that may explain this host's result but success output did not expose the reason. There is no old-main red for this test. No publisher metadata/state fix has been written. Do not add that lifecycle state without a host-independent contract reproducer. If a publisher state is eventually added, cover publisher-dead-after-spawn discovery with a separate process; the current same-JVM post-spawn test would otherwise exercise only publisher-live preservation.

10. Test cleanup now collects PID/start identities from starts.txt, PID records and descendants of owned nursery Process handles, before stopping handoff nurseries. It signals only matching identities. Three roots left by the earlier 30-second hard timeout were explicitly cleaned after process inventory confirmed no live owned JVM. All agents and targeted CLIs finished. No timers/watchers/goals were created.

11. Scope stop: review broadened from repeated adoption into target ownership failures and the unproven publisher state. User expressly directed stopping when a phase grows rather than drifting. All partial work is now pushed. Full local suite has NOT run, no PR/CI started, and no ready/final fix claimed. Seven tests plus five own test-fixture build rules are present. The current main checkpoint is an unfinished implementation.

Relevant PRs / refs

Every push was verified using matching git rev-parse HEAD and git ls-remote. No stash/local-only commits remain. Code checkpoint b5415c7979595037e8ef44b82062061b8f1fb409 is followed only by status-document commit 976271167d982c55006b0dd8aae77dc717a45b35. No PR was closed.

Deployed/published but unmerged: NONE from this effort. No Maven release, upload, production signal/restart/route write, merge or enqueue occurred. Production health/deployed bytes were not rechecked; the supplied deployed commit is historical diagnosis context.

Next steps

1. Re-verify current state and branches first. Clone fresh under the new lane's ws/ and read AGENTS.md/README. Keep all user prohibitions.
2. Inspect valid-target I/O red and broad adoption catch. Fix only after preserving the same failing reproducer. Cover target malformed metadata/missing record test-first if the design can otherwise publish a generated launch child without a lease. Keep the unrelated malformed-record green case.
3. Treat pre-spawn publisher issue as unproven, not a required speculative fix. If pursuing it, force exact OS/discovery conditions deterministically through public API; do not tune concurrency/time to this host. Preserve first red/green evidence per actual code revision.
4. Review local cleanup/exit/ownership invariants, then run only changed targeted scenarios. Do not accept runner retry-only success. The original first fixed attempt failed its method limit and stays recorded; the prepared fixture's first current-patch primary attempt is green.
5. Complete independent test and code review, including existing ordinary inspection restart/shutdown cleanup tests which have NOT yet run on this patch. Avoid tests weakening/timeout changes.
6. Rebase latest main before any build. Run the full LOCAL suite once on exact final code head, while no other full suite runs machine-wide. Record first-attempt pass/fail counts. There were 1,325 original test scripts before this task. Do not use CI as the debugging step.
7. Once green, open a PR, never merge/enqueue/deploy. Its opening explains the 2026-10-27 buildtest outage, missing executable pathname after adoption, fresh ZipFile failure and repeated compiler initialization/memory growth; then explain ownership change and honest test evidence. State deployment needs the owner's approval. User's stop condition is full suite clean + PR open with CI started; do not run extra verification rounds.
8. Update out/final.md with actual PR URL, red/green evidence and full-suite counts. It currently honestly states no PR and full suite not run.

Commands and operating details

Use pinned CLI 0.0.124 and an ABSOLUTE task-owned cache path:

git fetch origin
git rebase origin/main
df -h /
~/bin/coursier launch kompile.cli:kompile-cli:0.0.124 -r https://kotlin.directory -M kompile.cli.CliKt -- --local --cache-location /absolute/lane/out/buildcache --test testJarLaunchPinAdoptionRejectsUnreadableOwner --log /absolute/lane/out/target.xml --result-file /absolute/lane/out/target-result.json

For a few scenarios repeat --test; for full suite use --test . . CLI internally retries failures up to three times; top-level XML time/verdict is first attempt when failure remains, and attempt-history records each failure. Successful scenarios have no attempt history. Do not manually requeue unchanged assertions. Source-only fixture preparation can take minutes; local compile/cache output may be buffered until later. No --timeout increase is allowed.

Relative cache path caused an initial premain failure BEFORE test entry: IllegalArgumentException: Native SHA-256 agent argument must be an absolute classifier JAR path, but was ../../out/buildcache/prepared-test-classpath.../conscrypt-openjdk-2.6.3-linux-x86_64.jar. Using absolute cache path fixed the invocation; that harness failure is excluded from causal red evidence. report-challenge was read, but its automatic PR/enqueue/merge conflicts with the user's explicit prohibition, so no challenge CLI was called.

Durable summaries: out/findings.md, out/final.md, out/evidence.md, out/code-review.md, out/test-review.md on the main work branch. Raw XML/build logs, cache/JAR outputs and raw thread dumps are deliberately not committed. Evidence summary retains full first-attempt exception stacks, counts and source facts; no raw profiler/environment data was uploaded. Task-owned caches/build output are being removed at handoff completion.

rg is unavailable; use grep/find. Only ~/bin/coursier is the correct architecture. Repository-local Git author is codexcoder21 with its public GitHub noreply address; no global identity was changed. Local current directory was /tmp/claude-1000/-code/f67b4d83-4ebc-4507-bde7-f1a83d72de59/scratchpad/cnPin, but pickup must use remote branches rather than rely on that disposable path.

No user approval question is pending. This is an unfinished engineering checkpoint under the explicit scope-stop direction.