Handoff: Finish final-head verification of JAR-history review fixes

Written 2026-10-19 20:58 UTC. RE-VERIFY: all state below is a write-time snapshot. Re-check https://github.com/CodexCoder21Organization/ContainerNursery/pull/668 with gh pr view <URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, verify branch heads with git ls-remote, and use runner XML for counts before acting.

Final lane terminal state — 2026-10-19 21:01 UTC

The sole mandatory full-local-suite waiter reached its 38-minute cap and exited 124 without ever acquiring /tmp/cn-fullsuite.lock. There is no starting-head file or XML result; the full gate remains incomplete. The final CI watcher also reached its six-minute cap and is stopped. A fresh 21:01 UTC PR read confirms OPEN/BLOCKED, head efe03d374685ec9adc799b26f4e4d94a7b6bc864, required bld-build IN_PROGRESS at https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39323006002 ; the release-package check passed. All lane-owned test/CLI/watch processes exited; no other lane's lock owner was touched. The source checkout is clean and every code commit is pushed. Root is removing the ignored jars launcher directory and private 1.4GB build cache before the final response. The PR body was updated via the requested REST call with these counts and limits. No merge/queue/publication/deployment occurred.

The service accepted the final body update and a subsequent read verified the new title, summary, blocked reason and body. The previous ambiguous update is historical; this service record is again the current handoff. The draft Git fallback remains historical/unmerged at https://github.com/CodexCoder21Organization/PlanRepository/pull/11082 .

Design decisions in force

The owner asked, verbatim: "The ContainerNursery service should keep track of all recent jars deployed for each service, along with the date/time it was uploaded and the date/time that each of the jars was the jar actively serving the site and make it easy to roll to a different jar (eg. roll back a bad change), etc." and "This may involve updating the CLI/api too. Merge your changes, ensure test coverage is comprehensive."

The current lane brief (2026-10-19) supersedes merge intent for this lane: "Forbidden: merging, enqueueing, publishing, deploying; never dequeue anything (the supervisor may enqueue)." It also requires: "Review claims are unverified until a test shows them: confirm each finding with a failing test before you change code, and if a claim turns out wrong, say so with your evidence rather than changing code." Its rules are "no raised timeouts, no lowered iteration counts, no retries or sleeps to pass, no weakened assertions, no reflection, nothing made public for tests." Full local verification must use flock /tmp/cn-fullsuite.lock scripts/test.bash --local --test .; never bypass that lock. The current lane has a 100-minute budget ending about 21:06 UTC.

Contract I1–I8 and A1–A3 remain binding. Public upload success/restart/retention behavior is preserved after installation; history failure is additive and surfaced. Content captured for activation belongs to the actual started generation, with pre-start ownership and failed-start cleanup. No-data-directory item 4 is report-only for this review. Optional items 6–8 require a small test-proven fix or a documented reason to leave them.

Existing Handoff service record

The linked Git file is a fallback for this same service record hf-2026-10-19-finish-verification-of-jar-history-server-review-fixes; it is not a request to create another service entry. At about 20:48 UTC, an update RPC returned AmbiguousRpcRequestException after the write, and the reconnect/read attempt failed. Whether the service saved that update is unverified. When url://handoff/ is reachable, read this same ID and reconcile it with this file before updating it again.

Central service report and claim release — 2026-10-19 20:58 UTC

The Handoff service accepted the full BLOCKED report for hf-2026-10-19-finish-verification-of-jar-history-server-review-fixes and then released the fix-668e claim. The successful report and release are separate from the earlier ambiguous body-update RPC; whether that earlier write was saved remains unverified. The current service body is being read before any further body replacement.

At this timestamp, the required release-package check had passed, the required bld-build check was pending, and the full local suite had not acquired its lock or produced XML. Re-check the service and PR before acting.

Mission and current state

Finish final-head verification of B1–B4 and the confirmed small optional fixes, then update the open PR and complete the required local suite. The latest backup-pushed candidate is efe03d374685ec9adc799b26f4e4d94a7b6bc864 on checkpoint/jar-history-shutdown-review; its production source matches the previously tested a0b5b07f8dfa6536387d2fc9a7459beb30f70da6, with a corrected complete-snapshot test fixture added after that test failed for the wrong reason. The ContainerNursery PR branch has since been pushed to efe03d374685ec9adc799b26f4e4d94a7b6bc864; its required final build is being watched and has no result in this snapshot. The 13-test new/changed group passed 13/13 on a0b5, and the corrected complete-snapshot test passed 1/1 on efe03d. The broader 69-test group is not green and the full local suite remains incomplete.

The earlier required bld-build run on PR head c1282af failed with 701/703 tests; bld-build-release passed. The failed run is https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39316847356. Its failures were testJarHistoryPersistsAtomically (the stale 500 response expectation has since been corrected) and testJarHistoryReadKeepsCompleteSnapshotDuringHashAndPersistence (the old fixture did not force the intended hash). The corrected test fixture passed 1/1 on efe03d. Those results do not replace the current required build result; the final-head watcher is active. Do not rerun CI to diagnose the old result.

All 55 history scripts and 14 upload/admin neighbors were selected in a 69-test invocation on a0b5. It reported 66/69: the complete-snapshot fixture failure was corrected afterward and passes 1/1 in isolation; testJarHistoryBoundsDeployments and testJarHistoryBoundsPeriodsAcrossRestart each reached the 30,000 ms watchdog while their serial loops continued making progress (about 118/200 and 128/204 iterations). Their cause is not established, so this group is not green. The full local suite is still waiting for /tmp/cn-fullsuite.lock and has no XML. Run it only as flock /tmp/cn-fullsuite.lock scripts/test.bash --local --test .; do not bypass the lock. Record the exact admitted head and XML counts when it completes. Timeout dumps from the two bounds cases showed real FileDescriptor.sync0 stacks in JarDeploymentHistoryKt.writeJarActivationDocument (line 816) and JarReadPinStore.begin (line 47); the latter occurs while jarVersionLock is held. Some dumps had the writer and hash workers idle while the serial loops continued. This is a diagnostic pointer, not a root-cause conclusion. The durable sync writes are required and should not be removed without a proven replacement.

What was found and done

1. B1 confirmed. On 8c73d30c, forcing upload-history sink close returns empty 500 instead of 201, and holding receipt persistence blocks another route on jarVersionLock. The core fix captures receipt identity/enqueues during install, performs retention, then waits after releasing the global swap lock. It logs the failure and adds exact history_warning while all HTTP/HTTPS upload paths preserve success fields and restart behavior. Additional public journal-publication ordering proved that using the journal lock for writer admission still holds the swap lock; the candidate uses a distinct short writerSubmissionLock, with no disk work inside it. The upload test now also checks pruning and every existing success field. The atomic-persistence fixture's one stale 500 expectation now requires 201/status/SHA/full warning while retaining every byte/recovery assertion.
2. B2 confirmed. A provider reads first content, receives a different no-route upload during readiness, and starts serving the first bytes; old history labels the second bytes active. The core pre-start capture/pin fixes this. A live STARTING container keeps its original capture for resumed readiness, successful activation transfers/commits the capture, and failed or detached starts release it. All three public capture/readiness/cleanup regressions passed in the core group.
3. B3 confirmed. Empty/garbage pending JSON blocks later route persistence. Recovery now logs path/parse cause, renames it to .corrupt-timestamp and continues. Event and read-pin pointer data is synced before rename. A valid-shaped damaged record pointing at protected.bin also reproduced deletion of that unrelated file; generated-pin validation was added. That validator initially normalized only image paths; a second public test using nested/../app.jar failed because it quarantined a legitimate generated pin. Candidate efe03d374685ec9adc799b26f4e4d94a7b6bc864 normalizes both sides after that proof. The corrected complete-snapshot regression passes 1/1; two bounds tests remain unverified after reaching their 30-second watchdog while their serial loops made progress.
4. B4 confirmed. Clean local 0/1 baseline shows an upload remains pending after history close fails, EventStore is still writable, and the upload responds only after the writer is released. The earlier grouped watchdog also shows a second accepted upload stranded in FutureTask.get after shutdownNow removes its queued task. Candidate close tracks accepted FutureTasks, cancels every drained task, cancels outstanding accepted tasks after any unsuccessful close, and preserves recovery state while a worker lives. Admission closes under both locks before shutdown calls inside the cancellation finally. Nursery closes EventStore in finally. The final test checks two exact cancellation-warning responses plus EventLog-is-closed behavior and safe cleanup.
5. Item8 confirmed/core green. Cross-device hardlink into a symlink target returned 500 in public inspection. Shared native pin helper now falls back to atomic completed copy; the core public cross-device inspection/activation/upload/rollback test passes.
6. Item6 partial. Held cold digest-hint I/O no longer holds jarVersionLock; public second-route start passes in the core group. The full upload copy/sync still owns the global lock, so broader unrelated-install contention remains open. Removing it is a larger per-path locking change, beyond the optional small-and-test-proven change here.
7. Item7 unconfirmed/unchanged. The drafted filesystem barrier happens after publishRoute removes normalization, and therefore cannot establish queued-activation-before-old-normalization. That draft was removed, no result from it is accepted as proof, and no normalization implementation was changed.
8. Item4 report only. Production construction supplies a config-adjacent data directory. PID/start-time heuristics in no-data-directory cleanup remain unchanged; the review's liveness/path claims were not test-proven here.

Independent test and code passes checked capture ownership through startup/resumption/kill/shutdown, writer/journal lock ordering and closed visibility, FutureTask terminal paths, recovery retention, complete diagnostics, and failure cleanup. Review findings were corrected before relying on CI.

Relevant PRs and refs

Repo	Branch	Remote head	PR	What is on it / state
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/feature/jar-deployment-history-a	efe03d374685ec9adc799b26f4e4d94a7b6bc864	https://github.com/CodexCoder21Organization/ContainerNursery/pull/668	PR now points to the candidate; required final bld-build watch is active with no terminal result yet
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/checkpoint/jar-history-review-reproductions	304b757f8a25ef00f76d115c80d6251d9581ce44	no separate PR	Reproduction-only checkpoint, superseded by the candidate
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/checkpoint/jar-history-shutdown-review	efe03d374685ec9adc799b26f4e4d94a7b6bc864	candidate for the open PR	Production source matches tested a0b5; complete-snapshot fixture now passes 1/1; 69 group on a0b5 was 66/69 with two bounds watchdog results unresolved

| PlanRepository | https://github.com/CodexCoder21Organization/PlanRepository/tree/handoff/fix-668e-transport-fallback | 4324a896554e02035bd3d4ac157838dc5b37dadf (last committed snapshot before this note) | https://github.com/CodexCoder21Organization/PlanRepository/pull/11082 | Git fallback snapshot for existing Handoff service ID hf-2026-10-19-finish-verification-of-jar-history-server-review-fixes; no service record was created here | Draft document PR. This row records the last committed snapshot before this note; this note advances the branch, so re-check the current head via the PR URL and git ls-remote. |

The ContainerNursery checkout/stash/worktree/local-only-commit sweep is clean; both execution agents shared one checkout and have no unique unpushed work. All code and reproducers are on remote branches. No code was merged, enqueued/dequeued, published or deployed in this lane; only Git branches, the requested PR, and this handoff are being updated. Raw logs/XML/caches/JARs are deliberately excluded from source commits under the owner's artifact rule; essential evidence, commands and test names are retained here.

Latest candidate results at 2026-10-19 20:54 UTC

The 13-test new/changed group produced 13/13 PASS from the real runner on commit a0b5b07f8dfa6536387d2fc9a7459beb30f70da6. The corrected testJarHistoryReadKeepsCompleteSnapshotDuringHashAndPersistence fixture produced 1/1 PASS on efe03d374685ec9adc799b26f4e4d94a7b6bc864. The latest candidate branch's production source matches a0b5; the snapshot test fixture changed afterward. The 69-test invocation on a0b5 reported 66/69, with the corrected fixture accounting for one failure and two bounds tests reaching their watchdogs while still making progress. Do not call the 69-test group green. The full local suite has not acquired the required lock and has no XML. Raw XML/logs and build outputs remain outside source commits under the lane's artifact rule.

Runner evidence and resumption commands

Earlier core c1282af XML is 9/10: nine B1–B3/item6/item8 checks green; B4 deliberately unchanged and red. Corrected B2 readiness baseline was 0/1: expected original SHA 5dab60ed3799e4c8716140020ca92c7e0cfc66e888b87b125732df0094f48087, got replacement SHA 1d643e2c389ae732ad19d709f4022127022a9f83adfab4dbbea99ac7f91d82fa. Clean B4 baseline 0/1: upload TimeoutException while writer held, EventStore write kotlin.Unit, upload returned 201 only after release. B1 journal-admission probe failed in all three automatic runner attempts. Invalid-pin and legal-alias regressions each failed 0/1 at their intended assertions before fixes. No preparation interruption is counted as a passing suite.

Use scripts/test.bash --local --test <name> with repeated --test flags and --log result.xml; parse <test success="true|false"> elements, not JUnit testsuites. The 13 focused names are:

testJarHistoryUploadDocumentCloseAndCleanupFailures
testJarHistoryCapturesContentBeforeReadiness
testJarHistoryRestartSkipsCorruptPendingEvents
testJarHistoryUploadPersistenceDoesNotHoldVersionLock
testJarHistoryResumedStartupKeepsOriginalContent
testJarHistoryFailedStartReleasesCapture
testJarHistoryOtherRouteStartsDuringDigestHintRead
testJarHistoryCrossDeviceSymlinkActivationPinsTarget
testJarHistoryUploadDoesNotAcknowledgeAfterShutdown
testJarHistoryShutdownCompletesAcceptedUploadAndClosesEventStore
testJarHistoryRestartRejectsInvalidPendingPin
testJarHistoryUploadDoesNotWaitOnPendingJournalUnderVersionLock
testJarHistoryPersistsAtomically

All history scripts are tests/testJarHistory*.kts. The 14 neighbors are FirstJarUploadSurvivesColdTypeMetadata, MultipartJarUploadThroughAdminProxy, LargeJarUploadStreamsWithoutOOM, HttpsAdminApi_queryParametersAreHonored, AdminApiHttp_structuralBombAndHostileBodiesDoNotTakeDownServer, AdminMalformedContentTypePreservesKtorErrors, AdminRoutesDecodeUtf16AndPersistAcrossRestart, AdminTextEmptyBodiesPreserveExactErrors, AdminTextMatchesKtorCharsetSelection, PathMatching_httpsFacadeProvider_matchesImageWithQueryArgsToUploadPath, PathMatching_matchesImageWithQueryArgsToUploadPath, UrlRouteUploadNonexistentReturns404, UrlRouteUploadWithProtocolAndDomain and UrlRouteUploadWithProtocolDomainAndTransport (all have test prefix).

Next steps

1. Re-verify the remote PR, candidate and CI state first. If the local runners still exist, read their actual terminal result and XML before restarting anything. Source candidate, tests and public reproducers are all on the linked checkpoint branch.
2. The old complete-snapshot failure came from a fixture that persisted the hash before the held read. The corrected public fixture now passes 1/1 on efe03d; do not treat the old failure as a current production-code finding.
3. Resolve testJarHistoryBoundsDeployments and testJarHistoryBoundsPeriodsAcrossRestart locally; their 69-group watchdog outcomes are not explained by the recorded iteration progress. Re-run the 69-test group on the exact final source/test head, then run the full local suite behind the lock and report XML counts. Do not use CI reruns for diagnosis. Do not describe waiting/preparation without XML as a passing suite.
4. Preserve the prior handoff's two explicit unfinished coverage cases: a forced public upload overlapping a held cold digest source, and closing the ephemeral history owner while an inspection read is held. This lane has not silently claimed them complete.
5. Verify the PR description and Review fixes/Validation sections contain the exact final counts after the pending REST update.
6. Read the tracked final-head bld-build watcher to its terminal result. No CI rerun, dequeue, merge, publication or deployment is authorized.
7. Stop every owned process and remove checkout/private build outputs when the lane ends. If verification remains unfinished, update/release this handoff with exact counts and remote heads.

Operational notes

Kompile source compile takes several minutes; preparation validates/hashes dependency inputs and can stay CPU-active without a test verdict. Thread dumps showed genuine JVM code generation and InputValidationPass SHA validation. Prefer observable process termination and hard caps to polling for a success log line. The full old-head ver-668 run stopped after this lane's mandated checkpoint changed the PR head under that lane's stop-on-head-change rule; it ran bodies but produced no XML, so no counts are available. The full lock advanced to other waiting suites; never bypass it.

Required docs read in full: repository AGENTS/README, supplied review/contract/previous findings, https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md , https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md , current create-handoff SKILL.md and canonical handoff format. The old format README's git merge exception is retired by the current skill and also conflicts with the lane's explicit prohibition; service CLI updates do not merge anything. The report-challenge tool auto-enqueues/merges, so it was not invoked under this lane's explicit ban; coordination/preparation friction is recorded here instead.

Prior lane snapshot — historical, superseded by the live state above

Handoff: Finish verification of the JAR-history server review fixes

Written 2026-10-19. RE-VERIFY: this is a write-time snapshot. Re-check https://github.com/CodexCoder21Organization/ContainerNursery/pull/668 with gh pr view <URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt before acting. Verify the remote branch head with git ls-remote.

Design decisions in force

The owner asked to "keep track of all recent jars deployed for each service, along with the date/time it was uploaded and the date/time that each of the jars was the jar actively serving the site and make it easy to roll to a different jar" and to "ensure test coverage is comprehensive".

The review lane's binding instructions (2026-10-19):

For each candidate, write a public-API test that FAILS on the current head for the stated reason (deterministic ordering via the injected filesystem/clock seams, no sleeps) or record clear evidence it is not a real defect; fix only the real ones at their cause.

Rules: never raise timeouts, never reduce iterations, never add retries or sleeps, never weaken assertions, no reflection, nothing made public for tests.

kompile test-script rules apply (no top-level functions with declared return types or expression bodies; no if-as-expression without else); run changed and history/upload/rollback tests through the REAL runner (scripts/test.bash --test <name>) and report counts.

Keep build outputs small (disk was recently full).

Forbidden: merging, enqueueing, publishing, deploying.

The lane also imposed a 90-minute clock ending 19:18 UTC, milestone pushes, at-most-once-per-60-second PR polling and a 40-minute maximum bld-build watch. No permission to land or deploy code follows from this handoff. Contract I1 fixes the content identity at activation; I2 preserves activation order; I3 avoids hashing known content on startup; I4 preserves configured history on disk; I5 bounds records to 200 per route/path; I6 returns complete snapshots without waiting for an activation hash. A1 further bounds periods to 100 per deployment with dropped_period_count. Api-only A2/A3 remain outside this server change.

Mission and current work

Finish the current-head public regressions and history/upload/rollback neighbors, review the two remaining hostile-caller coverage limits below, and watch the current bld-build result. All four candidate mechanisms were confirmed and implementation is pushed. Final verification is still in flight; an earlier integration head passed 6/6 and the first final-head short checks passed 3/3.

Relevant PRs and refs

Repository	Branch	Remote head	PR	State
ContainerNursery	https://github.com/CodexCoder21Organization/ContainerNursery/tree/feature/jar-deployment-history-a	8c73d30c1afde79a7831ab09fdfb05c0b48d4138	https://github.com/CodexCoder21Organization/ContainerNursery/pull/668	Rebased onto 87764adf and pushed; final verification incomplete

No code was deployed or published. No PR was merged or enqueued. All unique implementation and reproducer scripts are on the remote branch. One checkout was used and the worktree/stash sweep was clean; no delegate has a separate checkout or unpushed code. Raw test outputs, caches, and JARs were deliberately kept out of source commits per the lane's artifact rule; essential evidence and exact test names are carried below.

Next steps

1. Re-verify current branch/PR/CI state first. Do not rerun a red job to diagnose it; reproduce any failure locally on the exact public path.
2. Read the final counts below. Run unfinished selected tests through the real runner. All 46 history scripts and 10 upload/rollback/admin neighbors were selected by the broad attempt, but that invocation ended during preparation with 0 bodies and is not a passing suite. A complete local/current-head suite remains required before declaring the whole verification complete.
3. Add and prove a deterministic public close-mid-read case for the new ephemeral owner: hold a read source, shut down, construct another nursery, verify its pin and live-owner record survive until the first reader finishes, then verify final cleanup. The implementation uses closeWhenIdle and was independently reviewed; this exact hostile-caller test has not been completed.
4. Strengthen the hint test to force a real public upload to overlap an open digest source. The current test checks open-source deletion tracking, startup and post-upload bounds, and cold persisted reuse, but does not force that overlap. Keep the existing 513-record seed and time budgets.
5. Watch bld-build with build-watchman 0.0.21 on the current head. Required gate is only bld-build; optional remote checks do not gate it. Do not merge, enqueue, publish artifacts, deploy, raise test timeouts, or weaken assertions.
6. Update the PR Review fixes/Validation sections with exact final counts and any corrected code. Use REST PATCH with a JSON body file if gh pr edit fails on deprecated projectCards.

The findings below contain the mechanisms, baseline assertions, reviewed limits and operational details. Local scratch paths are supplemental only; every reproducer needed to resume is on the linked branch.

---

JAR-history server review fixes — lane fix-668d

Written 2026-10-19 19:10 UTC. This is a write-time snapshot; re-check the PR and runner results before acting.

Plan and state

1. DONE: read the contract and prior findings, clone the requested branch fresh, and rebase onto main.
2. DONE: obtain intended public-API assertion failures for all four candidates.
3. IMPLEMENTED, VERIFICATION IN PROGRESS: fix the four mechanisms, including the additional ephemeral crash-discovery case.
4. DONE: independent test review and a separate code review; address concrete findings.
5. IN PROGRESS: final-head targeted checks and bld-build. The broad 56-script invocation did not complete.
6. DONE: push the code and update the PR's Review fixes and Validation sections.
7. PENDING: stop all owned local processes before 19:18 UTC and publish the final evidence snapshot.

Repository state

OBSERVED: source branch feature/jar-deployment-history-a is pushed at 8c73d30c1afde79a7831ab09fdfb05c0b48d4138, rebased onto main 87764adf. The working tree and stash list are clean. The README rebase conflicts were resolved preserving both the JAR-history behavior and main's admin JSON-body behavior.

• PR: https://github.com/CodexCoder21Organization/ContainerNursery/pull/668
• Source: https://github.com/CodexCoder21Organization/ContainerNursery/commit/8c73d30c1afde79a7831ab09fdfb05c0b48d4138
• Branch: https://github.com/CodexCoder21Organization/ContainerNursery/tree/feature/jar-deployment-history-a
• Current bld-build: https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39306681824
• Current release-package check (passed): https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39306681616

No merge, merge queue, artifact publication, deployment, production restart, or CI rerun was performed. Only the requested feature branch and PR description were changed remotely.

Candidate outcomes

1. History disk work under the route mutex — confirmed

OBSERVED: activation called digestCache.get() while completeStartBookkeeping still owned the route lease. Holding its persisted digest source blocked the public kill request. The public regression failed at the assertion that kill finish before that source was released.

The fix publishes RUNNING bookkeeping, then uses the existing lease-release helper for activation history. The existing start gate preserves activation order. No retry was introduced. The public fake provider honestly reports 503/stopped if management kills it during the held history operation.

The post-fix six-test group passed this case and the existing held-JAR-source management case.

2. Unpersisted working-set eviction — confirmed

OBSERVED: after 36 distinct public upload/activation routes with no dataDirectory, revisiting the first route returned zero deployments instead of one. The three 32-entry eviction guards now apply only when documents exist on disk to reload. Unpersisted route/path entries remain for the nursery lifetime; the existing 200-deployment and 100-period limits remain.

RATIONALE: I4 supplies a backing store when a data directory is configured; I5 limits records per route/path. Neither authorizes silently dropping an unpersisted route's whole history. The README now states the optional-persistence behavior.

3. Crash-window pins and activation records — confirmed; final integration verification pending

OBSERVED: activation created a native content pin before committing its event. The old deletion ordering could also leave a native pin with no event. A missing capture could keep an incomplete event indefinitely. Unknown-content inspection pins had no restart cleanup pointer.

The fix commits the activation pointer before native creation, uses an atomic partial-copy install, saves a separate resolved-content journal before deleting a pin, and removes incomplete captures. The writer applies an event before its pointer is removed. Existing same-content activation logic is idempotent; a separate old double-application mechanism was not confirmed. The replay regression removes the saved route and derived hints, rebuilds from the pending resolved event, then restarts again and checks the exact single period.

Inspection pins now get durable cleanup records before native creation. Failed cleanup keeps the record. Tests force deletion failure by replacing the pin path with a non-empty directory, then restore the real pin; no OS-user-dependent chmod assumption remains.

Further OBSERVED gap: unpersisted stores used random temporary paths, so a new instance could not discover a crashed instance's pointers. The new public seeded-dead-owner test failed because its native read pin remained. The final fix writes owner.json synchronously before any pin-producing operation. Startup scans only marked stores, removes dead/closed owners and process-start mismatches, and preserves matching live processes and unknown ownership. In-process registration protects startup re-entry. The regression also covers a live owner, a closed owner whose JVM remains alive, and a reused PID/start mismatch.

Readers own pins independently of the writer/hasher. closeWhenIdle rejects new pins and releases the store only after both workers have stopped and all active readers finish; its callback performs final cleanup before marking the owner closed. Independent review found this deferred-cleanup issue and it was corrected. A dedicated current-head close-mid-read public regression has not been run separately; the existing new shutdown case covers completed failed-read cleanup, and static review covers the deferred-reader path. Do not describe this as a completed comprehensive concurrency proof.

4. Persistent digest sidecars — confirmed

OBSERVED: the corrected baseline loaded 514 valid persisted hints and startup left all 514, failing the 512-file assertion. The initial draft's 513-upload loop timed out before its bound assertion; that was not accepted as defect evidence. The replacement deterministically seeds 513 real historical identities plus one genuine public upload, preserving the overflow condition without repeated retained-file scans.

The fix caps regular 64-hex JSON hint files on startup and after reads/writes. Disk access is serialized; active hint reads and the newly written hint are preserved. Eviction uses modification time and name. The test uses a hardlink alias with no upload receipt to require a cold persisted hint, checks zero JAR bytes read, and performs another real upload to verify the put path and full SHA value.

LIMIT: the test records any deletion of an open source but does not force a public upload to overlap the held read. Serialization and active-path protection were independently reviewed. A forced public-path overlap regression remains desirable before calling coverage complete.

Actual real-runner evidence

Every invocation used scripts/test.bash; preparation-only failures are excluded from assertion counts.

Baseline scenario	Result	Intended wrong behavior
Persisted digest source held during activation	0/1, failed	Public kill waits for source release
36 unpersisted routes	0/1, failed	First route has zero deployments
Activation journal/crash group	0/3, failed	Pin precedes event; pin survives interrupted deletion; incomplete event remains
Persisted interrupted read-pin restart	0/1, failed	Native pin remains after restart
Deterministic ephemeral shutdown	0/1, failed	Native pin remains after shutdown
Abandoned ephemeral store	0/1, failed	Startup leaves a dead owner's native pin
514 persisted digest hints	0/1, failed	Startup leaves 514 rather than <=512 files

OBSERVED: the earlier integration head c08ab4bf passed 6/6: digest-held management, existing JAR-held management, activation pin journal order, resolved activation replay, uncaptured activation discard, and persisted read-pin restart. This precedes the final hint-cap and ephemeral-owner additions and does not prove the final head green.

The final head's broad run selected all 46 history scripts plus 10 upload/rollback/admin scripts. It compiled the final source artifact, but was deliberately interrupted during preparation to prioritize short current-head regressions inside the 90-minute lane. Its result is recorded below; it is not a passing suite.

Final short runs: testJarHistoryUnpersistedRoutesSurviveWorkingSet, testJarHistoryBoundsDigestHints, and testJarHistoryRestartCleansAbandonedEphemeralStore. Results are appended from their structured files below. Other current-head history/upload/rollback checks and the full local suite remain unverified unless CI finishes them.

Reviews and limits

Independent review covered route lease/kill semantics, journal and deletion ordering, replay without a route snapshot, read-pin wiring/recovery, limits, live owner protection, deferred shutdown, and the digest cap. A private pending JSON naming a different pin requires changing a committed ownership record; no such state is produced by the atomic crash protocol. It was recorded as outside this review's crash-state model, not as an observed defect.

No test timeout was raised, existing iteration count reduced, retry or sleep added, reflection used, helper made public for tests, or existing assertion weakened. The temporary c08 baseline restore affected only two source files; the complete cap/close diff was saved and restored before final integration.

Required reading and operational notes

Read: the supplied contract.md (A1-A3 override), rev-668-final-findings.md, rev-668-final-coverage.md, fix-668c-findings.md, repository README/AGENTS, and the full official testing and philosophy documents:

• https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md
• https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md

Remote test preparation repeatedly connected without reaching bodies before bounded interruption. Local preparation involved a roughly ten-minute source compile plus artifact-cache contention; thread dumps identified the active compiler and exact file-lock owners. None was counted as a regression result. The old 513-upload timeout was traced to cumulative retained-file scans, not a first-upload deadlock. The revised bound fixture targets the persisted-cache contract directly.

This gh version's gh pr edit fails on deprecated projectCards; authorized description updates succeeded through gh api --method PATCH repos/CodexCoder21Organization/ContainerNursery/pulls/668 --input <JSON body file>. Build-watchman 0.0.21 gates on required bld-build; optional remote checks do not gate it. No hand-written CI polling loop or rerun was used.

The report-challenge CLI automatically merges its record. It was not invoked because this lane forbids merging. Friction is recorded here instead. The current Handoff skill uses the service directly; the older PlanRepository README's handoff-PR merge exception was not used.

Raw logs, structured test-result files, caches, and JARs are kept out of source commits as instructed. All implementation and reproducer scripts are on the requested remote branch. The handoff carries the essential evidence and commands so local scratch files are not required to resume.

Live PR snapshot at 2026-10-19 19:13 UTC

{
  "headRefOid": "8c73d30c1afde79a7831ab09fdfb05c0b48d4138",
  "mergeStateStatus": "BLOCKED",
  "mergedAt": null,
  "state": "OPEN",
  "statusCheckRollup": [
    {
      "__typename": "CheckRun",
      "completedAt": "0001-01-01T00:00:00Z",
      "conclusion": "",
      "detailsUrl": "https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39306681824/job/118245452353",
      "name": "bld-build",
      "startedAt": "2026-10-19T18:56:01Z",
      "status": "IN_PROGRESS",
      "workflowName": "Bld - Build and Test"
    },
    {
      "__typename": "CheckRun",
      "completedAt": "2026-10-19T18:59:54Z",
      "conclusion": "SUCCESS",
      "detailsUrl": "https://github.com/CodexCoder21Organization/ContainerNursery/actions/runs/39306681616/job/118245450886",
      "name": "bld-build-release",
      "startedAt": "2026-10-19T18:56:01Z",
      "status": "COMPLETED",
      "workflowName": "Bld - Build Release Package"
    },
    {
      "__typename": "CheckRun",
      "completedAt": "0001-01-01T00:00:00Z",
      "conclusion": "",
      "detailsUrl": "https://kompile-build-ci.kotlin.build",
      "name": "kotlin.build (kompile-remote-build)",
      "startedAt": "2026-10-19T18:56:03Z",
      "status": "IN_PROGRESS",
      "workflowName": ""
    },
    {
      "__typename": "CheckRun",
      "completedAt": "0001-01-01T00:00:00Z",
      "conclusion": "",
      "detailsUrl": "https://kotlin.build",
      "name": "kotlin.build (remote)",
      "startedAt": "2026-10-19T18:55:54Z",
      "status": "IN_PROGRESS",
      "workflowName": ""
    }
  ]
}