← Priority list
Blocked

Move BuildTestEmbedded storage and recovery preparation outside projection lock scopes

Partial projection preparation fix; original scenario bar and suite remain incomplete

Lane allowance reached; continuation needed for caller split and verification

Handoff document

Markdown

Lane I11c terminal partial checkpoint, 2026-10-21 19:03 UTC

Original eight-scenario acceptance is incomplete. No PR was opened. Primary pushed head is 56b76eb91aa290971136eed24db62e2864e2d71e on https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-resident-preparation.

Implemented partial source: detached direct resident mutation/snapshot preparation, detached deletion snapshot preparation, publication acquisition that releases native projection ownership while retaining logical reference pins, and preservation of the primary read error through retained-source cleanup. Getter recovery and enclosing event paths still retain projection ownership through work that must be moved outside it. The current audit covers 53 service projection scopes and 12 publication wrappers.

Checkpoint-specific XML evidence: original snapshot boundary 10/10 first-attempt passes, isolated reversal 3/3 red; publication boundary and reference lifetime each 10/10 first-attempt passes, each reversal 3/3 red; paired read/close error and close interruption each 10/10 first-attempt passes, each reversal 3/3 red. Other seven original scenarios remain unqualified; each established original baseline has three red attempts. Independent count review covers 67 distinct XMLs with zero mismatches. Counts and checkpoint URLs: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11c-suite-evidence/forensics/laneI11c/scenario-matrix.md. Earlier repetitions are not attributed to the final head.

The one required complete local suite was run through the shared suite-slot wrapper from frozen head 56b76eb91aa290971136eed24db62e2864e2d71e. Slot waiting was 2,565.032479 seconds and was excluded from the lane allowance. At the reserved terminal point it still had no XML; ordered process closure returned exit 143. Complete-suite counts and main-history classification are unavailable and incomplete. No second suite started. Controller and test subprocesses finished before shared-wrapper release; ps -eo pid,args -ww subsequently showed zero lane-owned processes.

Main advanced during the run from 253cfb58837c7cd9d4b01f635049ac8baa7a5da3 to 778a8bc3143b75b9cfc6ad41cdd395d66f06f25f. The 24 changed paths include BuildTestEmbeddedService.kt and RetiredRunDropletCleanup.kt. Earlier source-equivalence claims apply to the earlier main. Compare the new main before continuing; the frozen suite head was not rebased while running.

Next concrete work: finish the enclosing mutation/event and recovery caller split before changing the getter; retain disk writer order and processing admission; create a detached candidate outside projection/state ownership; validate source, deletion and accepted resident outcome at short publication; retain the publication-cycle protection. Qualify all original eight scenarios 10/10 on the requested final source and reversal 3/3, complete the local suite under the shared slot, classify every history against current main, then open the requested PR without merging or queueing it.

The two added run-read probes remain red and do not isolate state ownership from publication waits; their assertions are unchanged. The getter source-change scenario is a reviewed draft only, with no compile/runtime claim. Main-control checkout is fresh and has no completed control run. No inherited assertion was changed, no timeout increased, no iteration reduced, no test disabled, and no merge, queue action, artifact publication or deployment occurred.

Durable findings: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11c-suite-evidence/forensics/laneI11c/findings.md. Evidence, audits, classifier and continuation brief: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-suite-evidence/forensics/laneI11c.

Reverified pushed branches:

  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-error-reversal at 6cfd83db2112d7e2002975a453d0918f2fa573a0
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-projection-preparation at 9fcf8bc3aab232466091dea25edef99c306b25d6
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-publication-reversal at a62fb8166b8e0a1b7e3a0a99788c4edc6f63e1f7
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-resident-preparation at 56b76eb91aa290971136eed24db62e2864e2d71e
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-resident-reversal at 1ec7c00daef439ede01010d7ff2f331464ccbaed
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-source-order-experiment at 51e6800c1e88bd8e1222bb5609086fae10f23acb
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-suite-evidence at 461687ba98b71800ccc8cd6c2a6a7b258f5102c7

All older handoff content and branch references are preserved below.


Move BuildTestEmbedded storage and recovery preparation outside projection lock scopes

RE-VERIFY before continuing: this records the repository at 2026-10-21 09:45 UTC. Check the branch head, main, and linked service change again. Production was not queried during this lane.

Why this work exists

The owner reported that https://droplets.wasmserver.com/ "is super unreliable, frequently rendering various errors/notices due to bugs in UrlResolver or UrlProtocol or something else" and asked for the underlying root cause to be fixed; the ContainerNursery route stops the WUI after 5 idle minutes, so most human visits are cold starts.

Production createUploadSessionStructured reported Service handler timed out after 30 seconds without producing a result. Upload cleanup needed a run projection lock whose owner could be waiting for recovery capacity. The consuming-service change https://github.com/CodexCoder21Organization/BuildTestServerService/pull/434 moves cleanup off intake, but marker ownership is in run.json and still needs the same projection lock. This library owns the required change.

Current state and durable work

This is an incomplete implementation effort stopped at the lane checkpoint. Diagnosis, acquisition map, and seven consistently red public-boundary scenarios are saved. No production source change has been made. No implementation PR exists, and no artifact was published or deployed. There is no pending user decision; the remaining work is engineering work.

Repository Branch Verified remote commit State
BuildTestEmbedded wip/laneI11-projection-lock-scope 45cf6896a56f90b90d6f576c826150c455402697 Tests and forensics only; not merged, published, or deployed

Base main: 37e96475c17d02968a35552577625c9c99520ab3. The branch is the only checkout for this lane and has no stashes. Use a fresh checkout under your own workspace; do not depend on this local directory surviving.

Observed mechanism

The mechanism is broad native projection ownership around loading, storage, recovery admission, and publication. The evidence that demonstrates it is the source acquisition map plus seven public-boundary scenarios that each report their intended assertion in all three invocations, including all three runner attempts per invocation.

RunEventProjection.kt:5585 enters native ownership; getOrLoadTestEventProjection at :6103 loads and :3011 enters recovery admission. The store factory at :5080 is called through outsideRecoveryMaterialization, which returns a slot at RecoveryMaterialization.kt:76 and waits to renew it at :83/:35 while native projection ownership remains held. The same-run snapshot writer enters projection ownership at BuildTestEmbeddedService.kt:26753 and its outer mutation at :27185; deletion does so at :16155. Retained live folds also call store consume under processing admission at RunEventProjection.kt:1778-1780. Moving only store creation does not cover that path.

The complete map covers all 53 service scopes, two extra storage scopes, two memory-only seed scopes, 12 indirect history-wrapper callers, and candidate/cache cleanup. It records the file and line for every scope and the current acquisition order. A separate review found no missing or misclassified scope.

History wrappers acquire projection before publication ownership. Other service mutations acquire terminalization/disk/projection/state before file publication ownership. A generic temporary projection release inside the loader, while retaining file ownership, can form a cycle with another writer that takes projection and waits for file ownership. Preparation must occur outside the enclosing owners, with a source-generation/cut check during short publication. Current publication at RunEventProjection.kt:6104 only selects an existing cache or candidate and is not sufficient for that change.

Candidate disposal at :6115 and cache retirement at :6019 through :5640-5652 can call cleanup under the caller's native scope or another run's native scope. Native outer-release cleanup at :5614-5623 is already after that run's release. Preserve exception ordering, cleanup ownership, and the existing public binary shape.

Historical production captures were checked against the retained I6 evidence. They show the projection/recovery chain; they do not establish why production recovery slots were occupied. Do not present that separate question as resolved.

Tests and measured results

Seven new files in tests/, one scenario each:

  • testRunSnapshotPreparationDuringProjectionStorePause.kts
  • testRunSnapshotPreparationDuringProjectionCapacityWait.kts
  • testRunSnapshotPreparationDuringInitialProjectionCapacityWait.kts
  • testUploadMarkerReleaseDuringProjectionStorePause.kts
  • testUploadMarkerReleaseDuringProjectionCapacityWait.kts
  • testProjectionBoundaryDuringRunSnapshotWrite.kts
  • testRunSnapshotPreparationDuringProjectionStoreAppend.kts

Each is red in 3/3 invocations on unchanged main source. Each invocation records three unsuccessful attempts, for 21 case histories and 63 attempts total. The intended assertion, not compilation, accounts for each result. The baseline report and XML evidence summary preserve exact counts, source identities, hashes, and classifications. Raw generated XML and logs remain in .git/laneI11 locally and are not required to recover the work.

All fixtures use real engine instances with background workers disabled. Paused in-memory store collaborators and real FIFO journal readers control the exact order. Readers occupy precisely recoveryMaterializationStats().limit slots. A forwarding filesystem pauses an actual run.json.tmp write. Public cancellation's beforeRunSnapshotPersistence observer is inside the outer projection scope; it measures admission, not complete cancellation during recovery admission. Marker cases cancel first, pause controlled replay, delete a terminal owner, create a replacement owner before replay resumes, and verify it after restart. They contain a visible prior projected row so stale publication is observable. That continuation is not yet verified against a fix, since main stops at the earlier intended assertion. The append case pauses consume after recording a completion; its factory returns directly.

Eleven existing projection, recovery, and marker scenarios passed on their first attempt (11/11) against unchanged main source. These are baseline measurements, not proof of a fix. No existing test was edited. A separate review checked the admission boundary, marker path, acquisition map, resource cleanup, and append scenario; no material outstanding test review item remains.

No 10/10 green run, reversal control, or complete local suite has been run. These remain required.

Requirements to preserve

The lane brief supplied these requirements on 2026-10-21; they remain applicable:

a projection lock is held only for in-memory mutation

recovery-slot waits happen outside every lock

Storage writes and recovery-slot waits move outside it, and staleness is re-checked when publishing under a short lock.

Keep the existing guarantee that a concurrent recovery cannot race a writer, and say how.

Never raise a timeout, cut iterations, or weaken an assertion.

Run builds and tests with --local. No merge, queue entry, publication, deployment, or production restart is authorized. Before any build, fetch and rebase main. Run targeted scenarios while iterating. For the final complete local suite, check ps -eo pid,args -ww and /proc process working directories for the machine-wide slot; no outer cap shorter than 150 minutes. The final-suite wait is exempt from the ordinary 80-minute checkpoint stop; total lane limit is 3.5 hours. Do not watch CI after opening the requested PR.

Continue in this order

  1. Read README and the Testing Architecture, especially Test-Driven Development and Never Calibrate a Reproducer to a Machine. Read the saved scope/map/baseline and original I11 scope. Re-verify I10g marker notes and the service PR state.
  2. Design the shared writer/private-fold protocol before implementation. Specify who owns the storage operation, generation/cut capture, serialized writer admission, candidate publication, stale-candidate disposal, retained live-fold consume, cache retirement, re-entry, close, and restart. Keep recovery admission outside all locks and preserve recovery/writer ordering. Audit every mapped enclosing acquisition order. Do not apply a loader-only temporary unlock.
  3. Implement that protocol across the mapped paths, keeping the public API and existing assertions. Add any additional public-boundary scenarios needed for the chosen ownership protocol. Run the seven saved scenarios unchanged to green, 10/10 first attempts each. Run the reversal control to establish they become red when the implementation change is reverted.
  4. Run existing projection/recovery/marker tests and classify every history against the saved main baseline. Run the complete local suite from the final head once the machine-wide slot is available. Record first-attempt counts from XML and investigate results locally.
  5. Perform separate test and correctness review passes. Checkpoint each milestone to the branch. Fetch/rebase, run final targeted tests, push with the requested lease check, and open a PR to main with the required owner provenance, the production RPC text, mechanism, this map, and https://github.com/CodexCoder21Organization/BuildTestServerService/pull/434. End its description with the requested Claude Code attribution. Do not merge, queue, publish, deploy, or watch CI after opening it.

Local working details

Prior lane directory: /tmp/claude-501/-code/79f96be0-ae83-4479-b38c-8894e2722ba8/scratchpad/workspace/laneI11/BuildTestEmbedded-main Findings: /tmp/claude-501/-code/79f96be0-ae83-4479-b38c-8894e2722ba8/scratchpad/out/rI11-findings.md Durable findings mirror.

No current production version is asserted. No code is deployed or published but unmerged. The service PR is a separate existing open change, last observed at 2026-10-21 09:27 UTC; recheck before relying on it.

Final checkpoint: branch head 45cf6896a56f90b90d6f576c826150c455402697 includes the brief item audit and the central handoff link. The worktree is clean and the stash list is empty. No source implementation, complete suite, or fix PR was added.

Lane I11b continuation checkpoint

RE-VERIFY: Terminal lane checkpoint at 2026-10-21T11:08:55.898179+00:00. The lane stopped at its time limit. The requested slice is incomplete; no product fix or PR is adopted. All 53 mapped service scopes remain unchanged.

Durable branch: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11b-snapshot-preparation Final checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/0669e7bf66356d1ac5325935e92f729974c3896d Findings: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11b-snapshot-preparation/forensics/laneI11b/progress.md Evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11b-snapshot-preparation/forensics/laneI11b/test-evidence.json Review: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11b-snapshot-preparation/forensics/laneI11b/live-review.md Resident publication facts: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/laneI11b-snapshot-preparation/forensics/laneI11b/resident-publication-notes.md

The seven original scenarios remain red 3/3 each. The added public-API live waiting-order scenario testLivePendingCompletionDuringRecoveryCapacityUse is red 9/9. Twelve projection/recovery/snapshot/marker neighbours passed first attempt on final source. No original test, timeout or iteration count was changed.

The complete local suite entered through the supplied machine-wide slot after about five minutes queued. It was stopped at the lane limit with zero pass/red verdict markers and no final XML, so complete-suite counts are unavailable. The read-only process snapshot showed a runnable compiler worker and a coordinator waiting on a future. All tracked local processes were stopped; ps -eo pid,args -ww confirmed none remained. Raw local evidence is under .git/laneI11b-evidence in the checkout, while compact attempt data and hashes are pushed.

A generic getter unlock was set aside because retaining publication ownership while reacquiring projection ownership creates a two-caller ownership cycle. A live capacity bypass passed its experimental scenario 10/10 with a red reversal, but did not implement the requested admission order and was set aside. That experiment is retained at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/c6babb5d665803b2ecfbb25a41bcfa3def223711 . Product source and README in the final branch match main.

NEXT: Split outer mutation/deletion/recovery callers before changing the getter. Retain disk writer order, prepare a detached candidate outside projection/state ownership, then validate expected disk identity, accepted resident outcome, source generation and deletion state at short publication. ResidentRuns.beginPublication captures accepted outcome order; source receipts alone do not prove the live object stayed unchanged. Preserve processing admission and move its wait before projection ownership, or release and validate. Do not add a generic getter unlock or simply bypass the bound.

The requested first-slice green 10/10 bar, original-scenario reversal checks after a real fix, successful complete suite and implementation PR are not done. Nothing was merged, queued, published or deployed. Existing constraints and the lock map still apply.

Lane I11c active checkpoint

RE-VERIFY 2026-10-21T13:49:57.740379+00:00: Product source still matches main; no implementation is adopted. The current checkpoint is https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/e3ef5f56 and the branch is https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-projection-preparation .

All eight inherited scenarios were independently repeated: each has three red attempts, 24 total. XML attempt-history was read as a sibling of the test node. The new testProjectionBoundaryDuringPublicationWait also records three red attempts at the intended boundary: the event caller is waiting and a separate projection caller cannot enter while actual source staging is paused. It releases storage before joining callers, so the measurement does not leave a continuing wait.

The invariant table and complete current caller audit are in forensics/laneI11c/invariants.md and caller-audit.md on that branch. All 53 mapped service scopes and 12 history-wrapper callers were checked. The retention callback at service :4692 reaches replay through flush at :4714 while state/publication ownership remains; this detail was absent from the earlier map.

Implementation remains open. A private cold-fold change alone cannot release enclosing state/publication ownership or preserve a live retained fold during supplied-store processing. Ordinary state-only mutation at service :10702 must be accounted for alongside accepted outcome ordinals. No green/reversal implementation bar or successful complete suite is claimed. Nothing merged, queued, published or deployed.

Lane I11c current checkpoint — 2026-10-21 16:47 UTC

OBSERVED: primary56b76eb91aa290971136eed24db62e2864e2d71e is pushed; product source is151244ce. Resident mutation snapshot preparation and deletion tombstone staging now use detached memory decisions outside projection/state ownership. Publication-owner acquisition releases native projection holds while logical scopes pin references. Primary read errors retain distinct close errors. These are partial changes; most53 audited service scopes remain unchanged.

Measured counts (canonical first attempts, read from XML): required source-snapshot staging10/10; new publication boundary10/10; new publication reference lifetime10/10; paired read/close cleanup10/10 and close-interruption10/10. Source staging reversal is red3/3; publication boundary/reference reversals each red3/3; cleanup read/close and interruption reversals each red3/3, with shared-error-object control passing once. Deletion staging and accepted-outcome neighbours pass their recorded single first attempts. Source-head details are preserved in forensic JSON; these are not eight completed original scenarios.

The other seven original capacity/store/live scenarios remain without a passing fix. Getter preparation is not wired to the native suspension helper. Outer event/recovery state scopes remain. The sibling-preparation reference-lifetime scenario remains red on its baseline. Two lane-added public read probes remain red but do not isolate state ownership: they also serialize on source publication and terminalization. Their assertions remain unchanged and their assumptions require an explicit decision.

Complete local suite is queued from56b76eb91aa290971136eed24db62e2864e2d71e through the mandatory shared suite-slot wrapper. At16:46UTC its tracked process is waiting for the shared slot, with no tests begun and no XML. Waiting is excluded by the lane limit. No CI monitoring, merge, queue entry, publication or deployment occurred. No PR exists. Final suite counts/classification are pending; do not treat this checkpoint as complete.

A separate acquisition review refuted the proposed ordinary service-peer state cycle because those native-to-state paths first take the same disk owner. The existing corrected publication fixture uses a top-level event helper, not service state. An additional caller-composed state scenario was not established and was not added. Current scope map and readiness table are at the primary branch under forensics/laneI11c/post-split-callers.md and readiness.md.

Durable branches (controls and experiment are unadopted):

  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-projection-preparation at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/1cfe44fd0811af31f1fb5eb8e708a515c21d0419
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-source-order-experiment at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/51e6800c1e88bd8e1222bb5609086fae10f23acb
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-resident-preparation at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/56b76eb91aa290971136eed24db62e2864e2d71e
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-resident-reversal at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/1ec7c00daef439ede01010d7ff2f331464ccbaed
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-publication-reversal at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/a62fb8166b8e0a1b7e3a0a99788c4edc6f63e1f7
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneI11c-error-reversal at https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/6cfd83db2112d7e2002975a453d0918f2fa573a0

Model usage

Recorded agent work, including concurrent agents

No model usage has been recorded for this handoff yet.

Record work with handoff-cli usage report hf-2026-10-21-move-buildtestembedded-storage-and-recovery-preparation-outside-projection-lock-scopes.

No status reports yet.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.