Finish the active upload replacement and committed run metadata qualification

Written 2026-10-27 04:26 UTC by fable-W3-1027.

RE-VERIFY: This is a write-time snapshot. Re-list open PRs, read their current heads/checks with gh pr view, and run git ls-remote for every branch below. The two upload branches were actively changing during this review. Do not overwrite them or treat author-reported tests as W3-run evidence.

Design decisions in force

The current owner request (2026-10-27) is to work through open handoffs, skip separately claimed work, use best judgement and ask no blocking questions. This worker may not merge, enqueue, approve or deploy. The original incident brief requires public API reproductions, work-count or lock-hold verdicts, no timeout increases, no test weakening, and no request-path history scans. Code fixes require red, green and unchanged-test reversal evidence. Production stays read-only.

Mission and disposition

Original mission: remove historical reclamation work from upload admission and finish committed list metadata. The 2026-10-26 symptoms were upload creation exceeding the service handler limit and lists waiting for utilization replay. The original diagnosis split these into separate mechanisms; that distinction remains supported by main source.

W3 found the upload fix already under active development on a new replacement branch. It salvages the older persisted projection and adds worker-owned reclamation. A second active branch corrects session-expiry/retention ordering. The wider BuildTest campaign has a live claim by fable-code-2026-10-23, conversation claude:3f12ec4a-6ea3-4e2d-aa2e-1a54eca360d9; its heartbeat was 04:18:47 UTC. Its top-level body does not name all the newer PRs, so the branch evidence below, rather than that claim alone, establishes overlap.

This handoff is BLOCKED on those active owners finishing qualification and on the committed-metadata boundary work. It is not completed and no PR is certified ready by W3. No implementation was duplicated. No old candidate was closed: the salvaged replacement has no PR yet, so closing the old PR now would lose the review trail before a reviewable replacement exists.

Verified findings

1. Main is 5f720d2a10b9ad20bf4f1383d2574cf6a40abf6f. In src/buildtest/embedded/BuildTestEmbeddedService.kt, admitStorageWrite at line 19089 calls reclaimStorageMargin before checking reservations after local recovery. At lines 18924-18976 reclamation holds storageReclaimLock while listing run directories, loading complete snapshots and sorting the full candidate set. The mechanism is request admission joining a history-decoding reclamation pass, and the evidence is this source plus the original matching owner/waiter dumps preserved on the evidence branches. These source observations do not assert that the same production request is currently stuck.
2. Main activeRunForPage at line 11044 calls snapshotRun and runForPublicRead. The latter at line 9223 initiates evidence repair and reloads a snapshot before utilization. The active-utilization PR does not claim complete immutable resident/terminal metadata: its body explicitly retains that boundary as queued work. The newer metadata PR covers result/recovery ownership and terminal counts, but is still being qualified and changes existing test ownership. Neither is merged. Do not mark all metadata contracts finished from green checks on the narrower candidate.
3. The old upload candidate remains open at 1d4024b683c6bbe8d27b744a52befc285b53d448. It is 19 main commits behind its merge base and has seven unique commits. Latest run https://github.com/CodexCoder21Organization/BuildTestEmbedded/actions/runs/40059425311 has one failed shard test: resumeSingleDropletTerminalPreflightStateReprovisionsInsteadOfLooping. The assertion says the authoritatively off persisted droplet was read twice without creating a replacement. The aggregate bld-build failure follows shard 11; it is not a second independent compilation failure. No exact matching open fix was identified; causation by upload changes is unproven. Full ordinary trace and related-PR analysis are saved in the W3 review.
4. The old upload PR explicitly holds approval of https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/6b97383ef55c326f30e224f8619c9cdb60f95561. Independent read-only review found ordinary fixture/classpath corrections and no weakened original predicates, timeouts, iterations or heap limits. That assessment is not approval. The maintenance-only alternative https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1466 is already CLOSED and unmerged; W3 did not close it.
5. The active replacement's pushed FINDINGS.md records reuse of the old ServiceCurrentProjection and source intents, dependency on current async deletion, and worker-only request admission. Its latest 04:17 checkpoint records 41/43 first attempts in one register, two subsequent corrections and 10/10 review-register first attempts, with reversal underway. A new UncheckedIOException directory-walk gap is being reproduced. These are owner-reported results read from pushed evidence, not W3 execution. The BTLOCK branch has recent red/green/reversal evidence and an additional active-candidate/finalizer-bookkeeping correction. Neither branch has an open PR as of 04:24 UTC.
6. Read-only https://buildtest.kotlin.build/api/health returned JSON promptly. It contains no exact engine source identity, so it does not establish either fix is deployed. W3 performed no upload, test-service submission, production mutation, artifact publication or deployment.

Relevant branches and PRs

All repositories below are CodexCoder21Organization/BuildTestEmbedded. Heads are verified remote snapshots, not a promise that the active branches will remain unchanged.

W3 findings and independent review: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fable-W3-upload-triage-20261027/evidence/W3-upload-triage-20261027

Active owner's detail: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/upload-reclaim-scan-fix1-evidence/forensics/upload-reclaim-scan-fix1/FINDINGS.md

BTLOCK detail: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/wip/BTLOCK-retention-metadata/evidence/BTLOCK/findings.md

Campaign: https://www.handoff.wasmserver.com/handoffs/hf-2026-08-24-fix-startup-compaction-delta-heap-exhaustion-then-revalidate-the-extreme-suite

Remaining steps

1. Re-verify active branches and their owners first. Allow the upload replacement owner to finish the new walk-error proof, unchanged-test reversal, exact-head targeted/full gates required by its shared contract, and independent reviews. Open that replacement PR without duplicating the old scalar projection. Coordinate the BTLOCK/server expiry fix with the same owner; its public wire proof needs both source changes.
2. Once that salvage is reviewable, reconcile and, if superseded, close the old upload candidate with a comment linking the replacement and explaining preserved tests/metadata and any remaining differences. Do not close it before the review trail is recoverable. If the old candidate is used instead, retain its existing-test approval hold and resolve the exact preflight failure with a deterministic proof rather than blind CI reruns.
3. The campaign's metadata owner must settle the two utilization proposals and complete the immutable ResidentRuns/list/terminal boundary. Require public API tests with held ingestion/result-repair dependencies and work counts over all source reads, not only event journals. The narrower active-read green checks do not prove the whole boundary.
4. The orchestrator reviews and lands qualified PRs. W3 may not approve, merge or enqueue them. After merged code is consumer-tested, publish a new immutable Embedded artifact from default-branch source and finish the service pin/release PRs. Do not republish an existing version.
5. Only with separate production authorization, deploy the resulting BuildTestServerService build to the existing url:buildtest: ContainerNursery route, confirm old writer exit as required by the README, and verify the same upload/list paths. A final artifact version cannot be selected before the active branches finish. No live recovery is claimed here.

W3 limits and cleanup

Fresh recursive clone was /code/ws/W3/workspace/BuildTestEmbedded-W3. No AGENTS.md exists in it. Main README and the linked triage/testing/philosophy/slow-service guidance were read. No test/build was run because no behavior or tests were changed, and the exact mechanism already has active owned implementation. The evidence-only branch was pushed and verified. No new implementation PR, PR closure, approval, merge, publication or deployment occurred. No stash or unpushed product work exists. Earlier unrelated W3 checkouts were left untouched to preserve their work.