← Handoffs

Repository · handoffs

BuildTestEmbedded old-PR cleanup: salvage phase (triage finished)

View on GitHub ↗

id: hf-2026-09-23-finish-triaging-and-landing-the-old-buildtestembedded-pull-requests-oldest-first-re-triage-799-806-and-the-remaining-106-queued-prs-review-and-enqueue-the-20-salvaged-prs-mid-landing url: url://handoff/handoffs/hf-2026-09-23-finish-triaging-and-landing-the-old-buildtestembedded-pull-requests-oldest-first-re-triage-799-806-and-the-remaining-106-queued-prs-review-and-enqueue-the-20-salvaged-prs-mid-landing title: Finish terminal recovery tests and open the replacement PR summary: Four original repeats, two fail-first cases, scenario/neighbor and full-suite gates, new PR and CI remain. Checkpoint 52569db4 is pushed; original reproducer passed once locally before the lane cutoff. created: 2026-09-23T18:17:23.402Z completed: null dependencies:

010b-bte475c exit — RE-VERIFY on pickup (2026-10-02 19:25:34 UTC)

Remaining first: complete four more fresh local executions of the original restart reproducer; run the two review cases fail-first and fix only after proving each failure; run fourteen scenarios plus five neighbors, the full local suite, final reviews, a NEW PR superseding https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475, and one CI watch. This delegate reached its 75-minute work cutoff; no new PR was opened. The supervisor owns merge, queue, deployment, and handoff completion decisions.

Live checkpoint

Branch Head Validation
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/010b-bte475c-2026-10-02 52569db40a4dc52a3275ce1f67b6c629ffecb84e Original reproducer 1 passed / 0 failed; all further gates remain

OBSERVED: Fresh clone resumed d562e8172a5fa207b622066a092d423904f711d2 on main b46945c73fecb72e339505538538ca637dadb02f. Added only the missing import okio.buffer in terminalProjectRecoveryPreservesDeletionDuringFeedPublication.kts. No production implementation changed by this lane. The original local restart reproducer passed once (test 6439 ms, buildMaven 1284835 ms). Its first slot wait was 21m08s. The second separate invocation waited 1164 seconds, acquired slot 2 at 19:22:34, then was canceled during workspace preparation at cutoff with no test verdict. Its JVM stack was captured. No lock files deleted, other lane processes changed, timeout increased, assertions changed, or tests disabled. Final fetch/rebase was a no-op; final push and git ls-remote verified the head above. Working tree clean. No new PR, remote run, CI submission, merge, queue action, production action, Maven publication, or completion call.

INFER: The one pass supports the existing port for the original empty-name restart failure. Source-I/O settlement and final-read peer deletion/feed ordering remain source-review hypotheses, not runtime-proven diagnoses. Do not make their fixes without running the already-pushed public tests first.

Exact next sequence

  1. Resume the branch above. Original selector: residentTerminalRunRecoversProjectNamesFromSavedArchive. It needs four more fresh successful executions to reach 5/5. Use a separate shared-slot helper acquisition for each invocation, and fetch/rebase before builds.
  2. Source case: terminalProjectRecoverySurvivesArchiveReadFailure. The current catch clears pending on any non-interrupted exception while the archive remains a regular file. The test forces one source-read failure then expects the next restart to recover names and repository links. Prove it red before distinguishing unfinished source I/O from a readable invalid archive. Keep the corrupt-archive exact warning and settlement behavior intact.
  3. Feed case: terminalProjectRecoveryPreservesDeletionDuringFeedPublication. The public filesystem callback deletes through a peer after the third recovered-record read has captured bytes. It requires a tombstone and no later RUN_UPSERT. Prove it red before changing the shared projection-lock/deletion publication boundary. A further disk reread alone does not establish peer ordering.
  4. Run all fourteen scenarios: residentTerminalRunRecoversProjectNamesFromSavedArchive; archivedTerminalProjectRecoveryPublishesFeedUpdate; archivedTerminalProjectRecoverySurvivesMissingArchiveRestart; legacyTerminalProjectRecoveryDoesNotScanArchive; residentTerminalProjectRecoverySurvivesMissingArchiveRestart; terminalProjectRecoveryIgnoresLeftoverChunks; terminalProjectRecoveryInterruptedByCloseRemainsPending; terminalProjectRecoveryMarkerPrecedesWorkerStart; terminalProjectRecoveryPreservesDeletionDuringArchiveRead; terminalProjectRecoveryPreservesDeletionDuringFeedPublication; terminalProjectRecoverySettlesCorruptArchive; terminalProjectRecoverySettlesEmptyArchive; terminalProjectRecoverySurvivesArchiveReadFailure; terminalProjectRecoverySurvivesWorkerStartFailure.
  5. Five neighbors: completedAndFailedRunsArePreservedOnRestart; archivedRunStartupRecoveryFinalizesCountsWithoutHeapResidency; uploadInterruptedRightAfterCreateUploadSessionRecoversAcrossRestart; emptyUploadedArchiveFailsWithADescriptiveMessage; testArchiveAnalysisPinsCompleteResult.
  6. Run the full local suite once after fixes, then final test/code review. Only after these gates open the NEW PR. Its opening must explain this handoff and the original PR, link url://handoff/handoffs/hf-2026-09-23-finish-triaging-and-landing-the-old-buildtestembedded-pull-requests-oldest-first-re-triage-799-806-and-the-remaining-106-queued-prs-review-and-enqueue-the-20-salvaged-prs-mid-landing and https://www.handoff.wasmserver.com, and say it supersedes the old PR by full URL. Watch one CI run. Do not update the old PR branch, merge, enqueue, deploy, or complete this handoff.

Local counts from this lane: completed tests 1 passed / 0 failed. Original requirement 1/5 successful executions. One additional invocation canceled before any verdict; two review cases, the 19-test batch, and full suite not run. No CI counts exist for this lane. Prior main failure evidence remains below and is not counted as a new local execution.

RESULT

INCOMPLETE Four original repeats, both fail-first review cases and fixes, scenario/neighbor batch, full local suite, final reviews, new PR and CI remain; shared-slot waits and compilation consumed the 75-minute work window.

All earlier mission text, queue state, branch/PR rows, findings, and previous lane history are carried forward below.

010b-bte475c milestone — RE-VERIFY on pickup (2026-10-02 19:04:58 UTC)

Remaining: four more fresh local executions of the original restart reproducer, then prove and settle the source-I/O and peer feed-publication cases; run the fourteen scenarios plus five neighbors, the full local suite, final reviews, a NEW PR superseding https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475, and one CI watch. No merge, enqueue, deployment, artifact publication, or handoff completion is authorized for the delegate.

OBSERVED: The resumed port passed residentTerminalRunRecoversProjectNamesFromSavedArchive locally once (1 passed / 0 failed; test 6439 ms). Main remains b46945c73fecb72e339505538538ca637dadb02f. The only new source edit imports okio.buffer in terminalProjectRecoveryPreservesDeletionDuringFeedPublication.kts. No production behavior was changed by this lane. The second separate local invocation is waiting for a shared compile slot. No new PR or CI run has been submitted.

INFER: This one passing execution supports the original port; it does not validate the two review cases. Those remain hypotheses until their public tests run red. Current cutoff for new work is 19:23 UTC.

Lane checkpoint Head State
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/010b-bte475c-2026-10-02 52569db40a4dc52a3275ce1f67b6c629ffecb84e Original reproducer 1/1 locally; other cases untested

All earlier branch rows, the mission, findings chain, and queue state follow unchanged.

bte475b exit — RE-VERIFY on pickup (2026-10-02 17:55:02 UTC)

Remaining first: run the two new review reproducers, fix the proven source-I/O settlement issue and any proven final-read feed-ordering issue, then obtain original reproducer 5/5, all fourteen recovery scenarios and five named neighbors, the full local shared-record gate, a NEW PR and one CI watch. Delegate work is INCOMPLETE. No PR or post-fix test success is claimed. Only https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 was assigned; its branch was never modified, and the old-PR queue below remains carried forward.

Repo Branch Remote head SHA PR What is on it State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte475b-2026-10-02 d562e8172a5fa207b622066a092d423904f711d2 no PR Initial current-main recovery port, original reproducer plus thirteen follow-up scenarios, source-close/feed-order assertions, README UNVERIFIED port; independent review gaps remain

OBSERVED: This lane freshly reproduced the original defect on unchanged main e04680a0429b5dad204adaa966fff19f7eedcd33. withlock.sh scripts/test.bash --local --test residentTerminalRunRecoversProjectNamesFromSavedArchive returned 0 passed / 1 failed, all three built-in attempts failed with java.lang.AssertionError: Expected <[recoverable]>, actual <[]> at ResidentTerminalRunRecoversProjectNamesFromSavedArchive.kt:85. Source was unchanged under e122d0980c19e3f30314619e1ba0885d8011cc56. The module compiler progressed through type resolution and bytecode, completing inside its existing deadline. Full baseline logs and thread dumps are local under supervisor out/bte475b-red.* and bte475b-baseline-jstack*. The complete prior failure chain and stack below remain preserved.

OBSERVED: Initial port persists projectAttributionPending default false, sets it before submitted worker start, settles complete name reads, shares the existing root/name resolver, and adds terminal recovery before event-log and ordinary recovery-marker skips. It uses injected Okio archive input and current record publication for resident/archived runs, preserves terminal fields, avoids archived heap promotion, and does not veto leftover chunks. Source is NOT locally verified.

OBSERVED: Independent test and code review read Testing Architecture Documentation, Engineering Philosophy, README and the invariant table. Added source-close checks for successful resident/archived, empty, corrupt and interrupted reads. Deletion-feed assertions now forbid every RUN_UPSERT after the FIRST post-cursor tombstone, because demand-driven reconciliation can append a later tombstone that would otherwise conceal wrong ordering. The missing-source zero-open concern was explicitly refuted for this port: metadata probing deliberately avoids opening absent inputs; no existing assertion was weakened.

INFER: Review found a definite source-derived gap: recoverPendingProjectAttribution currently catches a source IOException together with parser exceptions, then clears the pending marker while the archive remains a regular file. That is not proof of invalid archive bytes, and can lose a later restart recovery. New public terminalProjectRecoverySurvivesArchiveReadFailure forces one Source.read IOException while the archive exists, verifies source closure and terminal fields/counts, then requires the next restart to recover names and links. It must FAIL before changing the handling. Distinguish source-open/read/close failures (unfinished, retain/report) from parser rejection of readable invalid bytes (settle/report once). The earlier handoff phrase “unreadable files settle/report once” was ambiguous; this distinction is explicitly recorded rather than silently changing its meaning.

INFER: A narrower peer feed-order window is still unproven: archived save loads the final run record, checks its old object and the service-local deleted ID set, then queues a prebuilt upsert. A peer may delete during the addressed read’s close, leaving old captured bytes. New public terminalProjectRecoveryPreservesDeletionDuringFeedPublication captures real run.json bytes and deletes through the peer API on close of the THIRD recovered archived-record read: published record, uncached streamed fingerprint, final record. Earlier second-read reasoning was corrected by inspecting archived fingerprint ownership; no runtime proof was claimed. withRunDiskLock is per service; projectionLockFor and deleted-directory state are shared path boundaries. Run the public reproducer before choosing a fix, and account for callback re-entry and peer feed publication.

OBSERVED: The 17-test batch and subsequent two-test run were stopped while queued, before compile-slot acquisition. Both exited143 by deliberate termination, not a test verdict. Source-fix tests0 executed. Final fetch/rebase advanced the base to b46945c73fecb72e339505538538ca637dadb02f; pushed head d562e8172a5fa207b622066a092d423904f711d2 is UNTESTED. git ls-remote matches and working tree is clean. Original unchanged-source baseline0 passed/1 failed unique test, built-in0 passed/3 failed attempts. No new PR/CI or five-pass/full-suite success. Old PR remains OPEN at unchanged3f71346819aeb0af179ec8aa03552ec44456c4a9, latest commitSeptember23/commentAugust15.

Next exact local selectors (every invocation under /tmp/claude-501/-code/4719c767-3557-4cb1-90f8-28975c4c3602/scratchpad/withlock.sh, fetch/rebase before builds):

scripts/test.bash --local --test terminalProjectRecoverySurvivesArchiveReadFailure --test residentTerminalRunRecoversProjectNamesFromSavedArchive
scripts/test.bash --local --test terminalProjectRecoveryPreservesDeletionDuringFeedPublication

Then run the fourteen scenarios: original residentTerminalRunRecoversProjectNamesFromSavedArchive; archivedTerminalProjectRecoveryPublishesFeedUpdate; archivedTerminalProjectRecoverySurvivesMissingArchiveRestart; legacyTerminalProjectRecoveryDoesNotScanArchive; residentTerminalProjectRecoverySurvivesMissingArchiveRestart; terminalProjectRecoveryIgnoresLeftoverChunks; terminalProjectRecoveryInterruptedByCloseRemainsPending; terminalProjectRecoveryMarkerPrecedesWorkerStart; terminalProjectRecoveryPreservesDeletionDuringArchiveRead; terminalProjectRecoverySettlesCorruptArchive; terminalProjectRecoverySettlesEmptyArchive; terminalProjectRecoverySurvivesWorkerStartFailure; the two new review tests above. Named neighbors: completedAndFailedRunsArePreservedOnRestart, archivedRunStartupRecoveryFinalizesCountsWithoutHeapResidency, uploadInterruptedRightAfterCreateUploadSessionRecoversAcrossRestart, emptyUploadedArchiveFailsWithADescriptiveMessage, testArchiveAnalysisPinsCompleteResult. Obtain five original-reproducer passes using separate helper acquisitions; run the full local suite for the persisted-record/shared-contract gate. No five-pass, selected-batch or full-suite success exists yet.

OBSERVED: No new PR, remote/CI run, merge, queue, deployment, production restart, route change, Maven publication or handoff completion. One progress report failed with “All 0 peers for service 'handoff' failed bytecode fetch (direct and relay). Direct candidate errors: []. Relay candidate errors: []”; retry succeeded, so no production action was taken. Shared compile waits are time-box limitations, not evidence of a code defect. Source and tests must be treated as an unfinished checkpoint.

All previous Mission sections, branch/PR rows and findings follow unchanged.

bte475b analysis checkpoint — RE-VERIFY on pickup (2026-10-02 16:42 UTC)

Remaining first: finish the local baseline, implement recovery, verify 5/5 plus twelve scenarios and named neighbors, open/watch a NEW PR. Only the terminal name-recovery port is assigned to this lane; the old-PR queue state below remains carried forward.

Repo Branch Remote head SHA PR What is on it State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte475b-2026-10-02 e122d0980c19e3f30314619e1ba0885d8011cc56 no PR Prior twelve public tests plus analysis evidence commit Source unchanged; baseline compiling locally on main e04680a0429b5dad204adaa966fff19f7eedcd33

OBSERVED: old PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 remains OPEN/DIRTY, unmerged at 3f71346819aeb0af179ec8aa03552ec44456c4a9. No commit or comment from an unaccounted recent session. Prior baseline proof remains 0/1 with three failed attempts; this lane's fresh baseline obtained slot 1 at 16:38:32 and has no verdict yet. Initial clone and source study completed. Branch ls-remote matches the row above.

INFER: port must use the injected Okio archive source, rather than the old raw input stream, so close/interruption and re-entry tests exercise the real read. Submitted archive ownership gains a durable pending marker before starter callbacks. Terminal startup recovers before event-log skips. Successful/empty reads settle; missing or interrupted reads remain pending; corrupt readable archives settle and report once. Resident writes use current mutateRunAndSave; archived writes re-read disk after archive I/O and use native identity-checked publication/feed. Chunks beside a submitted archive do not veto recovery.

OBSERVED: no source fix, new PR, remote/CI run, merge, queue, deploy, restart, publication or handoff completion. Read-only test review is running. Every earlier mission, row and finding follows unchanged.

bte475 source-reviewed test checkpoint — RE-VERIFY on pickup (2026-10-02 15:34:34 UTC)

Remaining work comes first: obtain the required LOCAL fail-first result, implement the terminal project-name port, prove original reproducer 5/5 plus neighbors, then open/watch one new PR. No fix or new PR exists yet. The required shared compile lock has prevented this lane from running a test. This is a test-only checkpoint, not completed delegate work.

Repo Branch Remote head PR State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte475-2026-10-02 844f79ca0f717ad3c91501210b8530cd98ac9838 no PR Original W79 plus eleven new public scenarios, source-reviewed, unexecuted; production unchanged

OBSERVED: The source-only independent review is complete. Invalid chunk bytes discriminate the saved-archive input. Resident and archived missing/restored/repeated-restart fixtures check preserved historical counts and one archive source read after successful repair. Empty/corrupt archives also test settlement. Before-launch ordering is exercised by public cancellation and a peer restart inside the worker-start callback, before its refusal reaches failure handling. Peer deletion during archive read and interruption during close have dedicated scenarios. No reflection or private recovery calls. Original W79 file is preserved unchanged; its existing close-throws cleanup limitation is recorded, while every new draft uses nested cleanup. No runtime results are claimed by the review.

OBSERVED: First lock wait expired without test execution; second flock -w 1500 attempt is still queued. Its log is empty and started marker absent. Observed local test counts remain 0 passed / 0 failed / 0 executed. Current branch is rebased to main 74fd336c4. The earlier checkpoint row below is historical; the table above is the latest verified remote head. All previous mission, branch/PR rows, and findings remain carried forward below.

bte475 checkpoint — RE-VERIFY on pickup (2026-10-02 15:21:58 UTC)

Mission for this lane: port the terminal project-name recovery from https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 onto current main, with the W79 public reproducer preserved. The existing mission, queue history, and every prior branch/PR row below remain carried forward. This lane owns only this one port; other queue items are not re-triaged.

OBSERVED: Fresh clone in the assigned lane directory. Old PR remains OPEN at 3f71346819aeb0af179ec8aa03552ec44456c4a9; latest commit/comment dates predate this lane by more than two hours. Original W79 test is cherry-picked onto main, now rebased to 74fd336c4. Nine added public scenario drafts cover resident/chunks and archived recovery, repository links, historical counts, missing/corrupt/empty archives, legacy no-marker behavior, deletion during the read, interrupted close, and worker-start failure. Production source is unchanged. No new PR, CI submission, merge, enqueue, deployment, publication, or handoff completion.

Repo Branch Remote head PR State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte475-2026-10-02 1cf543c9cce7bf68f8ae539eb2db029c2cb1803b no PR Test-only WIP checkpoint; all ten scenarios unexecuted locally; no implementation

OBSERVED: The first local baseline attempted flock -w 1500 ... scripts/test.bash --local --test residentTerminalRunRecoversProjectNamesFromSavedArchive. The flock wait expired without obtaining the shared lock; log remained empty and no result XML was created. A second bounded local attempt is queued from the rebased head. Neither attempt has a test verdict: observed local counts are 0 passed, 0 failed, 0 executed. The W79 remote failure is historical evidence, not a local result from this lane.

INFER: Main registers empty names and makes a run terminal when both archive readers cannot open the moved archive. It has no durable pending-attribution marker or terminal startup rescan. The old fix must use current snapshot identity/publication rules, and its blanket chunks-directory guard should not hide a valid saved archive. Source changes remain conditional on the required local fail-first verdict. Independent read-only review found gaps and improved source-choice discrimination, count preservation, cleanup, and no-second-read coverage; no test result is claimed.

Next: finish the local baseline, port through current persistence rules, verify original reproducer 5/5 plus neighbors, complete independent test/code reviews, and open/watch one new PR. The supervisor alone reviews, merges/enqueues, and completes the handoff. The lane time box remains in force; if the shared compile gate cannot open, preserve the exact incomplete state here.

W79 exit checkpoint (2026-09-27 13:50 UTC; re-verify on pickup)

PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 is OPEN, DIRTY, unqueued at 3f71346819aeb0af179ec8aa03552ec44456c4a9 against main 0eb3f0e1afc5b2b53f1f098d359c8dffe5fed320. Its five green checks cover only that old head. Main has no projectAttributionPending recovery path. W79 pushed a public-API restart reproducer on https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr475-W79 at 44a01bdc1dbfc32fa85b6e20856a4ef9ea109a80. Remote run https://buildtest.kotlin.build/run?id=e07302cb executed that single test: 0 passed, 1 failed, 2775 filtered out. The exact assertion was Expected [recoverable], actual [] in residentTerminalRunRecoversProjectNamesFromSavedArchive. The test restores a saved archive after its initial analysis failed, restarts the service, then reads the terminal run through getBuildRun. On main, the fallback at BuildTestEmbeddedService.kt:6461 tries the missing archive again; loadFromDisk at line 21612 has no pending-attribution rescan. That is the mechanism. The WIP branch is a test-only checkpoint; it has no fix and no PR.

Next for PR 475: port its recovery design onto current main with the fail-first test preserved; do not push the old DIRTY PR head as-is. Before adopting its implementation, write a public test for the chunks-directory guard in recoverPendingProjectAttribution: determine whether a terminal run can retain chunks/ alongside a valid saved archive, and remove or narrow the guard if it hides recoverable names. Cover resident and archived runs, old records without a marker, missing and unreadable archives, repository links, and deletion during recovery. Run affected tests remotely, perform separate test and code reviews on the final head, rebase, verify the PR is OPEN and unqueued, then update the PR branch or open a new main PR with a reason-first description. Watch required checks; the orchestrator alone enqueues or merges.

PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 is OPEN and unqueued at fac60fda19e3f2ad9adfd1518d933c173c21e78c. Its four remaining artifact test failures and salvage branch are captured in focused handoff https://www.handoff.wasmserver.com/handoffs/hf-2026-09-27-finish-pr-471-artifact-recovery-fix-four-failed-public-tests-and-update-the-pr. PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1075 remains OPEN, DIRTY, unqueued at 5794bcd2abbdff511fca66f461bbd390509ca846 under another actor; its OS lock is not on main. Nothing was merged, enqueued, deployed, or published in W79.

W79 PR 475 test checkpoint (2026-09-27 13:38 UTC; re-verify on pickup)

PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 is OPEN, unqueued, at 3f71346819aeb0af179ec8aa03552ec44456c4a9. Its project-attribution recovery is absent from main. A public resident terminal restart reproducer, adapted from its test without persisted-field assertions, is pushed at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr475-W79 (44a01bdc1dbfc32fa85b6e20856a4ef9ea109a80). Remote command: scripts/test.bash --remote --test residentTerminalRunRecoversProjectNamesFromSavedArchive --log /tmp/pr475-W79-red.xml. It was submitted but has no test verdict at this snapshot. The old PR's chunks-directory guard appears to skip attribution recovery if that directory remains after a valid upload; prove or refute this with a public test before reusing that implementation. No PR branch was pushed or merged.

W79 PR 471 disposition (2026-09-27 13:37 UTC; re-verify on pickup)

PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 remains OPEN and unqueued at fac60fda19e3f2ad9adfd1518d933c173c21e78c. W76 salvage is durable at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W76 (23eb68c298a302c5a033369dbd338b6c65fa38c0). W79's remote run https://buildtest.kotlin.build/run?id=6769c296 executed 29 selected tests: 25 PASSED, 4 FAILED. Candidate selection now passes. The four failures prove the remaining legacy read, digest marker, read/replacement race, and required override resume defects. Local candidate selector passed 1/1. The exact failures and next steps have moved to focused handoff https://www.handoff.wasmserver.com/handoffs/hf-2026-09-27-finish-pr-471-artifact-recovery-fix-four-failed-public-tests-and-update-the-pr. PR 471 was not updated. Continue this old-PR board at PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475, oldest after PR 471. The orchestrator alone enqueues or merges.

W76 exit checkpoint 2026-09-27 12:32 UTC (re-verify on pickup)

Only PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 was worked in this lane. It is still OPEN, unqueued, and untouched at fac60fda19e3f2ad9adfd1518d933c173c21e78c; its old-head checks do not validate the salvage. Main is 0eb3f0e1afc5b2b53f1f098d359c8dffe5fed320. PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1075 is OPEN, unqueued, DIRTY at 5794bcd2abbdff511fca66f461bbd390509ca846 and another actor owns it; its OS lock has not reached main.

Repo Branch Remote head PR State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W76 23eb68c298a302c5a033369dbd338b6c65fa38c0 no PR Five new public tests plus a candidate-selection fix rebased on main; fix is not yet tested
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/restart-preserves-run-outcome fac60fda19e3f2ad9adfd1518d933c173c21e78c https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 old PR head, still open and untouched

The mechanism proved in W76: artifact.tar.gz exists without a valid completion record, while artifact.bin has a valid record; committedRequestedArtifactPath picked tar.gz by existence first and recovery wrongly marked a COMPLETING run FAILED. The public test testCommittedBinPreferredOverUnverifiedTarGz failed first at Expected COMPLETED, actual FAILED. The code now checks the two candidates separately, preferring a verified tar.gz then verified bin, but there is no fixed-head test result. Local command: jvm-slot.sh scripts/test.bash --local --test testCompletedLegacyArtifactWithoutRecordRemainsReadable --test testCommittedBinPreferredOverUnverifiedTarGz --test testArtifactDigestRejectsSameSizeReplacement --log /tmp/pr471-W76-local-failfirst.xml. The build rule hit its 20-minute deadline; only the candidate test executed, with the intended assertion failure. Three remote batches ended at their 30-minute client timeout without executing selected assertions: run 17c401df BUILDING, 8e25d3d0 PENDING, eb26f44a BUILDING. Read-only host inspection confirmed 17c401df's selected tests were canceled, not failed by their assertions. The timeout was not raised and no CI was used for debugging. BuildTestRunner 0.0.114 POM returned HTTP 200. No artifact was published or production system changed.

Next: obtain a fixed-head verdict for the candidate test; then obtain fail-first verdicts for the other four W76 tests before implementing their fixes. Historical COMPLETED files with no marker must remain readable; a present but mismatching record must refuse reads. The marker needs a SHA-256 digest with parsing of old size-only records, and reads must use the validated open file rather than reopening its path. Resume with a mismatched override source must fail the run with a descriptive error. Once PR 1075 merges, use its per-run OS lock for artifact publication and a forked child-JVM public test; do not duplicate its active work. After all cases pass: run affected family remotely, rebase, update PR 471 only if OPEN and unqueued, do two final-head reviews, and watch CI green. Orchestrator alone enqueues or merges. The old PR board follows after PR 471.

W73 exit checkpoint 2026-09-27 10:26 UTC (re-verify on pickup)

Scope this lane to PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 only. Main advanced through PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1074 to 9d7ff80568acc355ab10718eaf618a3262aff536. W73 rebased cleanly on that head. PR 471 was OPEN, not queued, and untouched at fac60fda19e3f2ad9adfd1518d933c173c21e78c; its five green checks apply only to that old head. PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1075 is OPEN at be18124eea4b5ecc6c567def3b839cd079455ca9, out of the merge queue and DIRTY after main advanced. Its per-run .upload.lock OS protocol has not reached main; another actor owns that PR. BuildTestRunner 0.0.114 POM returned HTTP 200.

W73 reproduced the public-read bug with a revised public restart test: before the fix, a remote targeted run executed and failed 1/1 as the unverified old artifact remained readable after recovery marked the run FAILED (buildtest run c7bcc321). The first remote attempt failed before testing on a getBuildRun RPC timeout and has no test verdict. W73 also changed the real-worker publication-boundary test to demand unreadable bytes until the completion record commits. WIP code now gates whole, metadata, chunk, stream, and override-JAR artifact reads on a matching completion-size record. Existing direct-file test fixtures gained matching records; README documents the strict contract, including that old completed runs without records become unreadable. Remote focused run https://buildtest.kotlin.build/run?id=20ccf4aa passed 3/3 selected public tests from pre-rebase W73 head cdfe25db. The same three selected tests passed 3/3 locally from rebased head 7a5a89e3 under jvm-slot.sh. An earlier baseline local attempt waited over 20 minutes for a shared JVM slot and was canceled before test execution. No broad suite or PR CI ran on W73.

Repo Branch Remote head PR State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W73 7a5a89e3a2f420b7c39fa7d70c0f3138ad17560c no PR read-path fix and tests pushed; pre-rebase remote and rebased local focused tests each passed 3/3; cross-process and broad verification pending
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W69 bc5fcb7e8e986ffe1b73647c796e70d8e820c39d no PR preceding recovery work; 8/8 focused local and 1/1 remote before W73 changes
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/restart-preserves-run-outcome fac60fda19e3f2ad9adfd1518d933c173c21e78c https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 old PR head; do not count its green checks for W73

Next: the read-path focused tests passed, but do not update PR 471 yet. Do a public cross-process artifact-publication test after PR 1075's shared OS lock reaches main; reuse that protocol, do not cherry-pick the active PR. Review read/write interleaving, same-size marker match, and stale tar.gz beside valid bin. Add a public resume test for an unverified override JAR source: the existing resume path warns and continues without injecting the requested JAR, so it needs a tested failure path. Add a public test for a stale unverified tar.gz beside a valid bin, and check the documented behavior of an old COMPLETED run without a marker. Then run affected tests and broad suite, rebase immediately before updating PR 471, do separate test and code reviews, and watch required CI green. The orchestrator alone enqueues/merges. No production deploy or new Maven publish occurred in W73.

Exit update 2026-09-27 09:13 UTC (W69; re-verify on pickup)

  • WIP https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W69 is pushed at bc5fcb7e8e986ffe1b73647c796e70d8e820c39d and clean. The old PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 remains untouched at fac60fda19e3f2ad9adfd1518d933c173c21e78c; its five green checks apply only to that old head.
  • Final focused result on WIP: 8/8 local tests passed, plus one of those 1/1 remotely. The additional completing-missing-artifact neighbor that was running in the prior note passed. See W69 findings.md for exact selectors and review findings. No broad suite or CI ran on this head.
  • CHECKPOINT, not READY: final-head test review and code review each found ownership cases still needing proof or a contract decision: cross-process publication overlap, close during marker publication, and whether artifact reads should expose an unverified old final file after recovery marks the run FAILED. Do not update PR 471 or enqueue until resolved; then run broad tests and CI. Continue remaining 32 open PRs oldest-first afterward. Orchestrator alone enqueues or merges.

Update 2026-09-27 09:12 UTC (W69 checkpoint; re-verify on pickup)

  • Oldest PR https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 remains OPEN at fac60fda19e3f2ad9adfd1518d933c173c21e78c; five checks SUCCESS on that old head. Latest batch GraphQL merge state was UNKNOWN, and no other actor updated it. Current main 28ab3c3deb81df9a7cf33d8041045c3296a5c41d. Original PR branch untouched.
  • Durable new work: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W69 at bc5fcb7e8e986ffe1b73647c796e70d8e820c39d, rebased on current main and clean. It adds a saved-artifact completion-size record after atomic publication, unique temp names, tests for old partial final files and mismatched records, a marker-publication boundary test using a real in-process fake worker, a positive committed-artifact restart test, and README contract. No CI exists on this WIP branch. The old PR's checks do not cover it.
  • Fail-first: testCompletingRestartRejectsUnverifiedLegacyArtifact failed 0/1 on the W63 code at Expected FAILED, actual COMPLETED. It passed 1/1 remotely and 1/1 locally on the fixed source. Six neighboring local targeted tests all passed 1/1 each: testCompletingRestartAcceptsCommittedArtifact, testCompletingRestartRejectsTruncatedCommittedArtifact, e2eRestartWaitsForArtifactPublicationRecord, e2eRestartBeforeRequestedArtifactPersistencePreservesArtifact, testDestroyingRestartFailsWhenRequestedArtifactIsMissing, testPostCollectionBackstopPublishesFailedRequestedOutputWithoutArtifact. Total fixed-head targeted verdicts so far: 7/7 local and 1/1 remote. One additional completing-missing-artifact neighbor was running at this snapshot.
  • Review pass A: the current tests cover missing, partial, truncated, positive, marker-boundary, and saved-failure outcomes through public APIs; they do not yet cover close during the marker move or a positive legacy completed read. Review pass B: the marker is size-only, old unverified final bytes remain visible through artifact reads even when the recovered run is FAILED, unique temp files can remain after a crash, and cross-process publication interleaving is not exercised. A same-JVM overlap experiment could not reach a second artifact move because the per-run lock serialized it; the second instance also met a projection-journal conflict under separate active work. These are remaining review questions, not tested fixes. Do not mark PR 471 READY until resolved.
  • NEXT: run the remaining focused test, decide the public-read and cross-process ownership contracts with public tests, fix any confirmed issue, run the required broad suite remotely (never locally on this shared host), update the existing PR branch only after verifying OPEN/out of queue and rebasing on latest main, perform final-head review passes, and watch CI green. Then continue the other 32 open PRs oldest-first. Orchestrator alone enqueues or merges.

Update 2026-09-27 08:48 UTC (W69; re-verify on pickup)

  • Oldest open PR remains https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471, OPEN/CLEAN at fac60fda19e3f2ad9adfd1518d933c173c21e78c with five green checks on the old head. All 47 linked BuildTestEmbedded PRs were rechecked via GraphQL: 33 OPEN, 11 MERGED, 3 CLOSED; no open PR head changed since W63. Current main was 28ab3c3de at the most recent rebase.
  • Durable WIP: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W69 at deadc5ba1f63efffda5e93137a84564a84713a98. This includes W63's artifact recovery fix, a public legacy-partial regression, a completion-size record written after artifact publication, a truncated-record regression, unique temporary names, and README contract. The original PR branch is untouched. The new legacy-partial test failed on W63 code with Expected FAILED, actual COMPLETED (0/1). Fixed-head verification is still running; this is not READY.
  • An attempted two-service artifact overlap experiment could not reach the second move while the first held the per-run projection lock. Its second instance also encountered the separate concurrent projection-journal conflict being handled by another PR. The experiment was moved out of the source tree and is not a passing test. Do not claim this refutes cross-process overlap; finish the ownership review.
  • A local fixed-head test failed before executing because a kompile build child JVM did not connect within its 10000ms startup ceiling. Remote targeted verification connected to url://buildtest/ but had not returned at this snapshot; one local retry was queued under the shared JVM slot. Neither is a test verdict. The report-challenge CLI is prohibited here because it auto-enqueues and merges a PR.
  • NEXT: Obtain fixed-head targeted results, run neighboring tests, inspect the marker and overlap contract, perform separate test and code review passes, then update PR 471 only after checking its live state and rebasing on main. The orchestrator alone enqueues/merges. The remaining 32 open PRs follow oldest-first.

Update 2026-09-27 07:32 UTC (W63; newer than every section below; RE-VERIFY on pickup)

  • Live GraphQL checked every 46 BuildTestEmbedded PR links in this handoff: 33 OPEN, 10 MERGED, 3 CLOSED. The previously pending https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/581, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/587, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/775, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/788, and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1052 have merged. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1070 also merged. The old merge-wait notes below are stale. No referenced open PR was updated within the last few hours at this snapshot.
  • Oldest open candidate: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 remains OPEN at fac60fda19e3f2ad9adfd1518d933c173c21e78c, with five green GitHub Actions checks on its old head. W63 rebased its content onto origin/main 5125d3ba791a2790133f35b765d7d38aeab57f5b and added a DESTROYING restart regression plus README contract. Durable WIP: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/pr471-W63 at 2fc913afcb6b19f35ef8309d4185cf3924e72046. The original PR branch is untouched.
  • Verified mechanism: current main's terminal-evidence finalization and COMPLETING/DESTROYING restart fallback allow COMPLETED from passing test evidence without checking that a requested artifact was published. saveArtifact on main writes directly to the final filename. The existing PR test testCompletingRestartFailsWhenRequestedArtifactIsMissing fails on main with 0/1 passing and Expected <FAILED>, actual <COMPLETED> at line 94. Command from a current-main worktree with that test copied in: jvm-slot.sh scripts/test.bash --local --test testCompletingRestartFailsWhenRequestedArtifactIsMissing. This proves the bug is still live. The new DESTROYING test also fails on main with 0/1 passed at the same public status assertion. Both COMPLETING and DESTROYING tests pass 1/1 each on the rebased WIP head using local targeted commands. The COLLECTING and saved-test-failure neighbor tests each passed 1/1 locally on the WIP head. Total WIP targeted result: 4/4 passed; main fail-first result: 0/2 passed at the intended public status assertion.
  • NEXT: use a public-API test with two overlapping service instances and ordered artifact writes to decide whether the fixed temporary filename collides. Separately test a legacy partial final artifact left by the old direct-write path and decide what durable evidence can distinguish it from a complete artifact. Then fix at the service layer, rerun targeted and full required suites, perform fresh test/code review on the final head, and update the existing PR branch only after verifying it is still open and not queued. The orchestrator alone decides whether to enqueue or merge.
  • Review question before READY: the PR writes a fixed artifact.bin.tmp/artifact.tar.gz.tmp path. Two overlapping services can own one run directory; test whether concurrent artifact writers can collide through the public API. The existing tests/deletionWorkerPreservesAnotherInstanceTemporaryIntent.kts uses a unique temporary filename for overlapping instances. One deterministic experiment is to start two service instances on the same saved COLLECTING run and fake worker, block the first artifact atomicMove through an injected ForwardingFileSystem, let the second writer reach that path, then release both and inspect public run/artifact APIs. Do not assume the code needs a change without a failing test. A second review case is a partial final artifact.bin left by main older direct-write code: an existence-only check cannot prove it complete without expected-size or digest evidence; establish the desired recovery contract before changing code.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/850 is OPEN at afd18abe0, five checks green, DIRTY against current main. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/795 is OPEN at b826aa8e0, five checks green, DIRTY. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 is OPEN at 3c21c28ed, five checks green, DIRTY. The handoff heads for 850 and 795 below are stale. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1080 and https://github.com/CodexCoder21Organization/BuildTestRunner/pull/116 remain OPEN. BuildTestRunner 0.0.114 is published (POM HTTP 200 at kotlin.directory).
  • The remote targeted test connected to url://buildtest/ but returned no result in more than 20 minutes; its client was interrupted. The public health endpoint responded. Local test commands are serialized by the shared two-slot JVM lock and were queued. This was a verification delay; the four status test results above were obtained with the local JVM-slot runner. All four touched targeted tests have passed locally on the rebased WIP branch.

Open PR snapshot (GitHub returned UNKNOWN for many batch merge statuses; use a fresh per-PR query before acting):

PR State Head Check rollup Merge status
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471 OPEN fac60fda19e3 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 OPEN 3f71346819ae SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/486 OPEN 64a20ebbaefb SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/583 OPEN 572025cd8b92 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/698 OPEN 6c8bb65b932e SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/704 OPEN c98039819568 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/759 OPEN 246e6c61340d SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/766 OPEN 6281a2b235df SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/768 OPEN 45d64d96c172 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/774 OPEN fb772cbecfb2 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/783 OPEN 1807c957b762 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/790 OPEN a1b92043d931 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/795 OPEN b826aa8e01d1 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/808 OPEN b8b0ef63cfa8 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/810 OPEN 8328092b1c8d FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/812 OPEN a5ecfcb21d2e FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/814 OPEN e6d806a15d54 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/815 OPEN 1b205190f911 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/819 OPEN b33c162123dc FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/823 OPEN fe17fb5b1cc5 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/825 OPEN b6ac0d2098ab FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/826 OPEN c83069bd8b88 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/845 OPEN 75edc8658bed FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/850 OPEN afd18abe0705 SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/922 OPEN aa1329dfda8a FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/932 OPEN ceb7dd74d97e SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/947 OPEN ffd57cefe8e6 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/950 OPEN a6b2f2bed71a SUCCESS DIRTY
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/961 OPEN 99be23bf815c SUCCESS UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/962 OPEN 31ac310fbff6 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/969 OPEN 7b61caca944d FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1032 OPEN de46f4eed7e1 FAILURE UNKNOWN
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041 OPEN 3c21c28ed96b SUCCESS UNKNOWN

Update 2026-09-24 10:40 UTC (lane 6c; RE-VERIFY, newer than everything below)

  • MERGED: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/775 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/788 (09:56 UTC, via merge queue). 581 merged earlier.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/587: evicted at 09:56 on a build.kts coordinate conflict with #775; rebased, coordinate 0.0.69240835, head ebed6b293, CI all green, 11/11 touched tests pass locally. Waiting for the orchestrator to re-enqueue. Any later landing will conflict on the coordinate line again: rebase, next unpublished patch, diff otherwise unchanged.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/850: the branch was replaced with a port onto main (head 2b516a76d, coordinate 0.0.69240836). Two review rounds are fixed. Fail-first: 0/7 on main, three failing with the production "one projection change must fit" error. 10/10 pass on the head. CI is running. Remaining: CI green, a third review of the second-round fix delta, then the orchestrator's review.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/795: reworked after two review rounds (freshness check, a cold-run snapshot commit, discriminating stress tests). Head 8ce73466e, coordinate 0.0.69240837. Two tests fail on main; 8/8 pass on the head. CI is running. Remaining: CI green, a third review of the second-round fix delta, then the orchestrator's review.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1015 was CLOSED 2026-09-24 06:05 by another actor as superseded. The work continues in hf-2026-09-16-finish-the-terminal-row-test-proof-and-diagnose-restart-retries. It is not this handoff's any more.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041: untouched. It is LARGE: 47 files, the last review (09-23) was REQUEST CHANGES with P1 "source failure scoped run-wide instead of per dependent spec", and it is conflicting.
  • New flake seen on the 587 CI: partialJournalRecoverySurvivesCloseAndRepeatedRestart (tests/...kts:106 asserts the pending-deletions marker is gone). Suspected mechanism, not yet confirmed: the deleted latch counts down inside the provider delete, before the service removes the marker. It needs its own fix-flakey-test lane.
  • Follow-up (unchanged, needs its own PR with a fail-first test): dynamic-worker replacement re-admission can leave a stale failed-droplet deletion hold when a worker exits the deletion wait early.
  • Everything else in the body below (paused fix rounds 783/704/486/748-follow-up/790, unreviewed 471/475/583/698/759/766/768/774, the salvage backlog) is unchanged by this lane.

Update 2026-09-24 05:55 UTC (orchestrator, after the sandbox recycle stopped the working lane)

RE-VERIFY before acting; this section is newer than the body below it.

  • Orchestrator decisions applied: approved 587, 581 (in merge queue at a015667e), 785, 791, 780, 1052 (all merged except 581/587). 587 conflicts again on the build.kts coordinate: rebase after 581 lands, fresh unpublished coordinate, diff must equal 48e96ba9 except the coordinate.
  • Salvage lane 6b progress: PR 775 fix pushed (head e5d47991, BEHIND main). Its review found 3 tests constructing the internal ProjectionChangeJournal directly; rewrite them through BuildTestEmbeddedService.getProjectionChanges before READY. A local cold compile of 775 timed out at 37 min under host memory pressure (not test evidence). 788 fix on wip/lane6-pr788-fix eca00096 untested. Salvage 850 does not rebase (BTES.kt conflict with main's sourceFingerprint changes): port it, do not rebase.

id: hf-2026-09-23-finish-triaging-and-landing-the-old-buildtestembedded-pull-requests-oldest-first-re-triage-799-806-and-the-remaining-106-queued-prs-review-and-enqueue-the-20-salvaged-prs-mid-landing url: url://handoff/handoffs/hf-2026-09-23-finish-triaging-and-landing-the-old-buildtestembedded-pull-requests-oldest-first-re-triage-799-806-and-the-remaining-106-queued-prs-review-and-enqueue-the-20-salvaged-prs-mid-landing title: BuildTestEmbedded old-PR cleanup: salvage phase (triage done): land 581/587/1052, finish 6 paused fix rounds, salvage 850/1041/1015/795 first summary: Triage of all old PRs below 1071 is finished (89 closed with evidence on 2026-09-24). 43 remain, each listed in the body: 6 landing (orchestrator enqueues), 6 paused fix rounds on wip/lane6-* branches, 8 not yet reviewed, 21 salvage items led by 850/1041/1015/795, 845 held, 1070 active elsewhere. Blocked on remote buildtest RPC saturation. Ledger: PlanRepository wip/handoff-20260923-buildtestembedded-pr-cleanup. created: 2026-09-23T18:17:23.402Z completed: null dependencies:

BuildTestEmbedded old-PR cleanup: salvage phase (triage finished)

Updated: 2026-09-24 ~04:45 UTC by fable-opus-lane6, which released its claim. RE-VERIFY every state below before acting.

Mission (user's request, verbatim)

Let's work through all the old pull requests in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pulls - most of them are probably obsolete and should just be closed. If any of them have fixes worth salvaging, we should clean them up and get them merged. Unit tests are often salvageable. Delegate heavy lifting to codex/gpt, do code reviews and merge decisions ourselves. Drive all the pull requests to some sort of conclusion, starting with the oldest ones first. Parallelize (6 lanes) where each lane handles a pull request to completion before triaging the next one (work queue, not waves)

State

  • Triage is FINISHED for every old PR below 1071. 89 PRs were closed with evidence comments on 2026-09-24 (see closed.tsv).
  • 43 PRs below 1071 remain open, all accounted for below. PRs numbered 1071 or higher belong to a concurrent session: do not touch them.
  • Ledger: https://github.com/CodexCoder21Organization/PlanRepository/blob/wip/handoff-20260923-buildtestembedded-pr-cleanup/handoffs/artifacts/2026-09-23-buildtestembedded-pr-cleanup/lane6-20260924.md (branch wip/handoff-20260923-buildtestembedded-pr-cleanup; not a PR, never merge it). Salvage detail: https://github.com/CodexCoder21Organization/PlanRepository/blob/wip/handoff-20260923-buildtestembedded-pr-cleanup/handoffs/artifacts/2026-09-23-buildtestembedded-pr-cleanup/salvage-backlog-lane6.md. Per-PR triage and review reports: out/lane6-20260924/ on the same branch.
  • Nothing has been deployed or published.

Landing (the orchestrator enqueues; lanes never enqueue or merge)

  • Already enqueued by the orchestrator: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/785, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/791, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/780. Check that they merged.
  • Reviewed clean and re-coordinated, waiting for CI then orchestrator enqueue: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/581 at 43ea6dee (coordinate 0.0.69240101) and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/587 at 02f017e9 (0.0.69240102). Whichever lands second will conflict on the build.kts coordinate: rebase it, choose the next unpublished patch after main's value (check the POM 404 on kotlin.directory), keep the comment line, and confirm the rest of the diff is unchanged.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1052 (removes 44 committed delegate note files and adds .gitignore entries) rebased and pushed at 89561fd3. Lane review is clean (nit: the repo-wide *-findings.md / *-handoff.md patterns could hide a future legitimate doc). CI running, then orchestrator review and enqueue.

Paused fix rounds (all blocked until remote buildtest recovers; see Blocker)

Each branch holds untested work on top of the PR head. Finish it, verify with targeted runs, then push to the PR branch with --force-with-lease against the PR head shown.

PR WIP branch @ commit Work Remaining
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/775 wip/lane6-pr775-fix @ e162eafb Fixture fix: old-wave creates register an accepted droplet before blocking, so the requeue cleanup wait resolves. The PR's src fix (clearing the stale deletion hold on whole-run requeue) is still needed on main. 5 targeted runs of e2eWholeRunRequeueClearsFailedDropletAdmissionHold; mutation check (revert src, test must fail at readmittedCreateEntered); rebase; push (lease e5d4799); PATCH body
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/788 wip/lane6-pr788-fix @ eca00096 (rebased) Fixture fix: since #873 main skips the activation poll for droplets created already active, so the injected boot failure never fired. The target create now returns status=new with no IP. Independent of 775. 5 passing runs of e2eEligibleNeighborProvisionsBesideStaleFailedDeletionHold; push to wip/r44-status-invariant (lease 1679debf); PATCH body (draft body788.md in out/lane6-20260924)
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/783 wip/lane6-pr783-fix @ 13c2ff42 Rescope after a BLOCKING review (returning on exit-status can truncate output OpenSSH sends after exit-status): dropped that change, kept callback-exception propagation, added a JSch "session is down" callback test. fail-first run; widen the onLine catch to Exception; neighbour tests; push (lease 1807c95); retitle + body
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/704 wip/lane6-pr704-fix @ 385b249d Warm-up reads remove ordering bias in the allocation test. fail-first (src reverted) + 3 passes; rebase; push (lease c980398)
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/486 wip/lane6-pr486-fix @ bab9977b Review items: deploy-transition test, full-text assertion, suppressed cleanup diagnostic, no Throwable catch, release reserve on two startup paths; coordinate 0.0.69231612. Remove the temporary DIAG printlns; make renewal outcomes exact; fail-first; run batch; re-pick the coordinate; push (lease 64a20eb)
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/748 follow-up wip/lane6-pr748-fix @ 69d85517 PR 748 already MERGED (e0fbb8f). Its review nits: restore the repurposed test's ordinary path, put the new scenario in its own file, bump the generation, addSuppressed in cleanup. Open as a NEW PR from main after running it
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/790 none BLOCKING review: the new test reads commandStarts before the first channel's Command.start runs. Add a CountDownLatch(2) counted in Command.start and await it before the exact-equality assert

Not yet independently reviewed this pass: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/471, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475 (a known fix round first: drop the chunks-dir guard in recoverPendingProjectAttribution plus a regression test), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/583, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/698, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/759, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/766 (CONFLICTING; design fix; partial branch wip/handoff-20260923-pr766-astra-partial), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/768 (CONFLICTING), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/774.

Salvage backlog (order of value)

Real bugs still present on main:

  1. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/850: one TEST_STATUS_UPSERT larger than the 983,040-byte page budget stalls the projection feed ("one projection change must fit", ProjectionChangeJournal.kt ~2710; 64 production errors on 2026-09-02). Keep ProjectionChangeRecordBounds.kt and its 4 tests. Keep main's sourceFingerprint parameter. Justify or drop the 900 KiB to 4 MiB startup record raise.
  2. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1041: an unbuildable workspace is re-leased until the 60-minute cap and then reported with buildSucceeded=true. LARGE: conflicts in the service and LeaseScheduler. BuildTestRunner PR 116 is still open although runner 0.0.114 is published.
  3. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1015: a terminal run summary can publish before its canonical test rows (run ed55d648 showed COMPLETED with RUNNING rows; main's own plan requires rows before the summary). LARGE re-port.
  4. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/795 (preferred, small, rebases cleanly) or https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/962 (large, the same bug): getBuildRun on an active run rewrites and re-parses the full test-results.json. Settle whether the cached in-memory fold can lag the event log. Smaller: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/969 (close() hang; replace its two swallowed-exception catches), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/815, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/947, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/961, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/812, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1032 (reproduce first), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/922 (test fix plus a check against #979), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/819, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/826, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/825, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/823, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/814 (must keep its #570 assertions), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/810, https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/808 (child probe only), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/932 (caveat: it drives the reconciler manually, the same reason 805 was closed), https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/950 (large). Every item needs a failing test on current main before its fix, then targeted remote runs.

Owner design items (PRs closed; the problem is still on main)

  • Capacity idles behind a blocked queue head (closed https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/966; main's admission is FIFO by tested contract).
  • Abandoned client runs keep holding droplets (closed https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1029; main documents queue activity as authoritative liveness).

Left open on purpose

  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/845: held until https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1080 (concurrent session) resolves.
  • https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1070: actively worked by another session (updated 2026-09-24 04:27; waiting on verification run 39f267ad). Leave it alone. If it goes quiet for more than a day, it is a small, clean salvage.

Blocker

Remote buildtest verification failed before executing any test on every attempt from 02:10 to 03:30 UTC: the url://buildtest/ health and getBuildRun RPCs timed out at 30 s while the web UI /api/health answered in 1.2 s. Recorded at https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-09-24-0313-remote-buildtest-url-buildtest-rpc-path-saturated-test.md. Check buildtest health before starting any fix round. Local runs are limited by the shared 10 GiB sandbox (one JVM per lane).

Operating notes

  • The shared GitHub REST budget ran out repeatedly. Use GraphQL: pullRequest queries; addComment, closePullRequest and updatePullRequest mutations; statusCheckRollup for CI.
  • Clone with plain git and --reference to a local clone to save API calls.
  • /usr/local/bin/coursier is the wrong architecture here; use ~/.local/bin (JVM wrapper).

Preserved preceding snapshot (bte475b-latest-handoff.txt)

bte475 exit — RE-VERIFY on pickup (2026-10-02 16:04:04 UTC)

Remaining first: implement the terminal project-name recovery port, make the original reproducer pass 5/5, run the eleven drafted scenarios and named neighbors, complete independent implementation review, then open and watch one NEW PR. Do not push the old PR branch. Delegate work is INCOMPLETE; no production fix or new PR exists. The local fail-first requirement is now satisfied on the exact earlier main/head listed below. The final rebased checkpoint has not been tested.

The existing Mission, every earlier branch/PR row, and the complete prior finding chain below are preserved. This lane handled only https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/475; remaining queue items were not re-triaged.

Repo Branch Remote head PR State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/bte475-2026-10-02 08e381aab546a582143315e38cab08634e860cbc no PR Test-only checkpoint, rebased to e04680a0429b5dad204adaa966fff19f7eedcd33; current head untested; production diff empty

OBSERVED: scripts/test.bash --local --test residentTerminalRunRecoversProjectNamesFromSavedArchive --log .../bte475-red-second.xml, under the required shared flock, exited 1. Unique-test summary: 0 passed, 1 failed. Its built-in attempt history contains 0 passed, 3 failed attempts, all with Expected <[recoverable]>, actual <[]>. No CI was used to diagnose it. The tested tree was 844f79ca0f717ad3c91501210b8530cd98ac9838 on main 74fd336c421677e0be6f3279808219423db9054d. The module build succeeded in 689601 ms. The first local attempt expired its 1500-second lock wait without execution; the second acquired the lock at 15:41:21 UTC and completed around 15:58. No third attempt was submitted. All lane JVMs ended normally with the failed test run; the shared lock was released.

OBSERVED: The test moved the saved archive before analysis, both initial readers failed, the public run became FAILED, then the archive was restored and the service restarted. Startup logged that terminal status skipped remote execution recovery, and the terminal name list stayed empty. This confirms the handoff's stated mechanism on main: the unfinished name read has no durable pending marker and terminal startup recovery does not revisit it.

OBSERVED: Old PR is still OPEN at 3f71346819aeb0af179ec8aa03552ec44456c4a9, merge state UNKNOWN, unmerged. Newest commit is 2026-09-23T17:20:25Z and newest comment 2026-08-15T16:03:48Z, so no unaccounted recent actor was found. Final rebase moved the checkpoint onto main e04680a0429b5dad204adaa966fff19f7eedcd33 (repeated finalizeUpload behavior); the earlier red result is not claimed for this rebased head. git ls-remote matches 08e381aab546a582143315e38cab08634e860cbc; working tree is clean.

Twelve public scenarios are preserved: original W79 plus resident/archived missing-restore and settlement, invalid leftover chunks beside a valid archive, historical count preservation, archived feed publication, legacy no-marker behavior, corrupt/empty settlement, deletion during read, interrupted close, worker-start refusal, and public peer recovery before the starter returns. Independent gpt-6-luna source-only review completed; the eleven new scenarios have not compiled or executed. Original W79 is unchanged; every new draft uses nested resource cleanup. Reviews are evidence about test design, not runtime proof.

Implementation plan already scoped: add an internal persisted pending-attribution field defaulting false for legacy rows; set it durably only at submitted-archive ownership before worker launch; settle successful empty and nonempty reads; use the existing project-identity resolver so roots keep repository links; add terminal recovery independent of event-log presence. Missing files stay pending, unreadable files settle/report once, and interruption/close preserves pending state. The explicit submitted marker should govern recovery, rather than vetoing a saved archive because chunks remain. Resident writes must use current mutateRunAndSave; archived writes must reload current disk state and use main's identity-checked native snapshot publication plus feed update, without re-admission. Recheck deletion and close after the archive read.

Named neighbors for the next targeted run: completedAndFailedRunsArePreservedOnRestart, archivedRunStartupRecoveryFinalizesCountsWithoutHeapResidency, uploadInterruptedRightAfterCreateUploadSessionRecoversAcrossRestart, emptyUploadedArchiveFailsWithADescriptiveMessage, testArchiveAnalysisPinsCompleteResult. Prefer local selectors under the shared lock. No timeout/iteration changes, test weakening, reflection, merge, enqueue, deploy, publication or handoff completion is authorized. Supervisor owns final review and landing.

Full local failure stack:

java.lang.AssertionError: Expected <[recoverable]>, actual <[]>.
	at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16)
	at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:652)
	at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11)
	at kotlin.test.Asserter$DefaultImpls.assertEquals(Assertions.kt:671)
	at kotlin.test.DefaultAsserter.assertEquals(DefaultAsserter.kt:11)
	at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:63)
	at kotlin.test.AssertionsKt.assertEquals(Unknown Source)
	at kotlin.test.AssertionsKt__AssertionsKt.assertEquals$default(Assertions.kt:62)
	at kotlin.test.AssertionsKt.assertEquals$default(Unknown Source)
	at buildtest.embedded.ResidentTerminalRunRecoversProjectNamesFromSavedArchiveKt.residentTerminalRunRecoversProjectNamesFromSavedArchive(residentTerminalRunRecoversProjectNamesFromSavedArchive.kt:85)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at community.kotlin.kompile.testrunner.bootstrap.BootstrapRunner.main(BootstrapRunner.java:416)

No new PR, no remote/CI run, no implementation review, no merge/enqueue/deploy/publish/handoff completion occurred. The original task's time box leaves these items incomplete; this is not a claim that the port is complete.