Handoff: Investigate the round-five cache probe timeouts and finish the required BuildTestEmbedded gates

Written 2026-10-11 07:33 UTC. RE-VERIFY: this is a write-time snapshot. Check the PR state/head with gh pr view https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1421 --json state,isDraft,headRefOid,mergedAt; check source refs with git ls-remote origin refs/heads/wip/fleetfix1-fix refs/heads/wip/fleetfix5-evidence. CI monitoring belongs to the orchestrator, and was not performed after this round's push. Check run statuses through https://buildtest.kotlin.build/api/runs and native rows through https://buildtest.kotlin.build/api/test-results?id=e8ac36aa&page=1 (pages start at1; read all43pages).

Mission and constraints

The user asked round5 to “close the independent review's two SHOULD-FIX findings and push ONE new head.” Original source was0c286f1711fd07c4e3b810f4158368eb7d863a67 on wip/fleetfix1-fix. The two issues were premature deletion of a released activating/blank-IP follower with pending work, and omission of the original prepared workspace cache on reassignment. The independent review is https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/bddebc24f072296bfefd5096817ce5b5e010e024/investigations/rev1421a/evidence/review-findings.md . Both full review reports and round4 findings were read.

Binding limits: at most2remote runs in flight, at most6submissions per round; this round used6/6, so no further submission under this round. No merge, queue entry, deploy, artifact release, existing-test assertion edits, timeout increases, iteration reductions, artificial load or CI reruns. The existing non-draft PR state remains unchanged. One new source head has already been pushed this round. Future work needs an orchestrator-directed new round. Local named ownership/scenario gates used shared slots under the task's explicit allowance; no local full suite or broad neighbor sweep. Required documentation: https://github.com/CodexCoder21Organization/BuildTestEmbedded/blob/main/docs/UTILIZATION_ARCHITECTURE.md , https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md , https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md . All were read.

What was found and done

1. Both exact review probes were adopted unchanged. They first fail on original product0c286f17 at the intended deletion-intent and missing-WORKSPACE_CACHE_INSTALLED assertions, through public service methods and real local SSH with FakeDropletService. Native baseline XML and complete stacks are in evidence. No compile failure was counted as a reproducer.
2. Activating or blank-IP followers with pending work retain their original owned allocation name and reservation; subsequent observations can assign the same ID. The first released observation is persisted in the existing phase map. Retention is bounded by existing MAX_RUN_PROVISIONING_DURATION_MS=600000ms from that first observation. Repeated observations do not reset it. Exact599999/600000ms test also checks phase stability and that the waiting peer remains PENDING. Retention cannot reclaim a reservation that was already returned.
3. Reassignment receives the wave's original prepared cache. The seed downloads a run-owned snapshot, verifies byte count/hash and atomically installs it; a future makes those validated bytes available to reassigned workers after the sharing server has stopped. A new variant requires actual prepared marker bytes before a real nonempty lease. Preparation/download failure completes the future exceptionally, and owned worker joins/cancellation keep snapshot lifetime within the wave. No interface/public method was added.
4. An initial ownership batch found4existing assertion failures: an overbroad recovery binding-null branch retained ready followers with no pending work or no original reservation. It was narrowed in product code, without changing those tests. Final source164db1f4d8 passes ownership30/30 and scenarios24/24 first attempts,0sibling histories. Dedicated test/code review passes are recorded in findings.
5. Mutation controls: readiness alone reverted gives the intended deletion assertion failure, with cache correction retained. The final readiness-only mutant is byte-identical to that already-executed mutant (SHA2562acd1aa7daaede78243a929acb619140fd275965c573680a090d313a8f37ebb2), so a redundant slot wait was stopped before execution. Cache alone reverted on final readiness code gives the intended missing cache assertion failure. Native failed rows include3internal failed attempts each; none is presented as a pass. Final source restored, clean, SHA256348d15b6df653c7d9fa5180d270fe454414728c47ef846b66b885b004abfdfb6.
6. First final repetition https://buildtest.kotlin.build/run?id=a7e7582a passes all6selected rows, canonical/highest attempt1, single PASSED attempt, flaky=false. Second final repetition https://buildtest.kotlin.build/run?id=e8ac36aa fails2rows at30seconds, with4first-attempt passes. Adopted cache probe stops at testRev1421aAssignedReleasedFollowerInstallsWorkspaceCache.kt:375, peerDecision.await, which follows cache installation/concrete lease assertions. Unchanged e2eShardedWorkspaceCacheStagesAreStampedPerDroplet stops at TailingInputStream.awaitNextPoll:164. Native output fields for both are empty. Full native error stacks are preserved. The mechanism behind these two timeouts is NOT established; do not infer a host cause or loosen the timeout. This is NOT a qualifying3/3gate.
7. Remote ledger:0b6bff2e is supplemental original-product red; e7d3374e is old-candidate6/6pass (excluded from final gates);5e2d1862 is old candidate canceled before TESTING; a7e7582a final repetition1; e8ac36aa final repetition2; cd5a8bc2 final repetition3was already submitted before failed rows were recovered and was canceled before TESTING. At most2were nonterminal at once. Canceled IDs:5e2d1862 and cd5a8bc2. An attempted e7d3374e cancellation was refused because it had already COMPLETED.
8. Exactly one new PR head164db1f4d80bd0e09e70a28167571a1f3bb6c79e was pushed with force-with-lease after fetch/rebase (main331819489630c52809b476b2a66639bf62b84e43, no rebase change). Description PATCH/readback was byte-exact. It states NOT READY for round5 because3/3is unmet, while keeping non-draft state. No CI wait, merge, queue entry, deployment or artifact release occurred. This is verified by the work/action record; no live deployment was changed by this session.
9. Separate task-supplied CI failure remains untouched: community.kotlin.buildtest.projection.tests.review1277DeltaRemovalPublicDiagnostics, shard2of4on0c286f17,30s in ProjectionPayloads.forceDirectory at ProjectionSourceStore.kt:63. It came from https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1277 , merged05:39UTC, and is tracked separately. No claim that it explains these two cache-test timeouts.

Relevant PRs / refs

Main has neither this round's source changes nor the evidence. No deployed or released code from this effort exists outside these unmerged refs. Both touched worktrees were checked clean; no stash or unpushed source commit remains.

Next steps

1. Re-verify current source/PR/run state first. Do not push a second source head under the completed round5 allowance.
2. Investigate the two30second waits locally through the original public scenarios. Preserve existing assertions/timeouts. Reproduce the exact narrow paths with genuine concurrent callers (the failed remote first wave had the cache probe, activation probe, blank-IP variant and unchanged cache control in flight), and collect complete owned-thread stacks while the waits exist. Capture public observer boundaries around cache preparation/lease delivery/peer admission and source pipeline completion. An outer timeout frame alone does not identify the cause.
3. Before changing code, write the mechanism and a reliable public reproducer. Once identified, make the trigger deterministic. Fix its owning layer, with consumer/upstream tests if it is in a dependency. Do not use another remote submission as a diagnosis tool or treat a later green run as refuting the saved failure.
4. In an authorized fresh round, complete both adopted probes3/3first attempts on the exact final head, plus the task's required ownership/scenario/mutation gates when source changes warrant them. Respect at most2inflight/6submissions. CI shards are the full suite, owned by orchestrator. No broad local neighbor sweep.
5. PATCH/read back an honest description with the new measured gates, and obtain a fresh independent review. No merge/queue/deploy/release without new explicit user instructions.

Operational knowledge and evidence

Evidence root: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fleetfix5-evidence/investigations/fleetfix5/evidence . Full findings include original intended stacks and later30s stacks. Selected native JSON rows retain all attempt fields; -api-results.xml files are explicitly labelled renderings, not runner-emitted XML. The CLI did not emit the requested remote XML; all43pages were read before selecting rows. Catalogue-page hash manifests are stored; unrelated FILTERED_OUT rows are intentionally omitted from git. No binaries, dependency caches or credential files are pushed.

Original probes remain byte-identical to review copies: activation SHA256cc362bcbac33f2b4c02f1ad70c2324a2711c84de7fc0e2678fc1c6eca885cfd9; cacheb991331e6d0df5a11f5a2c6242e099692913efacb614d36cd9b1d7a1e6462445. Source diff relative0c286f17contains only BuildTestEmbeddedService.kt,README and5new tests. No old test assertion edits.

Portable targeted command after fresh fetch/rebase and taking a local slot: LC_ALL=C.UTF-8 scripts/test.bash --local --test testRev1421aAssignedReleasedFollowerInstallsWorkspaceCache --log /tmp/cache-probe.xml. Normal control: selector e2eShardedWorkspaceCacheStagesAreStampedPerDroplet. Remote final repetition selected those2plus testRev1421aLateCreatedActivatingFollowerWaitsForNextObservation, testReleasedActiveFollowerWithBlankIpWaitsForNextObservation, testReleasedFollowerActivationUsesExistingProvisioningBound, testReleasedFollowerUsesPreparedSnapshotWithoutSeedServer. Helper scripts on evidence branch document exact slot/gate/native-row checks; update their lane paths rather than reusing old directories. Read the project README and the current common verification brief before building.

Final cleanup audit: both touched worktrees clean; all owned remote runs terminal, canceled5e2d1862andcd5a8bc2 before TESTING; no owned background process remains in the lane. Native source jars were generated build outputs and removed; compilation caches kept. The final process audit and handoff creation record are on the evidence branch.

Round 6 live update

2026-10-11 07:58 UTC — RUNNING. The request is to attribute the prior cache-test timeout and finish final-head gates for https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1421. Fresh verification: OPEN, non-draft, unmerged, head164db1f4d80bd0e09e70a28167571a1f3bb6c79e. No source correction in round6: healthy phase traces rule out the proposed attach-clock sleep on the sampled path but do not identify the saved failure. Local originals2/2, amplified copies8/8, recorded profiles2/2 first attempts. Two remote submissions have completed with12/12 and9/9 reported passes; native all-page attempt validation is running before counting gates. Mutation readiness control is running locally; cache control next. No delegated agents. No decision needed from the user; missing failure-path worker evidence remains the attribution gap. Landing sweep: both allowed remote gates ran concurrently, now stopped pending source restoration from the local mutation control. Actively advancing through traced phases, native result collection and controls. Next steps: restore exact source, complete remaining four remote gate submissions, preserve proof, update PR and handoff, audit owned processes. Lessons: outer wait stacks do not locate worker delay; passing repetitions do not identify a prior timeout; preserve exact native attempts rather than trust a client exit or summary.

Evidence checkpoint: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fleetfix6-evidence . Orchestrator retains completion ownership.

Round 6 evidence checkpoint 08:09 UTC

Source remains164db1f4d80bd0e09e70a28167571a1f3bb6c79e; no new source correction or assertion edit. First remote pair21/21first attempts, current pair4b0cb14f/6ea6eeb4BUILDING,4/6submissions total. Mutation controls2/2intended failures and source restored byte-identically. Recorded phase traces plus8concurrent public callers complete; the saved failure mechanism remains unestablished. Evidence checkpoint https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/3abfa63a42 (branch https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fleetfix6-evidence). No completion claim; orchestrator retains completion ownership.

Round 6 terminal update — gates complete, timeout attribution unresolved

2026-10-11 08:24 UTC — final report, BLOCKED on timeout attribution. The original request was to identify the remaining cache-test timeout, finish final-head gates and update https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1421. Freshly verified PR status: OPEN, non-draft, unmerged, head164db1f4d80bd0e09e70a28167571a1f3bb6c79e. No source or existing-test assertion changed in round6.

Final-head remote gates pass63/63selected first attempts: adopted activation3/3, adopted cache3/3, cache control3/3, ownership30/30, inherited scenario gate24/24. Mutation controls2/2fail at intended public assertions; source restored byte-identically. All6owned remote runs COMPLETED, maximum2in flight, no cancellations and no new test runs after gates met. Local originals, isolated copies, phase recordings and8concurrent public callers pass; they do not reproduce or attribute the saved timeout. The200ms public log tail is a reader wait; the2000ms attach-clock sleep was not entered by healthy recorded paths. Cache installation/lease assertions had completed before the saved peer wait, but no failed worker phases establish why the peer had not reached admission. The unchanged cache control still executes changed product code and cannot establish an unchanged-product baseline.

The existing correction keeps an owned activating follower available within its original provisioning bound. It gives a reassigned follower the validated prepared cache. It conserves the original admission slots while the accepted lease is held.

Delegated work at a glance: no subagents started. Landing parallelization sweep: all allowed gates completed; none left idle. Remaining blocker: failure-path phase/owned-worker evidence and a clean-main comparison are needed to prove the timeout mechanism; the six-submission allowance is used. READY cannot be claimed from fresh passes alone. No user question is pending in this bounded round; the orchestrator receives NOT READY and retains control of the next diagnostic round. Next steps for resumption are to verify that timeout diagnostics survive result collection, capture the failing worker boundaries, then force the identified condition before changing code.

PR description PATCH/readback is byte-exact and preserves the opening and footer. Evidence: https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fleetfix6-evidence . Handoff: url://handoff/handoffs/hf-2026-10-11-investigate-the-round-five-cache-probe-timeouts-and-finish-buildtestembedded-gates ; the body is updated, never archived by this lane. All owned watchers stopped; preliminary ps audit finds0owned processes, final audit follows after handoff operations.

Lessons: a caller wait does not identify the worker delay; unchanged test text does not prove unchanged implementation; filtered native rows and client exits do not replace exact selected-test attempt evidence.

Round6 does not resolve the resumption note: the timeout cause is still unestablished. The orchestrator will complete the note once that work is verified; this lane did not call complete. Source remains https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/164db1f4d80bd0e09e70a28167571a1f3bb6c79e . Exact gate ledger, full selected native rows, all-page manifests, local phase traces, owned worker snapshots and mutation failures are preserved at https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/fleetfix6-evidence/investigations/fleetfix6/evidence . No deployment or publication occurred.