Collect three fresh SearchableBucketEmbedded first-attempt records after remote scheduling resumes

Written: 2026-10-27 04:53:29 UTC by fable-W1-1027.

RE-VERIFY: All state below is a write-time snapshot. Recheck the three run IDs with native getBuildRun/getTestAttemptResultsPaginated, use build-watchman --run <id> --once, check current main with git ls-remote, and verify PR states with gh pr view. A PENDING run with zero rows is not a clean suite.

Design decisions in force

Owner direction, 2026-10-27: "Use your best judgement, do not ask blocking questions."

Orchestrator ruling, 2026-10-27: "That is not a blocker: the evidence is gone for good and waiting cannot bring it back. The goal was never the old run; it is that SearchableBucketEmbedded's suite has no first-attempt failures."

The acceptance check is three fresh current-main full remote suites, inspected by per-test FIRST attempt, not final summary. If all three have zero first-attempt failures, report DONE and complete. Otherwise reproduce and fix the most frequent current defect test-first and open a PR. Historical run 5b93c074 recovery is no longer a dependency.

Worker constraints: "Never raise a timeout, cut a stress iteration count, skip/disable/delete a test or weaken an assertion." "Take no production action at all - except reading logs, metrics, and status endpoints, which you may do freely." No merging, enqueuing, auto-merge, approving or production deployment is authorized for this worker. Maven publication is allowed only from default-branch commits after relevant consumer verification.

Mission and current blocker

The original handoff named eight failed first attempts in run 5b93c074 after recovery PR https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21. Those historical records were removed by retention. The orchestrator explicitly replaced recovery with fresh current-suite evidence.

W1 cleared the historical blocked reason, claimed, cloned main fresh, fetched/rebased and started the full suite three times concurrently. Each admitted run has 188 tests from main cf052452ce5b0db549cb176337b7ef7c48376f9f. Each CLI call then exited 2 because its built-in 1,800,000 ms remote wait expired while the run remained PENDING. No tests executed. Final native reads still return PENDING, testsTotal=188 and zero attempt rows for all three.

This is a new remote-execution blocker, not the deleted-history blocker. No first-attempt verdict or SearchableBucketEmbedded defect can be identified from zero executed tests. No code fix or clean-suite claim is made.

What was observed

1. Main is cf052452ce5b0db549cb176337b7ef7c48376f9f. The recovery and reader-cleanup PRs are merged. The separate interruption-read PR remains open at bfd2cabeab181c90d4b69c9da84411b0af2b9fdb, was read only, and is not linked to a demonstrated failure from this batch.
2. Three commands started at 2026-10-27T04:09:45Z, using scripts/test.bash --remote --test . --log <runN.xml> --result-file <runN-result.json>. The working ~/bin/coursier was supplied as jars/coursier because the binary on PATH has the wrong architecture. No source or test was edited.
3. Native gitContext for each run confirms repository SearchableBucketEmbedded, branch main and the exact cf052452 commit. Source contains 188 test scenario files; the earlier 179-test total is stale.
4. Local command 1 maps to 00653c5b and wrote its terminal record at 04:40:07.389Z; command 2 maps to b360f7c9 at 04:39:59.553Z; command 3 maps to b80e0a89 at 04:40:05.456Z. Each result is status=PENDING mode=remote tests=0/0 failed=0 exit=2. The logs say Remote build wait expired, limit=1800000ms, with a resume command. The outer 80-minute deadline was never reached or changed. No timeout was raised and no test was re-run.
5. Final native metadata/attempt pages after the client exits still show all three PENDING with 188 selected tests and zero rows. The collector's paginationComplete flag only means all rows through the API-reported totalCount were read. It does not mean execution completed or that zero first attempts failed.
6. Nine OTHER recent runs failed provisioning before their tests. Complete summaries/errors are in provisioning-refusals.json. The complete 3,421-byte raw log of https://buildtest.kotlin.build/run?id=57da2aca independently records the failure at 2026-10-27T04:41:19Z:

RPC request 'listDroplets' failed: UNKNOWN_METHOD - No service registered for identifier: digitalocean-droplets

7. BuildTest fleet snapshots retain asOfEpochMs=1793075454374 (04:30:54Z), zero running/starting droplets, zero active runs, and five droplets awaiting teardown while the pending count changes from 51 to 48. These values do not establish why scheduling stopped.
8. Keep the contradiction: a fresh direct provider connection reported health=OK and its read-only listDroplets call succeeded with six rows at 04:43-04:44 UTC. The provider URL route is configured. A global provider outage is therefore not a demonstrated current root cause. The BuildTest dependency path may still be wrong, or it may have recovered without scheduling resuming; no fix is claimed.
9. Access limits: the documented HCF certificate directory is absent. ContainerNursery CLI 0.0.20 health fails with JSONObject["uptime_human"] not found; container inspection returns endpoint-not-found, and its buildtest log read returns zero rows with total_lines=29907. No raw SSH, restart, upload, route change, or deploy was attempted. The exact errors and scoped responses are preserved. Broad route/provider payloads and private process listings were deliberately not committed.
10. Only evidence and read-only helper sources were changed. Both working trees are clean; all evidence is pushed. The helper compiled and its paging/error handling was independently reviewed. No Maven publication, production change, PR creation/closure, merge or queue mutation occurred. All three local suite clients ended naturally at their built-in bound; no local suite watcher remains, and no remote cancel/delete RPC was issued.

Fresh runs and evidence

Evidence directory: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/tree/evidence/w1-5b93c074-retention-20261027/handoffs/artifacts/w1-fresh-suite-20261027

It contains all three command logs/result JSON files, final native metadata/attempt pages, source inventory, paginated collector source/instructions, provisioning refusal records/raw log, fleet observations and direct-provider read summary. CHALLENGE.md distinguishes observations from inference. The report-challenge CLI was not invoked because it automatically enqueues and merges, which this assignment prohibits; the note is preserved for the orchestrator.

Relevant PRs / refs

Nothing was deployed or published by this worker. No code from this effort is live outside GitHub evidence.

Next steps

1. Re-verify current state first, especially the three admitted runs. If any has executed or completed, collect every attempt page immediately before retention removes evidence. Do not count an expired/PENDING/zero-test record as a pass.
2. An authorized owner must diagnose and restore BuildTest's provisioning/scheduling path. Compare its provider client connection with the fresh successful direct listDroplets read; inspect provider/coordinator logs or thread state. Do not blindly restart or add downstream retries/longer budgets. This worker has no production-change authority and could not obtain further host diagnostics through the allowed interfaces.
3. After execution is working, resume each surviving run with scripts/test.bash --remote --attach <runId>. If a run has expired or disappeared, document that terminal state before submitting a replacement from freshly fetched/rebased main. The orchestrator owns remote waits.
4. Use CollectBuildEvidence.java and COLLECTOR.md to fetch complete paginated attempt rows. Reconcile every expected test identity, preserve per-test attempt counts and every failed FIRST attempt/output across the three full runs. Complete only if all three are first-attempt clean. Otherwise reliably reproduce the most frequent current defect, write a failing test, fix at its owning layer, record red/green/revert-control evidence and open verified PRs. No timeout increases, iteration cuts, disabled tests or weaker assertions.
5. No merge or deployment step is currently prepared: there is no fix PR or artifact. Once actual failures exist, the normal review and targeted/full-suite gates apply.

Documentation read

PlanRepository handoff-triage README; SearchableBucketEmbedded README; SearchableBucket project overview; Testing Architecture; Engineering Philosophy; relevant Code Review guidance; github-repos, create-handoff, status-report, fix-flakey-test, watch-build, investigate-broken-service, hardware-fabric-processes, container-nursery-deploy and report-challenge skills.