← Priority list
Blocked

Repair buildtest provisioning after fleet reads hit the local hourly window

Restore remote runner provisioning, then release the remaining SearchableBucketEmbedded test proof. Run 4921e082 failed before tests when listDroplets hit the local hourly window; its log reported PENDING while the run stayed PROVISIONING until the watchdog ended it. No infrastructure or production changes were made.

Observed fleet-read request-window rejection and PROVISIONING/PENDING disagreement prevent test runner allocation; no infrastructure change attempted.

Handoff document

Markdown

Handoff: Repair buildtest provisioning after fleet reads hit the local hourly window

Written 2026-10-26 00:42 UTC.

RE-VERIFY: This is a write-time snapshot. Read getBuildRun, getBuildLogChunk, and getFleetStatus on url://buildtest/ before acting. The recorded failed run is https://buildtest.kotlin.build/run?id=4921e082.

Design decisions in force

2026-10-26 user direction: “Production is read-only: no restart, deploy, upload or route update.” “Do NOT merge or enqueue any PR; the orchestrator decides merges.” “Never raise a timeout, cut iteration counts, weaken assertions, or skip tests.” No production changes are authorized by this handoff.

Mission summary

Restore the ability to provision the remote test runner so the remaining verification for https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21 can run. This blocker was observed while completing a requested test-only performance change; no infrastructure fix was attempted and no failed run was resubmitted.

What was found

  1. First fixed remote trial https://buildtest.kotlin.build/run?id=6e306ba6 passed both selected tests without retries.
  2. Next trial https://buildtest.kotlin.build/run?id=4921e082 failed before either test started. At 00:33:12 UTC createBuildDroplet reported an unknown creation outcome because its failed-creation recovery could not observe the fleet with listDroplets.
  3. Exact returned error: DigitalOcean request GET /v2/droplets?per_page=200&page=1 cannot be admitted before caller budget 29995ms expires at 2026-10-26T00:33:42.644Z; earliest possible retry time=2026-10-26T00:34:00Z; binding local window=hour: queuedWrites=0, queuedReads=1, windowSpend=4500/4500; local budget recovers at 2026-10-26T00:34:00Z; provider reset=2026-10-26T01:16:28Z; ratelimit-remaining=3496.
  4. The build log says Allocation capacity unavailable; requeueing build as PENDING, but the returned run remains PROVISIONING, dropletId=0, and its five-minute provisioning watchdog marks it FAILED. This state/log disagreement needs investigation. The full source mechanism behind the request-window state has not been established; do not treat the rate-window error as proof the provider actually exhausted its allowance.
  5. Read-only fleet snapshot at 00:35:02 UTC reported runningDroplets=12, startingDroplets=20, awaitingTeardownDroplets=110, activeDroplets=123, dropletLimit=200. This is evidence to inspect bookkeeping, not a justification for raising limits or adding capacity.

Evidence and refs

All captured evidence is on the evidence branch. In particular:

No source changes, deployment, publication, restart, route change, merge or enqueue occurred for this blocker.

Next steps

  1. Re-verify current state first; this error can become stale as request windows and fleet state advance.
  2. Trace the fleet observation and local hourly-window calculation in the owning service. Force the observed rejection deterministically in a public-API test before changing its implementation. Do not raise limits, widen deadlines, or add blind retries.
  3. Trace why a reported PENDING requeue remains PROVISIONING and reaches the provisioning watchdog; test the transition and unknown-create bookkeeping so a repeated create cannot be issued while an earlier outcome is unresolved.
  4. Obtain any required production authorization separately. Once the remote runner can provision, complete this blocker so the test-verification handoff can continue.

Model usage

Recorded agent work, including concurrent agents

No model usage has been recorded for this handoff yet.

Record work with handoff-cli usage report hf-2026-10-26-repair-buildtest-provisioning-after-fleet-reads-hit-the-local-hourly-window.

Latest status report

1 report
Blocked fix21e f67b4d83 1 hour ago

2026-10-26 00:46 UTC — FINAL: BLOCKED

Original request: make testCursorlessRecoveryWork and testProbeCursorlessOpenWork in https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21 fit their unchanged 30-second deadline remotely, preserving all fixture sizes, iterations and work assertions; prove three remote passes each followed by a remote 169/169 full suite with zero retries; push/comment and do not merge.

The source change is pushed, locally verified and commented. Remaining verification is blocked because the second remote trial failed in provisioning before either test started. The backend rejected listDroplets against its local hourly request window (4500/4500 spent while the provider reported 3496 remaining), then retained PROVISIONING despite logging a PENDING requeue and reached its five-minute watchdog. No user approval question is pending; the needed external change is working remote provisioning. Per the user's explicit scope-stop rule, no shared production change or unchanged rerun was attempted.

Pending PR: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21 — freshly verified OPEN, mergeStateStatus BLOCKED, head https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/commit/28f5d1999b76042da9a8944dab18b6a83c256e9f. CI started at https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/actions/runs/40250504842/job/121341132854. Comment: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21#issuecomment-6405710827. No PR was merged or enqueued.

The tests now extract unchanged historical inputs generated by the real published 0.0.9 artifact. This removes repeated fixture generation and old-version index construction from their timed bodies while each run still uses fresh temporary state. Recovery also constructs the same five-digit expected ids without repeated String.format calls, retaining every current-code open/search/recovery assertion and real crash attempt.

Measured mechanism: probe previously generated and reparsed a 106 MB fixture on each timed run (first remote trial: 2.812 s generation + 7.351 s seeding versus 1.237 s current open/search). Recovery repeated legacy preparation across the 1,000/3,000 corpora and formatted expected ids across every full-corpus check. Coursier resolution is already outside the bodies; old commits are batched, and offline JFR measured 57/42 file-force calls rather than one per document. Source and fixture provenance are preserved in the PR; full phase data is in the evidence branch.

Test Remote before 1 / 2 / 3 Remote after 1 After 2 / 3
testCursorlessRecoveryWork 16.456 / 26.383 / 24.181 s 19.014 s Provisioning failed before tests / not run
testProbeCursorlessOpenWork 13.764 / 19.706 / 19.050 s 6.017 s Provisioning failed before tests / not run

Full local suite: 169/169 passed, 0 failed on exact source head before push. First fixed remote trial: 2/2 passed, zero automatic retry warnings. Fixed remote full suite: NOT RUN; remote 169/169 and its zero-retry requirement remain unproven. No manual retry of the failed provisioning run was made.

Delegated work at a glance: no delegated work active or recently finished. No subagents were used. All finite verification drivers and watchers exited; final ps sweep found no owned build/test/watch process. Lane caches and development build outputs/classes/JARs were deleted. No active goal exists.

Holds audit: remaining remote proof depends specifically on runner provisioning; no unrelated work is held. Landing parallelization sweep: no idle landing gates. Owner direction captured: none new; all original behavioral requirements are preserved verbatim in the handoff and original brief on the evidence branch. No republish was needed because only tests/support changed; the existing 0.0.20 POM was verified and no production action was taken.

Next steps: restore provisioning, complete two further successful remote targeted trials for each test, then run one remote full suite requiring 169/169 with zero retries and update the existing PR without merging.

Handoff: url://handoff/handoffs/hf-2026-10-26-finish-remote-proof-for-the-cursorless-work-tests — https://www.handoff.wasmserver.com/handoffs/hf-2026-10-26-finish-remote-proof-for-the-cursorless-work-tests. Infrastructure dependency: url://handoff/handoffs/hf-2026-10-26-repair-buildtest-provisioning-after-fleet-reads-hit-the-local-hourly-window. Source checkpoint: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/tree/wip/fix21e-2026-10-25. Evidence: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/tree/wip/fix21e-evidence-2026-10-26/handoff-evidence/fix21e.

Lessons: measure fixture setup, compiler/JVM work, and the behavior under test separately before optimizing; preserve exact historical inputs as independently owned immutable fixtures when their construction is unrelated to the contract under test; distinguish a test failure from a provisioning failure using backend state and full logs before rerunning anything.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.