Handoff: Finish remote proof for the cursorless work tests

Written 2026-10-26 00:43 UTC.

RE-VERIFY: This is a write-time snapshot. Run gh pr view https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21 --json state,headRefOid,mergeStateStatus,statusCheckRollup and recheck url://buildtest/ before acting. The PR is OPEN at 28f5d1999b76042da9a8944dab18b6a83c256e9f; its GitHub Actions bld-build has started.

Design decisions in force

The full user brief is preserved verbatim below and on the evidence branch. Key 2026-10-26 directions: “Never raise the 30 s deadline, and never shrink the corpus sizes or iteration counts. Both are absolute rules.” “Keep what the tests verify identical: real published-0.0.9 fixtures, the same sizes, and the same work-count assertions. Keep them hardware-independent.” “run both tests remotely ... 3 times each ... then run the full suite remotely once; it must be 169/169 with zero retries.” “Do not merge.” “Production is read-only: no restart, deploy, upload or route update.” “If a phase blows up in scope, push what you have, write down exactly where it stands, and stop rather than drifting.” The latter is the stop rule used here; do not infer permission to repair/deploy shared production infrastructure from this test task.

Mission summary

Finish two more successful remote targeted repetitions of each changed test, then one remote full suite with 169/169 and no retries. The tests exist because restarting the aicli supervisor manager made search unavailable while an old index rebuilt; the introducing PR keeps committed search immediately available while recovery proceeds in bounded maintenance. The work-count guards were timing out at 30 seconds remotely even after correct zero-byte open results appeared.

What was found and done

1. Fresh clone under lane ws/, original head https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/commit/d9b94ca630e802237001b6792684aee09922cc88. README and testing/slow-services/philosophy guidance read. Both tests already resolve the real 0.0.9 artifact through @WithArtifact; runtime Coursier resolution is zero. Old generation already batches commits every 1,000 events, rather than syncing per document.
2. Instrumented three remote baseline runs without changing assertions or timeouts. Durations (total runner ms):

Before runs: https://buildtest.kotlin.build/run?id=dfa2b312 , https://buildtest.kotlin.build/run?id=ca7bba26 , https://buildtest.kotlin.build/run?id=f4b39e90 . First after: https://buildtest.kotlin.build/run?id=6e306ba6 . No retry warning occurred in those successful targeted runs. 3. The mechanism in the probe is reconstructing and reparsing its complete 106 MB 0.0.9 fixture inside every timed body. In first remote baseline it spent 2,812 ms generating its journal and 7,351 ms in the old seeder, versus 1,237 ms in the current opener. Recovery included 388 ms fixture generation, 1,312 ms helper compilation, 2,449 ms old seeding, 1,590 ms current open checking, and repeated full-corpus checks across three recovery children. Expected ids repeatedly used String.format. The seeder lifecycle measurements separate JVM/process overhead: 143 ms recovery and 230 ms probe beyond measured manager lifecycle time. Parent waits were executing this child work, not sleeping. 4. Offline JFR file-force measurement (JDK 21, actual 0.0.9 public API) found recovery six commits / 57 force calls / 95 ms force time across 4,000 documents and probe four commits / 42 calls / 61 ms across 3,000. This rules out the per-document fsync theory. The probe's original generator was rerun and matched the archive's journal exactly: 106,025,772 bytes, SHA-256 2a5727c38e4ce86d5f18722b1f601f021b70f224a3ea2ac5d2018b8ba85d3384. An older note said one byte more; the actual generator is authoritative. 5. Test-only commit https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/commit/28f5d1999b76042da9a8944dab18b6a83c256e9f saves the unchanged real-0.0.9 fixtures as separate per-test compressed inputs (111 KB / 434 KB), with hash-checked generators and provenance. Each test extracts a fresh directory and pins its archive hash. Recovery constructs the same fixed-width expected ids without Formatter. All 1,000/3,000 corpus sizes, body lengths, 250-event batches, three batches before each of two real halts, final completion loop, exact search ids/JSON checks after every batch, zero changed rows, resume positions and 1.25x journal-read limit remain. Probe retains its 8,192-byte limit and ten hits and adds a 3,000-row count/unchanged journal-size check. No deadline or current-code operation was removed. 6. Dedicated test-comprehensiveness and code-review passes found no remaining gap in this test-only change. Full local suite passed 169/169, zero failed, on the exact committed head before any push. First fixed remote trial passed 2/2, zero retries; outputs still show zero-byte/replay opens at both sizes and recovery total reads 3,731,023 <= 4,508,631, with halts applying 750 each and final attempt 4,502. 7. Second fixed trial https://buildtest.kotlin.build/run?id=4921e082 failed before either test started. Its failed-creation fleet read hit the local hourly budget (windowSpend=4500/4500, provider remaining 3496), then the coordinator logged PENDING but remained PROVISIONING until a five-minute watchdog ended it. Full evidence and the precise remaining infrastructure investigation are in url://handoff/handoffs/hf-2026-10-26-repair-buildtest-provisioning-after-fleet-reads-hit-the-local-hourly-window. No unchanged run was resubmitted. The finite verification driver stopped, so the third trial and remote full suite were not started. Remote 169/169 is NOT proven on the fixed head. 8. Applied the user's scope-stop rule: pushed the locally green source to the existing PR branch, preserved all timing/phase/error evidence on a separate remote branch, and left the PR open. CI started: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/actions/runs/40250504842/job/121341132854 . No merge/enqueue or production changes.

Relevant PRs / refs

All remote heads were verified with git ls-remote. Only one checkout/worktree was used, no stash and no unpushed commit remain. Build outputs, compiled classes/JARs, dependency caches, and copied guidance documents are intentionally excluded from evidence.

Published but unmerged: searchablebucket.embedded:searchable-bucket-embedded:0.0.20 was already published by the preceding effort from the original PR head above (not merged). Its public POM was freshly fetched and confirms version 0.0.20. This session changed no production source or build version and published/deployed nothing. No republish is required for the test-only commit; do not publish 0.0.21 merely to finish verification.

Next steps

1. Re-verify PR/current head and the infrastructure blocker first. Read the original brief, README and current instructions before running anything. Preserve the no-merge/no-production-change rules.
2. Once provisioning actually works, run scripts/test.bash --remote --test tests/testCursorlessRecoveryWork.kts --test tests/testProbeCursorlessOpenWork.kts for two further successful first-attempt trials. Report all durations, including the already captured first trial. Do not treat the failed pre-start infrastructure attempt as a test timing or a passing trial, and do not blindly resubmit it while the known fault remains.
3. Then run exactly one scripts/test.bash --remote --test .; require 169/169 and zero automatic retries. The user explicitly requires this sequence. A full remote runner script is preserved in the evidence branch; adjust only its local paths, never timeouts/corpora/iterations/assertions.
4. Check df -h / before every build and stop below 3 GB. Use ~/bin/coursier. Delete this lane's caches after each build. If the head changes, repeat the exact-head full local suite before pushing; the current unchanged head already has that proof.
5. Update the existing PR's validation comment with completed proof, keep CI started/current, and stop without merging. No production deployment or republish is part of this task.

Operational knowledge

--result-file contains the overall summary; --log has test timings. Full stdout/errors come from UrlProtocolBuildService.rpc("getTestResults", mapOf("runId" to id)); the preserved FetchRemote.java is a small direct client. The current method list also advertises getTestAttemptResultsPaginated and getTestResultFieldChunk, useful for checking retries. Build WUI test-results may be stale/empty, so use the direct backend evidence before assuming no tests ran. Each test's support README regenerates the exact real-version input; never replace it with a current-version simulation. No background verification or watcher is left running by this session, and no goal is active.

Original user brief (verbatim)

Make two new tests in https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21 fit their 30 s limit remotely (head d9b94ca630e802237001b6792684aee09922cc88, published 0.0.20)

Why the PR exists

Redeploying the aicli supervisor manager (the backend of https://aiclisupervisor.wasmserver.com) made search unavailable while the index rebuilt from history. The operator called that a design violation of https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/playbooks/SLOW_SERVICES.md. This PR makes opening any old index constant-time, with recovery happening in the background.

Problem

The full suite passes 169/169 locally on d9b94ca. Remotely (scripts/test.bash --remote --test ., runs c0f7a717 and 33e905c2) it was 167/169.

• testCursorlessRecoveryWork and testProbeCursorlessOpenWork hit their outer 30,000 ms test deadline while the parent waits for a child process.
• The work assertions had already printed the correct results: 0-byte opens at 1,000 and 3,000 rows.
• Full traces: /tmp/claude-1000/-code/f67b4d83-4ebc-4507-bde7-f1a83d72de59/scratchpad/fix21d/out/full-final-results.json and .log. Notes: /tmp/claude-1000/-code/f67b4d83-4ebc-4507-bde7-f1a83d72de59/scratchpad/fix21d/out/findings.md, lines 69–104.

The required kotlin.build (remote) CI check runs remotely, so these must pass there.

Rules for this fix

• Never raise the 30 s deadline, and never shrink the corpus sizes or iteration counts. Both are absolute rules.
• Find what makes each test slow, with measurements. Time each phase on a remote-like run: child JVM start-up, Coursier resolution of the real published 0.0.9 artifact, fixture generation, fsyncs per document, waiting. Write "the mechanism is X, and the evidence is Y".
• Then remove the waste. Examples, if the evidence points there:
  • build the large fixture with batched commits instead of a commit per document;
  • share one child process across both sizes;
  • avoid a cold dependency resolution inside the timed window, by resolving through the test's @file:WithArtifact classpath rather than at runtime;
  • stream instead of materialising.
• Keep what the tests verify identical: real published-0.0.9 fixtures, the same sizes, and the same work-count assertions. Keep them hardware-independent.
• Proof:
  • run both tests remotely (scripts/test.bash --remote --test <file>) 3 times each; all must pass with clear headroom, and report the durations;
  • then run the full suite remotely once; it must be 169/169 with zero retries.

Task

1. Clone the PR branch fresh.
2. Fix the two tests as described.
3. Push to the PR branch. This is a test-only change, so no republish is needed, unless you also had to change main source, in which case publish the next free version (0.0.21).
4. Comment on the PR.
5. Do not merge.
6. Do not stop at scope checkpoints.

Rules (all lanes)

• Workspace. Clone into a NEW directory under your lane's ws/ folder, using gh repo clone CodexCoder21Organization/<repo> <dir> -- --recurse-submodules. Read each README.md and AGENTS.md before working. Use ~/bin/coursier, never /usr/local/bin/coursier.
• Disk. The disk is ~98% full and shared with other jobs.
  • Run df -h / before every build. If less than 3 GB is free, stop and write that down in your findings instead of building.
  • Delete build outputs you no longer need, and delete your clones' caches when you finish.
• Tests. Read https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md before writing tests. Tests are end-to-end through public API with real in-process instances; no mocks, no reflection. Verdicts are hardware-independent: assert work counts, never wall-clock thresholds.
• Never raise a timeout, cut iteration counts, weaken assertions, or skip tests. "Cap", "throttle", "retry" and "keep warm" are not fixes. "The host was overloaded" is never a root cause.
• Before pushing, run the repository's full local test suite once on the exact head, and report pass/fail counts. Do not use CI as your test runner.
• PRs. Open each PR with a description that starts with the why (given in your brief), followed by what changed. Publish the artifact version per the repo README if downstream consumers need it.
  • Do NOT merge or enqueue any PR; the orchestrator decides merges.
  • Production is read-only: no restart, deploy, upload or route update.
• Checkpoints. Push a wip/<lane>-2026-10-25 branch at the first real milestone (failing work-count tests) and at each one after.
• Findings. Write findings incrementally to your lane's out/findings.md, not only at the end. Every ~5 minutes, append one paragraph with your current state. Separate OBSERVED from INFER. Before claiming a mechanism, write: "the mechanism is X, and the evidence is Y."
• Waits and cleanup. Any wait loop must have a deadline and must exit on every terminal state. Never pgrep -f or pkill -f on text in your own command line. Before your final message, kill every background process you started and check with ps -eo pid,args -ww.
• Stop condition: once the failing tests pass, the full suite is green and the PR is open with CI started, stop. Do not do extra verification rounds. If a phase blows up in scope, push what you have, write down exactly where it stands, and stop rather than drifting. Your lane folder is /tmp/claude-1000/-code/f67b4d83-4ebc-4507-bde7-f1a83d72de59/scratchpad/fix21e (ws/ and out/ inside it). Delete ~/.aibuildcaches/*fix21e* after each build.

Final message

Report:

• the mechanism behind each slow test;
• before and after durations, remote, 3 runs each;
• the remote full-suite counts, including retries;
• the new head;
• whether a republish was needed.

Final PR follow-up

Comment posted: https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/pull/21#issuecomment-6405710827. All source/evidence branches are pushed, the working tree is clean, and final process/cache cleanup is complete. The latest evidence branch head is https://github.com/CodexCoder21Organization/SearchableBucketEmbedded/commit/86d84ff26cd3a5633b63df66372b6b2bd85e3a4f.