Handoff: Verify queued CI for the published URL client update

Written 2026-10-20 20:57 UTC.

RE-VERIFY: This is a write-time snapshot. Re-check https://github.com/CodexCoder21Organization/SearchableBucketWui/pull/2 with gh pr view <full-PR-URL> --json state,headRefOid,statusCheckRollup,mergedAt and inspect https://buildtest.kotlin.build/run?id=bbd51502 before acting. Queued CI is not a failed test or proof of a stuck executor.

Design decisions in force

The October 20 p69 brief says: “No merging, enqueueing, publishing, deploying, route changes, restarts or any production change (connecting to production services as an ordinary client for measurement is fine; nothing else). No timeout increases, no retries/keep-warm/fallbacks added, no iteration reductions, no disabled tests.” This limits the remaining work to verification, not merge or deployment.

The October 20 p72 continuation says: “Inspect it first; keep every finished piece of work.” It also requires checking df -h / before every build/test and stopping if less than 8 GB is free. Preserve the completed measurements and reviews; do not redo them. Remove only task-owned caches and extra checkouts.

Mission summary

The original user request was: “We need to investigate why https://my.externalbrain.wasmserver.com/ is failing to load and fix the underlying problem.” The p69 slice updates SearchableBucketWui's old published URL clients to pick up already released cold-start fixes, in one PR with no deployment. p72 continues a session stopped by a full disk; its remaining local suite, push and PR tasks are complete. Green required CI verification remains blocked. The remote check completed FAILURE at 20:51:08 UTC before tests; the final summary reports 0 passed, 0 failed, 11 total. A handoff was prepared during the queue wait, then claimed when the run advanced to provisioning. The ordinary run-page log subsequently established an executor-creation failure before any test; required CI verification is blocked at this snapshot.

What was found and done

1. Preserved the two application commits on the requested branch. No application source or visual output changed: build dependencies, own version and test declarations changed, and a real-service cold-start compatibility test was added.
2. Selected published Resolver 0.0.1325 / Protocol 0.0.1426, SJVM trio 0.0.55, clocks-simple 0.0.16, API 0.0.3; WUI version is 0.0.2. These were the newest published pair checked in the original session. Resolver's POM declares Protocol 0.0.1421, SJVM 0.0.55 and clocks 0.0.16. Bytecode inspection confirms the newer Protocol getter matches Resolver and the assembled fat JAR contains one matching resolver/protocol class each.
3. Bytecode-fetch discovery no longer calls the relay-registration wait, and its acknowledged-relay getter no longer acquires the RelayService monitor held during connection setup. These are already published upstream changes: https://github.com/CodexCoder21Organization/UrlResolver/pull/1263 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1068 . Direct-before-relay ordering remains separate work; this PR does not solve or claim to solve it.
4. Preserved three fresh-JVM, read-only first-overview measurements per version, with debug logs and bounded thread samples. Each returned HTTP 200 with 6,368 bytes. All three old runs sampled the registration wait, none of the new runs did. Neither group sampled a monitor block; monitor removal is structural artifact evidence, not a measured duration reduction. The provider helper can itself use relay transport and emits no direct-success message, so exact physical transport and relay candidate counts are unknown.

5. The new compatibility scenario starts the public WUI API, isolated loopback URL provider/client nodes, real BucketManager and RPC handler, and the published client bytecode. Its first data-bearing HTTP request renders the seeded bucket/document counts, with one bytecode delivery and one call each to listBuckets/listDocuments, without a warm-up read. Its temporary disk directory is required because the published Lucene implementation bypasses the supplied Okio filesystem. It closes resources, restores the previous public-IP setting and removes the directory. Independent test and code reviews are complete with no open findings. The corrected scenario previously passed on its first execution in 11,081 ms under the unchanged 30-second default.
6. Fresh fetch/rebase was a no-op. Exact application head passed the full serial local suite: 11/11 passed on first attempts, with 0 failures and 0 reruns. The preferred remote attempt connected but yielded no execution results; an earlier targeted remote submission also remained unscheduled, so local execution supplied the required suite proof. No failed tests were blindly rerun. No code changed during p72.
7. Pushed both preserved application commits and opened exactly one PR with original motivation, dependency choices, measurement table, evidence limits, suite counts and required footer. The PR remains OPEN/unmerged. Required kotlin.build (remote) began 20:09:59 UTC. At 20:43 the monitor changed the run from PENDING to PROVISIONING. At 20:46 the API still reported PROVISIONING, scheduledAt 20:38:59 UTC, no droplet id/IP, and all 11 test rows PENDING. Its run-list projection was marked outOfDate, so treat these as reported state, not direct host observations. Root cause of queue delay is unknown; do not call it a code failure or infer an executor defect.
8. Archived completed evidence remotely, excluded JARs/caches/build outputs and production HTML response bodies. Diagnostic trailing whitespace was removed; stack frames and measured values were retained. Removed the task-owned r69 build cache. No production code was published/deployed/restarted by this effort; the command record contains only local execution, ordinary client reads, GitHub branch/PR actions and handoff workflow metadata. No route or server state was changed.

Relevant PRs / refs

Evidence root: https://github.com/CodexCoder21Organization/SearchableBucketWui/tree/wip/r72-verification-evidence/investigations/r72-cold-start . It includes p69/p72, both findings, old/new logs and thread samples, artifact inspection, reviews, original fixture failures and final passing suite. Both remote heads were verified with git ls-remote. No stashes, unpushed application commits or uncommitted source remain.

CI gate blocked before test execution

Terminal check result, verified at 20:53 UTC: GitHub reports COMPLETED / FAILURE at 20:51:08 UTC. Its summary says Build failed: 0 passed, 0 failed, 11 total. The build error is Provisioning deadline exceeded for runId=bbd51502: elapsedMs=600292 limitMs=600000 firstAdmissionAt=1792528739038 currentAttempt=1 createdDropletIds=[] lastFailure=DropletServiceCallStalledException: Droplet service call createDropletAsync stalled for 30000ms while creating 'buildtest-bbd51502-a60a7b2' (region=sfo3, size=s-2vcpu-4gb, image=245388985). No application test ran. No rerun, timeout increase or code change was made. Final output: https://github.com/CodexCoder21Organization/SearchableBucketWui/blob/wip/r72-verification-evidence/investigations/r72-cold-start/github-final-check.json .

The ordinary run page embeds its build log in a <pre> element. Its 20:38:59 UTC entry says provisioning started. At 20:39:29 UTC it records provisioning_failure, method createBuildDroplet, elapsedMs=30130, followed by DropletServiceCallStalledException: the asynchronous createDropletAsync request result was not delivered conclusively after 30,000 ms. The next line says Allocation capacity unavailable; requeueing build as PENDING. All 11 test rows remain pending. This is an executor-provisioning failure before tests, not an application test failure.

The summary/API still reports PROVISIONING with no executor id/IP and is marked outOfDate. That disagrees with the log's PENDING requeue. The underlying cause of async result non-delivery and the stale summary is unknown; no source repair or CI retry was attempted. Full log: https://github.com/CodexCoder21Organization/SearchableBucketWui/blob/wip/r72-verification-evidence/investigations/r72-cold-start/ci-build-page.log . Snapshot and access-failure trace are alongside it. Do not hide these observations with downstream retries or test changes.

Fabric host-log access could not start: its default mTLS certificate files are absent from both configured home locations. Full trace begins java.lang.IllegalArgumentException: Certificate file not found: /home/helena/.config/hardware-control-fabric/fabric-client-cert.pem, through CertificateUtils.kt:43, FabricApiClient.kt:134, Main.kt:58. No host request or server change occurred. Raw SSH was not used because the preferred daemon/API was not shown to be down. The ordinary run page supplied the concrete provisioning error without host access.

No broader infrastructure repair is authorized by this one-PR/no-production-change brief. The pending gate needs a working executor creation path, then a verified CI result. No user decision or merge is requested here.

Next steps

1. Re-verify current PR/head/check/run state first. Read the archived p69/p72 briefs and findings, then project README/AGENTS and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md if any code/test work becomes necessary.
2. If provisioning has recovered and CI remains in progress, watch it with bounded build-watchman plain PR mode: coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/SearchableBucketWui --pr 2. Do not add --to-merged. Do not delete/re-request a queued run; queued time alone is not evidence of a stuck executor. If the same executor-provisioning failure persists, report it with the archived log; do not blindly rerun, delete the run, or change application code/tests. A separate infrastructure repair effort needs its own authorized scope.
3. On green, record the verdict and finish this verification task, leaving the PR open and unmerged. On a real test failure, obtain the full failure and reproduce it locally before any fix or retry. The existing completed local suite does not need repeating unless head changes or new evidence justifies it.
4. Do not merge, enqueue, publish, deploy, restart or alter production. Any broader infrastructure repair needs separately authorized scope. Complete/archive this handoff after the required verdict is verified.

Reusable operational knowledge

• scripts/test.bash --test . uses the repository's serial local full-suite path. Prefer remote when usable, but do not count PENDING/FILTERED_OUT rows as passed tests. Use one build cache per checkout; check that 8 GB is free before every build/test.
• The ordinary buildtest run page embeds the full build log in a <pre> element; this provided a useful read-only alternative when Fabric certificates were unavailable. The /api/runs summary was marked outOfDate and disagreed with the requeue log, so keep reported state separate from direct host evidence.
• The installed gh metadata edit command queries a retired Projects classic GraphQL field and fails with repository.pullRequest.projectCards. Use gh api --method PATCH repos/CodexCoder21Organization/SearchableBucketWui/pulls/2 --input <JSON-body-file> for description edits. The branch/head is unchanged by that workaround.
• No extra challenge PR was filed because report-challenge automatically opens/enqueues/merges one, conflicting with this task's one-PR/no-merge scope. Tool/queue evidence is in the archived findings.
• TESTING.md and the independent reviews distinguish compatibility coverage from a deterministic proof of upstream timing removal. Keep that distinction in any further claims.

The local watcher was stopped after the terminal verdict. The task-owned build cache and extra evidence worktree were removed. No active local test, probe, watcher or delegate remains from this continuation.