Handoff: Finish causal proof of delayed buildtest droplet RPC dispatch

Written 2026-10-21 07:38:33 UTC.

RE-VERIFY: This is a snapshot from a read-only investigation. Query ContainerNursery containers, current process command lines and public RPC health before acting. Another session replaced the gateway twice and the coordinator once while r96 was gathering evidence. Re-check PR states with gh pr view <full URL> --json state,mergedAt,headRefOid. A fresh query wins.

Design decisions in force

Task p96, 2026-10-21, verbatim: "Hard rules: no restart, redeploy, kill, or config change of buildtest or ContainerNursery; no deleting droplets or runs; read-only forensics only" and "Never stop to ask the user." No production mutation is authorized by this handoff. No merge, enqueue, publication, timeout increase, test weakening, artificial CPU load or reflection. Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.

Mission

The original user request was: "We need to investigate why https://my.externalbrain.wasmserver.com/ is failing to load and fix the underlying problem." This lane investigated the remote-build blocker: runs 61c5f652 and eb474f49 reported an inconclusive createDropletAsync result, droplet ID 0, and later a provisioning deadline/watchdog failure. Implement a failing-first root fix only if clear and not already owned; otherwise preserve the precise evidence and remaining work. Do not duplicate the existing journal startup fix.

Confirmed observations

1. DigitalOcean did create the machines. Gateway request 86669e56-d0f5-4e20-beb5-64a57b82bd7a persisted READY for 611109130 (run 61c5f652, created 06:47:22); request 73558315-3741-45a8-b3fb-075256055513 persisted READY for 611109541 (run eb474f49, created 06:48:37). First caller failures were 06:46:59 and 06:48:30. A second 61c5f652 attempt created 611111418 at 06:56:57; its caller later failed with no assigned ID. Gateway logs also record cleanup of the first two IDs by existing owners. No deletion was performed by r96.
2. The coordinator started the first create at 06:46:25, before the gateway handled it around 06:47:18. The second began 06:47:56, gateway handling around 06:48:35. Thus the original 30-second async acknowledgement budgets expired before those submissions reached the gateway. Later gateway/provider success is not evidence that the original caller received it.
3. Exact original HTTP POST status/raw body were not retained. Source only returns provider JSON for 2xx, and parsed droplet bodies/READY are durable; the numeric code and omitted original fields cannot be reconstructed honestly. No new POST is authorized just to obtain a response.
4. Provider GET/account and GET/droplets returned 200. At 07:30:59 there were 73 active buildtest droplets, account limit 500. Buildtest control snapshot revision 5 enforces 200; route environment BUILDTEST_DROPLET_LIMIT=30 is only the older bootstrap value. At 07:05 provider count was 107 (meta.total 105 during churn). Do not confuse these observations or treat route 30 as the live limit.
5. The old coordinator's entire 16-worker shared resolver dispatcher was captured at 07:05 inside schedulePeerPrune -> isPeerReachableForPrune -> verifyPeerReachabilityAttempts -> relay probes. Logs show 609–651 failed attempts in a verification pass. The old fat JAR's UrlProtocol2, EffectAwareServiceHandler and HandlerSession class fingerprints exactly match published resolver 0.0.1322. Runtime owning lines: UrlResolver.kt:21223, :21271, :26932; RelayService.kt:8498 / :11379.
6. This is consistent with the already-owned departed-peer/catalogue freshness problem. The closing comment on https://github.com/CodexCoder21Organization/UrlResolver/pull/1265#issuecomment-6287060000 explicitly rejects route bounding/backoff as a mitigation and points to source freshness plus correct registry removal. Current work is https://github.com/CodexCoder21Organization/UrlProtocol/pull/740 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1350. Protocol 0.0.1429 POM is already published (HTTP 200); older PR wording saying it is unavailable is stale. Do not reopen closed 1265/1329 or duplicate the active fixes.
7. Separately, unchanged upstream public test testEffectAwarePendingHandlersLeaveResolverDispatcherAvailable fails against resolver 0.0.1326 at its unchanged 5000 ms probe deadline, then passes against 0.0.1327. The old HandlerSession CountDownLatch.await is EffectAwareServiceHandler.kt:956. This known handler-wait issue is already fixed in merged https://github.com/CodexCoder21Organization/UrlResolver/pull/1311. It is not a complete reproducer of these two production runs and must not be claimed as that proof.
8. Historical admission diagnostics show real charges, including reservationsSum 115, 30 holders, 5 without observed droplets, observed fleet 153 and fresh projection 206/200. The live pre-replacement fleet read was limit 200, active 89, running 17, starting 14, teardown 46, unclassified 26. The complete reservation map is private and not exposed by the served RPC handler; old PID is gone. No current full table or reservation leak is established. Do not clear reservations.

Confidence and remaining gap

Confirmed mechanism at the caller/gateway boundary: both named requests exceeded the acknowledgement budget before gateway dispatch, then completed provider creation after the caller had stopped waiting; cleanup observation also suffered relay transport failures. Background peer verification occupied the complete shared resolver dispatcher in the captured old process, and source-freshness repairs are already owned.

Unproven: the exact dispatch/admission step that delayed these specific two requests. The capture has no per-request mapping from their async submissions to dispatcher admission. Do not claim the complete production root cause or a successful fix. A local public-path causal reproducer is still required. The original numeric POST response and the old private reservation table are unavailable under the retained evidence/read-only constraints.

Production changed outside this lane

• At 06:46:10 gateway was changed from075c84e4 to9330ad25, followed by another management restart06:46:28; final gateway46577 became registered06:46:58, overlapping the first caller deadline.
• At 07:06:47 gateway was rolled back to075c84e4 and restarted again. Live gateway PID1178414, TCP45267, image droplet-service-server-075c84e4c796-20261020T110702Z.jar. Its direct health/status reads completed36/4ms; a public network route read completed207/42ms.
• At07:13:41 the coordinator was replaced. Live PID1240255, TCP37717, image buildtest-server-e7ce3ecb3b62-embedded-0.0.69274771-journal-0.0.18.1-20261021T070333Z.jar. At07:34 its health wasSTARTING/RECOVERING_LOCAL_STATE, elapsed1,223,728ms. This existing recovery is the current direct blocker to new remote runs. The former PID4167738 must no longer be used for diagnostics.
• These actions belong to other sessions. r96 deployed, published, restarted and merged nothing. The new image's intended release is associated with the still-open wrapper/embedded/journal PRs in the table below; r96 did not independently byte-fingerprint the new release. Re-verify that lineage before any later deployment.

Relevant refs

Related active campaign: https://www.handoff.wasmserver.com/handoffs/hf-2026-08-24-fix-startup-compaction-delta-heap-exhaustion-then-revalidate-the-extreme-suite . Its 07:13 report still named the old PID; actual process evidence and the droplets ledger subsequently recorded the replacement. Do not overwrite another owner's body from this snapshot.

The active droplets ledger is https://github.com/CodexCoder21Organization/PlanRepository/blob/wip/droplets-wui-unreliability-2026-10-20/investigations/droplets-wui-unreliability-2026-10-20/LEDGER.md . It records the source-freshness and removal review lanes. Existing relay-read declaration, relay retry and cleanup-retention ownership is also in the table; those do not prove the initial create-delay cause.

Next steps

1. Re-verify live state first; distinguish the original failed-run epoch from the replacement's recovery. Preserve existing runs and droplets. Do not restart/redeploy/configure either production service, increase a limit, clear reservations, or submit new production probes that create resources under this handoff's authority.
2. Coordinate the work with existing upstream owners. Complete a local public-API reproduction of foreground RPC progress while genuine peer-prune verification occupies the shared resolver pool. Use real in-process network nodes or the supported in-memory network fault controls; do not use internal bridges, reflection, fake symptoms, artificial load or machine-tuned counts. Record the exact call path and baseline failure before a product fix.
3. Determine whether that reproducer is resolved by the active source-freshness/removal work. If the mechanism differs, fix it at the upstream owner after the reproducer, with the resource/lifecycle invariant table and independent test/code review. Avoid duplicate code for the already-owned chain.
4. Finish the owning release gates and consumer adoption only after the exact causal proof. A production deployment requires separate owner authorization and a concrete verified artifact; there is no such artifact/command proven to fix the original transport delay yet. No immediate operational reset is recommended.
5. Once the current coordinator isREADY, let the existing authorized remote-validation owner decide how its retained run campaign resumes. Do not re-submit its runs or interpret FILTERED_OUT records as executed test results.

Reproduction and operational details

The evidence README gives portable published-artifact commands for the unchanged handler-wait control. It uses direct JVM test-body execution, not a full kompile suite: old runtime exits1 with the complete timeout stack; new runtime exits0 with R96_EXISTING_TEST_PASS. No product test or timeout was changed.

CN read-only CLI: ~/bin/coursier launch containernurserycli:container-nursery-cli:0.0.20 -r https://kotlin.directory -- containers --url https://api.nursery.wasmserver.com. Its singular container command returned an encoded-route endpoint404, so use containers for metadata. Fabric certificate directory is absent in this environment. The task explicitly allowed read-only SSH with ssh -p23 -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes root@198.199.106.165 for logs/jstack.

Use a freshly queried target port for a bounded SSH localhost forward. A tunnel that has expired or a backend stopped by an outside deployment can produce a connection error orStreamclosed; that is not a valid health result for the replacement. Raw native RPC state is authoritative over the stale WUI projection.

All meaningful new evidence is on the linked remote branch. Compiled classes, dependency caches, credentials, build outputs and41MBof unrelated repeated network stderr were deliberately omitted; relevant complete failures and thread dumps are preserved. The five fresh clones and tracked local diagnostic tunnels are cleaned up on exit. No background CI watcher or delegated agent was started.