id: hf-2026-10-06-make-the-remote-test-service-s-droplet-provisioning-survive-a-slow-ssh-key-grant-the-droplet-service-s-blocking-grant-outlives-the-30-second-sandboxed-call-budget-and-fails-whole-runs url: url://handoff/handoffs/hf-2026-10-06-make-the-remote-test-service-s-droplet-provisioning-survive-a-slow-ssh-key-grant-the-droplet-service-s-blocking-grant-outlives-the-30-second-sandboxed-call-budget-and-fails-whole-runs title: Resolve journal CI submission and finish async grant, typed recovery and ownership tests summary: Journal https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/145 at2678b62: corrected provider fixture targeted1/1 green, both Actions checks pass, required remote failed during interrupted-submission recovery. Local full fallback queued30minutes with zero tests. Final green-head reviews and ready status pending. Original async compatibility, typed recognition, combined ten-shard ownership and requester attribution remain excluded follow-ups. No deploy or worker merge. created: 2026-10-06T05:52:05.241Z completed: null blocked-reason: Required remote check failed during interrupted-submission recovery on2678b62; no full-suite result, final reviews pending. dependencies:

Finish the slow SSH grant fix and its remaining provisioning tests

RE-VERIFY: This checkpoint was refreshed by w6 on 2026-10-08. Items 1, 2, 4 and 6 are earlier snapshots, explicitly excluded from the round-two continuation; reverify them before work. Recheck every linked PR and remote branch before acting. No worker may merge, enqueue, close PRs, complete this handoff or deploy. The sweep orchestrator owns those decisions.

Mission and why

The owner asked the sweep to finish useful open handoffs and close obsolete ones, while skipping deployment and disk-exhaustion work. This handoff still has desirable unfinished engineering work: make a slow SSH grant survive the sandbox call budget in its owning layers, preserve ownership and diagnostics, and add public tests. It is not ready for closure or completion.

Current item-by-item state

Async SSH grant: unfinished. https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/144 @ c7f8283dfaf9e4e8f7e21e22fac4e9168b769ea5 is OPEN. Actions build passed and all-tests failed; the required remote suite was queued with zero runs. Its own description records O7 legacy Java-IO cause-chain compatibility and O8 erased List<Interface> conversion. Current UrlResolver source still reconstructs a raw host object before selecting implemented-interface proxies. Fix the conversion upstream, with reliable public consumer/upstream reproducers first; publish a fresh artifact version, verify its bytes by SHA-256, then update the consumer pin. Do not overwrite a claimed version or imitate the conversion downstream. PR was not modified by w5.
Typed terminal-generation recognition: unfinished. BuildTestServerService current main still matches an English resolver error in BoundedDropletService.kt. Later merged DTO detachment and local-generation discard fixes do not implement this typed contract. Start with the current public retired-generation reproducer, keep unknown creates durably owned, and fix the owning resolver/wrapper contracts. No wrapper changes by w5.
Fresh lifecycle journal capture: blocked on required CI submission. https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/145 is pushed at2678b62d9deb45df92b074879af19c9e2ae04445. Nine public scenarios failed against unchanged baseline before the fix; targeted12/12+2/2 passed. Round two corrected the paired-action fixture provider total1->2 and reran it:1/1 passed, original assertions/counts/budgets unchanged. Regular catch-up captures one current action page during historical recovery waiting/terminal state; fresh or changed rows remain uncertified, cannot discharge the old gap, and do not rewrite the retry record. Historical-only ingestion preserves zero-request waiting behavior. Both Actions checks passed; required remote failed during interrupted-submission recovery. Final post-green reviews are pending and PR remains draft. Coordinate with saved-boundary/head-growth recovery work; do not duplicate this backoff capture.
Combined ten-shard ownership test: unfinished. BuildTestEmbedded contains separate lost-response ownership and bounded-grant tests, but no evidence yet proves the specific ten-shard lost-submit plus grant-timeout combined scenario. Existing investigation refutes abandoned droplets: all 60 were reclaimed. Add the exact public combined scenario without inventing an abandonment fix.
Deploy merged coordinator fix: skipped. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1266 is MERGED (2026-10-06 03:02:46 UTC; all five checks passed). Production deployment/restart is excluded by the sweep. No production changes by w5.
Restart requester attribution: unfinished. Current ContainerNursery AdminContainerProvider passes only route and timeout to restartContainer; that method logs those values without requester identity. Read-only review of auth/facade/event paths began, but this does not establish the original actor or prove all possible logs lack it. Continue owning-layer investigation and public requester-recording tests. Do not restart anything.

Round-two validation and remaining scope

This continuation is limited to the journal PR. The orchestrator instructed that async grant compatibility, typed terminal recognition, combined ten-shard ownership and requester attribution remain follow-ups; no work on those items was started here.

The two saved flakes positively occurred on unchanged main0eb32bf513f8e977c3538cb6e28f7916bd340353 using ten copies each of the exact public scenarios. Readiness queue observation failed1/10 first attempts (queuedReads expected0, observed1); persistent RPC stream opening failed2/10 (UrlResolutionException/ConnectionClosedException). The same probes on journal source9d316402 failed1/10 and0/10 respectively. Production files and selected scenarios match final2678b62; the only later change is the unrelated paired fixture total. An unchanged canary neighbor also ran once; baseline first attempts18/21, candidate20/21, final21/21 on each after automatic runner retries. This is evidence of pre-existing failures and no observed worsening, not a flake fix or reliable-reproducer claim. Rates below50% do not authorize a guessed fix; none was attempted. Preserve all existing budgets/counts/assertions in any separate investigation.

Final2678b62 Actions checks SUCCESS, required kotlin.build(remote) FAILURE at https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/runs/113308714183. Exact summary: “Remote submission was interrupted before its run ID was recorded. Recovery found the accepted run and is completing its cancellation.” Accepted run https://buildtest.kotlin.build/run?id=22ed1f71, no failed test or stack trace supplied. Local fallback spent30minutes queued with zero tests started and was stopped; no full-suite result exists on the corrected head. Local full-suite work is serialized under the sweep fullsuite.lock, with a30-minute execution cap. If local cannot complete within30minutes, the continuation authorizes required kotlin.build(remote) as the full-suite gate. Never accept that required result red or missing, and never requeue hoping for green. The run-list projection reported outOfDate=true, refreshFailing=true and unreconciled membership; it cannot establish current-head execution. This observation alone does not prove that all execution stopped.

After green required checks on final2678b62, conduct both independent review passes against that head, mark the PR ready and return the merge decision to the orchestrator. Do not merge, deploy, close or complete this handoff.

Refs and durable work

Repo / branch	Remote SHA	PR / purpose
https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/fix/w5-fresh-lifecycle-events-during-backoff-20261008	2678b62d9deb45df92b074879af19c9e2ae04445	https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/145
https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/wip/w5-lifecycle-journal-checkpoint-20261008	f2adac0017e4f8c538e715c98679bdcb6cb7fef5 (later final findings checkpoint may advance this evidence-only branch; recheck HEAD)	Evidence only, never merge; candidate patch, restore instructions, incremental findings, complete failure/console records
https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/wip/w6-journal-validation-20261008	56306842728c3a5a6474851358c6aa7c5180dcfa	Evidence only, never merge; exact public probes, full first-attempt stack traces and logs, incremental findings
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/laneQX17-provisioning-incident	6baf8e30d0eb1ef715b3729540359965c138b747	Original sanitized evidence, never merge
https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/timing-card-lane-findings-2026-10-05	53255335ab7f09c99fa801d263bc355c641f996b4	Original timing evidence, never merge

Nothing published or deployed by this worker. Read-only current hosting image names were buildtest-server-e95a02b6-20261005-dep10.jar and droplet-service-server-0.0.82-r-derivation-fix-20260827T2205Z.jar. Names are not a byte-identity claim. The original snapshot's byte-identity and throughput were not repeated as current facts.

Closure state

The orchestrator already closed superseded https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/130 on2026-10-08T11:04:52Z; round two verified CLOSED. It truncated synchronous grant at the existing caller boundary instead of implementing a service-owned asynchronous operation. No further PR closure recommendation; no disk-exhaustion step identified. The original unfinished non-deploy items prevent closure of the whole handoff.

Tools and reusable notes

Fresh clones under the worker workspace; do not rely on their continued existence. ARM Coursier 2.1.24 asset returned 404 (already tracked by https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/135). Link ignored jars/coursier to a working installed launcher. The required long cwd produces a NAME_MAX cache leaf; use the same ignored pinned launcher directly with bash jars/KompileCli.jar --cache-location /tmp/w5-kompile-cache --local --test <scenario> after fetch/rebase. Prefer remote mode when healthy. Full-suite fallback used a 75-minute tracked cap and /tmp/handoff-sweep-local-full-suite.lock. No source timeout or stress count was changed.

Read Testing Architecture, Engineering Philosophy and handoff triage guidance fully. Droplet README/ARCHITECTURE and wrapper README read. Coordinator README overview/build/ownership sections read; do not claim its whole long README was read. Meaningful tool friction is recorded in evidence; report-challenge's auto-merge invocation was skipped because this brief forbids merging.

Sweep verdict and next dispatch

Journal verdict BLOCKED: repair/resolve shared CI submission recovery, obtain successful required full suite on2678b62, then conduct both final reviews and mark ready. No CI requeue was attempted; no code failure was supplied by the check. Original non-deploy items1,2,4,6 remain unfinished; item5 is skipped deployment. The orchestrator owns every merge and handoff completion decision. No worker publication, deployment, restart, merge, enqueue or PR closure occurred.