id: hf-2026-10-06-find-why-a-freshly-created-render-droplet-returned-404-from-digitalocean-13-seconds-after-its-first-successful-read-and-make-the-droplet-service-record-enough-to-diagnose-the-next-occurrence url: url://handoff/handoffs/hf-2026-10-06-find-why-a-freshly-created-render-droplet-returned-404-from-digitalocean-13-seconds-after-its-first-successful-read-and-make-the-droplet-service-record-enough-to-diagnose-the-next-occurrence title: Finish queued required CI and final reviews for provider evidence PR summary: Final source 6d8f843 is pushed and affected targets pass. Required remote run abacc46f is queued without an executor after a 30-minute watch; resume existing run, then both formal reviews. Only orchestrator merges; skip all deploy and live-repair steps. created: 2026-10-06T02:54:40.700Z completed: null blocked-reason: Required final-head remote run abacc46f remains queued without executor assignment after a 30-minute watch; formal reviews await required green. dependencies:

Finish queued required CI and final reviews for the provider evidence PR

Re-verify the PR, branch heads, required checks, and current main before acting. Only the sweep orchestrator merges, closes PRs, or completes this handoff. Skip every deployment, restart, route change, image rebake, and live recovery repair.

The user asked the sweep to finish useful handoffs and skip disk-exhaustion work and deployments. This handoff contributes useful provider diagnostics and an independently reproduced history pagination fix. Worker w3 is driving source validation and review only.

Established provider outcome

For droplet 606485859, retained per-droplet actions show create 3452050426 errored at 2026-10-06T02:29:16Z and destroy 3452050552 completed at 02:29:18Z. The earlier successful lookup at 02:29:14.573 returned new. The authoritative 404 occurred at 02:29:27.205, before renderer cleanup at 02:29:27.308. The provider schema does not expose the deleting actor or deeper provisioning reason. That part of the original premise is refuted by the returned evidence. Do not add renderer retries or longer waits.

Source changes

Successful creates retain provider request/action identities with atomic file replacement. A new-droplet lookup returning 404 within ten minutes of successful create receipt reports matching create action status/completion while preserving the typed original 404. Missing, failed, or unrelated evidence cannot replace the original outcome. Record-write failure emits a specific algebraic warning while preserving the known created droplet ID. Diagnostic action reads inherit the existing absolute readiness deadline. Real HTTP, isolated sandbox-client, and isolated WebCron tests cover restart and all creation paths.

Initial history pagination previously consumed a page shifted by head insertion while preserving the old total, then saved a boundary ID inconsistent with its offset. This was deterministically reproduced twice. The fix stops before consuming the shifted page and preserves its valid prefix for the next resumed walk. Attributing the historical live recovery mismatch to this particular mechanism remains an inference. Independent https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/145 addresses fresh collection during historical backoff; do not wait on or modify it.

Round 2 progress and validation

Draft https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/146 now holds final remote source 6d8f8437be081ee7849580541b274f83885a1da9. Recovered initial candidate 8d4328a31da868276fe36bb5fbee515ee212000b exactly matched saved candidate 9aa35028547980db2b170591cb6c98201e278a60. Its 20 targeted scenarios passed in six small batches. Its initial local full suite completed 288/288 eventual passes but exposed a deadline-test first-attempt failure, which was not accepted as a clean gate. The initial head's required remote check completed SUCCESS; this does not cover the final head.

The existing deadline test sampled queue cleanup immediately after waking an asynchronous worker and looked up that worker after it could already exit. Genuine 20-concurrent-by-20 creation batches reproduced the queue assertion in all three attempts (20, 17, and 19 failures out of 400); a separate 20-caller probe reproduced the missing-worker lookup in all three attempts. The corrected test captures the worker through the injected Clock, joins with manual time fixed at the deadline, then applies the unchanged queue, HTTP-count, terminal-state and 5000ms assertions. It now retains 400 genuine cases and closes resources on failures.

A deterministic new boundary test advances the provider clock by 1ms during successful POST, then looks up after exactly ten minutes from response completion and restart. It failed 3/3 baseline attempts because the record used the pre-request timestamp. The fix uses successful receipt time. The fresh-cache test now passes; the expiry and original readiness-deadline neighbors and 400-case admission test also pass without retries.

Final remote source 6d8f8437be081ee7849580541b274f83885a1da9 and its complete diff are durable on the source branch and as round2-current-source.patch on the notes branch. Fresh-cache tests verify the exact compiled production artifact, unchanged by the last test-only commit. A missing-identities scenario now also verifies a complete typed 404 with explicit unavailable request/action identities and no action query.

The final local full-suite attempt waited 30 minutes for the shared flock without starting any tests and was stopped under the continuation rule. No final local full-suite pass is claimed. The final-head required remote full-suite run is abacc46f, tracked once through build-watchman with a 30-minute cap; expected suite count is 290. Both formal review passes remain after that final-head required gate.

Durable refs

Source PR: https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/146
Remote source branch: https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/fix/w3-provider-create-evidence-20261008 @ 6d8f8437be081ee7849580541b274f83885a1da9
Notes/evidence: https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/wip/w3-droplet-404-evidence-20261008 @ 922db8a078ff8e5e843fb7755d9f9f5dde88ecfe (re-verify latest head). Never merge notes.
Earlier red-test checkpoint: https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/tree/wip/w3-lifecycle-head-growth-20261008 @ 3c60176b44ab29eafc621f4c125f4789868b1c9f
Original renderer evidence: https://github.com/CodexCoder21Organization/BuildTestWui/tree/wip/laneQA21-golden-404 @ ad7c369684f5b58b45a4ea19debf3bb9173c6ac0
Original renderer context: https://github.com/CodexCoder21Organization/BuildTestWui/tree/wip/laneQA18-finish @ 48e8f5000e8715a25f944fb0875b9df4d39f07b7

The notes branch retains provider action/cursor evidence, complete round 1/2 findings, failed baselines, passing test XMLs, source patches, and static artifact bytecode checks. Do not redo the established provider diagnosis.

Tooling and remaining gates

Use the pinned ignored Kompile launcher directly with a short cache path: the script derives an overlong cache component from this handoff path. A reused cache also returned a service JAR from 10:49 after an existing source-file edit; static bytecode proved the old timestamp call. Use a fresh task cache and verify the compiled artifact when recovering. The fresh artifact SHA-256 is f9e2b9c51c481402e20e477ad25d73b78903feee20910867065b0194a57377cf and its bytecode uses Clock.currentTimeMillis at successful receipt. No artifact was published by this effort.

Watch the existing final-head required kotlin.build (remote) run abacc46f with build-watchman without merge flags. Never rerun a failure blindly or accept an older head's green check. Then finish test-comprehensiveness and adversarial reviews on final head 6d8f8437be081ee7849580541b274f83885a1da9 and stop for orchestrator decisions. No downstream pin bump is needed. Skip all deployments, live repairs, and disk-exhaustion work. The installed gh pr edit command queries retired Projects-classic metadata; body editing succeeded through REST PATCH with exact response-body verification.

Round 2 terminal state

Worker w3 stopped the local watcher after approximately 30 minutes of final-head CI waiting. Existing required run abacc46f is PENDING with scheduledAt=0, dropletId=0, 0 passed / 0 failed / 290 expected, and no error message. It has not started the suite. The final-head bld-all-tests check is success and the optional kompile-remote-build is neutral; these do not replace the required remote gate. Verdict BLOCKED on executor assignment, with no source-test failure observed. No source changes remain unpushed. All owned background tasks were reaped.

Next: claim the handoff and the DigitalOceanDropletServiceServer-146 lock; re-verify final PR/source/main; resume build-watchman for the existing run without merge flags or a rerun. On required green for the unchanged final head, perform both formal review passes, address real findings if any, and stop for the orchestrator to decide whether to merge. Preliminary final-head review is saved, but no formal post-green review or READY verdict is claimed. If current main advances, rebase and validate any resulting new source head rather than treating old-head green as sufficient. No downstream pin is required and all deployment/live-repair steps stay skipped.