Decide the first-cause investigation scope and preserve the live facade release

RE-VERIFY: Snapshot 2026-10-04T16:34:20.901491+00:00. Recheck claims before update; check PR URLs using gh pr view --json state,headRefOid,statusCheckRollup,mergeStateStatus,mergedAt; verify branch SHAs using git ls-remote; GET /containers and authorized read-only host ps/sha256sum establish running artifacts. A deployment or a successful call after this snapshot does not rewrite historical causality.

Mission and remaining operator decision

The request is to “establish why coordinator-to-droplet-service RPCs fail through the ContainerNursery URL facade and fix the fleet-wide provisioning failures.” On October 4, the supervisor supplied live 30-second create-call failures for runs 1f23c644 and 0ffb0828. This continuation verified missing provider receipt for particular calls and tested the handoff's proposed remedy against deployed code. It did not find the first closing or read-pause decision.

NEEDS_USER — scope decision: Consolidate the first-cause continuation with the already active ordinary-path local experiment, or commission a dedicated diagnostic CN release after its unresolved release gates are cleared? Recommend consolidation with the real-component experiment and keeping this handoff open. A restart coincides with recovery, but is not a verified repair. No already-merged change or deployment artifact is proved to resolve the captured stall. Do not approve a blind main deployment as though that proof exists.

All merge, enqueue, deploy, publication and completion decisions remain with the supervisor. This lane performed reads and evidence checkpoints only. The coordinator-finalization defect in which a passed shard is released at 600 seconds remains excluded.

Findings and actions, in order

1. Lost delivery is observed. On October 4, create buildtest-0ffb0828-a71841c begins 13:43:32.737 and fails with ambiguous EOF after write at 13:43:35.555; no provider receipt is in the saved stdout/structured window. Create buildtest-1f23c644-a71845a begins 13:49:41.779 and reaches the caller bound at 13:50:11.941, also with no provider receipt. Earlier creates for 1f23c644 were accepted; a71843d's status reply ends in EOF at 13:46:12.433. This narrows the failed path to outside the observed provider handler. It does not identify the transport's first close. Full surfaced exception text and following relevant stacks are checkpointed.
2. The old bridge remedy is inapplicable to ordinary calls. Live CN lacks the BackendRpcApplicationException guard merged at https://github.com/CodexCoder21Organization/ContainerNursery/pull/634. However its handler obtains BackendPushBridge only when metadata.pushSupported is true. Deployed coordinator factory → empty-header resolver overload → sandbox send → persistent request metadata never adds this key. Its catch therefore has no bridge object to close on this path. This refutes the guard as a complete repair of the observed ordinary calls; the merged push-enabled test remains valid and should be retained. The same bytecode chain is present after the 15:55 coordinator replacement.
3. Provider and facade stacks differ. Live CN admission/stream classes match protocol 0.0.546; provider classes match resolver 0.0.1092/protocol 0.0.470. The provider main upgrade at https://github.com/CodexCoder21Organization/DigitalOceanDropletServiceServer/pull/142 is undeployed. Its fail-first P2P admission test is useful but does not exercise the deployed localhost TCP backend leg; do not attribute the current facade root failure to that test. A fresh provider 404 at 15:22 follows an actual coordinator deletion, so keep accepted-and-deleted creations distinct from missing delivery.
4. Retained facade evidence cannot identify the first close. Managed logs rotate at 50 MiB with four archives. The October 4 native captures cover approximately 14:20–15:02 and 15:11:30–15:59:32.286, after the 13:43–13:50 anchor. A bounded structured capture at 15:18–15:22 contains 4,662 facade records, 213 provider-port forwards and one getDroplet 404, without any bridge-close record. Generic unknown-peer admission-retirement records do not identify a droplet root. October 2 first-close history likewise rotated away; historical caller/provider evidence remains on the two L6 branches below. The older statement that a provider 404 “fits the bridge bug” must not be carried forward as proof on this ordinary path.
5. Thread snapshots have limits. Standard jstack misses virtual handlers; a later full JSON dump includes 642 threads. It shows no contemporaneous droplet handler wait, but cannot settle an earlier close. At the authoritative old coordinator jstack header 15:08:07, a TCP attachment reader is waiting in FrameAttachmentLease/SizeAwareReservationScheduler, while the separate digitalocean reply reader is waiting for a frame length. Already-merged https://github.com/CodexCoder21Organization/UrlProtocol/pull/619 fixes whole-pool sequential attachment reservations; https://github.com/CodexCoder21Organization/UrlProtocol/pull/620 fixes untimed bounded-reader parse admission. Source descriptions name real bugs, but no retained evidence links those waits to this droplet close. No host-capacity explanation is established.
6. Another operator replaced the coordinator. PID 4083911 started 15:55:09 with buildtest-server-302856bf-20261004-deploy6.jar, SHA256 41462ced20515068097e896066957d33a7e40876642744abfb43f32ba0d802e0. Its embedded 0.0.69274749 pin is merged at https://github.com/CodexCoder21Organization/BuildTestServerService/pull/394, merge 1c1c6676d1b1b47da3d3d38c219d535909d859f9. Old/replacement RemoteDropletService, ReopenableSandboxProxy, SandboxGenerationHealth, resolver and core protocol classes are identical; embedded coordinator class differs. The core protocol hash matches published 0.0.527. This lane made no production change.
7. New calls are recovering, not yet a proved repair. Generation-22 caller log spans 15:55:09.487–16:17:10.365. All 24 new create calls have acknowledgments, with independently matching provider request IDs and all 24 provider READYs. The earlier caller capture has 19 Droplet created records; the five recent provider READYs fall after that capture. The errors published near 16:04 refer to persisted old attempts admitted before 15:55. The public /api/runs endpoint returns HTTP 200 but outOfDate=true, refreshFailing=true and upToDateAsOfEpochMs=1791101091862. Its rows cannot certify present provisioning. New success does not distinguish cleared connection state from changed embedded behavior.
8. The release prerequisites remain concrete. The live CN filename revision a2fc6415da7c9fec508ff8337f68e59ec7eff4dd belongs to unmerged outbox work. Do not replace it with main's protocol 528 pin and lose live behavior. The faithfully rebased outbox branch at 856eba5fcd18315586c678b0f93f435abe6785b5 failed the existing 9 MiB response gate under resolver 1259's 7 MiB assembly budget. Keep https://github.com/CodexCoder21Organization/UrlResolver/pull/1161; its required remote checks remain FAILURE. Published protocol 0.0.566 includes raw-byte marker conversion, parse fixes and close/admission diagnostics from https://github.com/CodexCoder21Organization/UrlProtocol/pull/623. No validated CN diagnostic release artifact was built here.

Running artifacts and deployed-but-unmerged state

Relevant PRs and refs

Evidence: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/L22-rpc-evidence-20261004/handoffs/artifacts/L22-rpc-evidence-20261004 . Independent related ordinary-path experiment: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/BS4-allocation-reply-loss-2026-10-04/investigation/BS4 . It was hypotheses only at inspection; no result is assumed and no other handoff was changed. Settlement-design work is independently preserved at https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/DS1-evidence-2026-10-04/investigation/DS1 .

No PR or branch is recommended for closure. Salvage both merged scoped tests, raw-forwarding/outbox work, upstream budget fix and timestamped evidence. Incorrect broad incident attribution does not make those contracts unnecessary.

Validation and two review passes

Fresh tests/builds: 0 passed, 0 failed, 0 skipped. This is an evidence-only continuation with no reliably reproduced new owning-layer defect. Production observation counts above are not tests. Prior test evidence remains historical: bridge baseline 0/1 and fixed neighbor batches 4/4; provider admission baseline 0/1 and upgraded 2/2, sandbox neighbor 1/1.

Test-comprehensiveness pass: inspected real CN/TCP/P2P setup, held sibling response, full application-error assertions and cleanup. The bridge scenario explicitly requires push metadata. Provider admission scenario is P2P, not the live TCP leg. Protocol parse tests force exact resource conditions and preserve accounting; retain them and the 9 MiB facade test. No exact first-close reproducer exists.

Adversarial pass: checked the deployed metadata chain, class/version identities, concurrent deployment and persisted timestamps; corrected platform-only dump interpretation; separated missing delivery from accepted/deleted droplets; retained the failing large-reply gate and stale projection limitation. No timeout increase, retry, fallback, weaker assertion or speculative patch.

Next steps

1. Re-verify current state first, then supervisor choose the continuation owner and diagnostic scope stated above. Recommendation: keep this handoff open for the first cause and consolidate the pushed BS4 public-API experiment.
2. Run real ordinary sandbox client → real CN URL facade → real TCP provider in separate JVMs with production protocol pair 527/546/470 and matched-current controls. Hold a status reply after provider receipt. Force separately a sibling application error with empty metadata, caller cancellation/original budget expiry, exact shared parse-pool exhaustion by a valid incomplete attachment, and the actual facade idle boundary. Record receive/return/frame-write/root identity and first close; assert full held reply and no replay of ambiguous create. The forced condition must fail first at its owner before any source patch.
3. If the local matrix still cannot identify the production first close, supervisor/operator choose archived identity-correlated records or an approved diagnostic rollout. A diagnostic CN build must reconcile the live outbox/raw forwarding, main guard and 9 MiB contract, and carry published protocol 566 close/admission records. Target the CN router listening on 35000, forwarding url:digitalocean-droplets: to TCP 44207; consider the coordinator's 527 client records too. Full required CI and the existing large-reply gate precede any production authorization. There is no tested artifact/version/route combination proved to repair this incident today.
4. Once one first cause is observed, write the invariant table, deterministic consumer and owning-layer fail-first tests, fix upstream, and prepare rebased main-target PRs with remote gates green. Supervisor owns every merge, publish, deploy and completion. Keep coordinator-finalization work with its existing lane.

Operational knowledge

Read-only SSH uses the supplied ~/.ssh/cn_host_diag key to root@198.199.106.165 port 23; no write credentials are available. GET /logs/server has a short shared ring (the fresh capture covered only several seconds); managed log archives retain native WARN/ERROR history. Embedded child timestamps can be artificial; date retention by native CN prefixes. JAR filenames alone do not prove contents: class comparison and disassembly establish the relevant deployed contracts. Standard jstack excludes virtual handlers. Full raw logs/configuration/jars and unrelated traffic are deliberately not committed; sanitized relevant records and full surfaced stacks are remote.

Use PATH=/code/ws/bin:$PATH. The JVM cs launcher works; /usr/local/bin/coursier is wrong-architecture. Do not run full local suites or large local builds on this shared 6 GiB host. Long remote tests use tracked, bounded build-watchman without --to-merged. Do not rerun a red check as diagnosis. Claims unexpectedly disappeared twice despite recent claim/report; explicitly recheck and reclaim only after confirming no other live agent.

Required readings retained: PlanRepository handoffs/README.md (Format; Triaging handoffs), DocumentationRepository PHILOSOPHY.md (root cause; upstream fixes; refuted hypothesis), architecture/TESTING.md (public API hermetic tests; deterministic conditions), CODE_REVIEW.md (Claims That a Host Is Overloaded), and read repository README/AGENTS. Their old merge/delegation advice does not override this lane's no-merge/no-production-mutation rule. report-challenge's automatic merge conflicts with that rule, so logging/lease friction is recorded in this evidence.