Handoff — Observables: client-generated idempotent command ids & the resilience contract

• Thread date (first message): 2026-06-22
• Author: Claude Opus 4.8, in conversation with the user
• Last verified against main: 2026-07-05
• Type: design thread. One plan PR merged; one design deliverable still open (the idempotency/RpcScope write-up).

---

0. One-paragraph status

This started as an investigation into an infrastructure failure on UrlResolver PR #649's CI check, but the user's real question was "why didn't our e2e tests catch this, and how do we make tests e2e-comprehensive?" That led to a design discussion that converged on: reads are remote observables that never throw on transport failure; writes are RemoteCommandDerivedObservable handles; and — the new part — commands should carry a client-generated idempotency id so they are safely replayable across transient failures. The resilience half is already merged and largely implemented. The idempotency-id + RpcScope (server request-context) half is NOT yet written into any doc — it exists only in this conversation, and this handoff captures it in full so it can be driven to completion. No repository has uncommitted code (see §9).

---

1. How this started — the triggering incident

The kotlin.build (remote) check on UrlResolver PR #649 was RED, but not because of the PR's code. It failed with an infrastructure error mid-build:

UrlResolutionException: Failed to reconnect persistent RPC connection to service 'buildtest'.
Original peer '12D3KooW…scMT' is unreachable:
  - …/p2p-circuit/… : NothingToCompleteException      (relay path dead)
  - …/p2p-circuit/… : NothingToCompleteException      (relay path dead)
  - /ip4/198.199.106.165/tcp/35000/… : Connection refused   (direct port dead)
Also tried 0 alternative peer(s), all failed.
   at BuildRunner.pollUntilComplete(BuildRunner.kt:1339)

The CI runner (kotlin-build-ci) holds a long-lived persistent RPC connection to the singleton buildtest service and polls getBuildRun for the whole build (minutes). buildtest became transiently unreachable mid-poll; the resolver's bounded reconnect gave up and threw; the consumer had no resilience around the poll, so one blip aborted the whole build.

---

2. Diagnosis (grounded, with file:line — verified this session)

Repos cloned & inspected: UrlResolver, UrlProtocol, kotlin-build-ci, PlanRepository.

• Consumer has no resilience around the poll. kotlin-build-ci BuildRunner.pollUntilComplete calls val buildRun = api.getBuildRun(runId) bare at BuildRunner.kt:1367 — an escaping UrlResolutionException aborts the build. Ironically the non-critical calls right below it (renewBuildLease ~1409, check-run progress ~1421, self-keepalive ~1438) are each wrapped in try/catch and "log and continue." The one critical per-iteration RPC is the only unprotected one.
• SynchronizedBuildTestApi (kotlin-build-ci) is a pure passthrough — no retry layer.
• Resolver reconnect factory UrlResolver.kt:2679-2742 (connectPersistentRpc): tries original peer → relay-aware discovery → a bounded ~10s gossip wait → throws if it finds 0 live alternative peers. For a singleton mid-restart that window is too short, so it throws (the prod error's "Also tried 0 alternative peer(s)").
• Test-gap analysis (why e2e missed it):
  1. UrlResolver reconnect tests model the wrong failure shape — e.g. tests/testPersistentConnectionReconnectFailsWhenNoAlternativePeers.kts kills the provider permanently and asserts reconnect should throw (codifies the prod-breaking outcome as "correct"); the success path (testPersistentConnectionReconnectsToNewPeerAfterOriginalPeerDies.kts) only passes because it spins up a second distinct live peer. Neither models a singleton restarting transiently and re-announcing after the window.
  2. All reconnect tests use direct 127.0.0.1 addresses, never the relay-circuit path that failed in prod (p2p-circuit → NothingToCompleteException).
  3. Consumer poll tests (kotlin-build-ci/tests/testBuildRunner_submitAndPollSuccess.kts:125, …getBuildRunNotFound.kts) use an in-process cooperative fake BuildTestApi that never throws transiently — the throw path is never exercised.
  • Structural cause: the broken invariant — "a long-running consumer survives a transient restart of a singleton dependency" — spans 4 repos; each tests its own layer against a stable neighbor, so the seam where the dependency churns mid-operation is untested.
• Key mechanical fact: the observable poll loop already absorbs a reconnect failure as stale-but-recovering — RemoteDerivedObservable.kt:323 wraps each sync round in try { … } catch (_: Exception) { back off; keep polling }, retaining the last-known value. So migrating the consumer from an imperative throwing poll to an observable read dissolves this class of failure by construction.

---

3. The design direction — and what is ALREADY DONE

Direction agreed with the user:

• Reads = remote observable property reads on a live projection: a loaded read degrades to stale-last-known-good and never throws on transport failure (REMOTE_OBSERVABLES.md §3 / §11).
• Writes = RemoteCommandDerivedObservable: a cancelable, status-bearing handle bounded by liveness + cancel(), never a request deadline.

Already merged / implemented (verify against these before re-doing anything):

• PlanRepository PR #51 — MERGED. Added to workstreams/Observables.md: the resilience committed-shape bullet, the "commands subsume the imperative submit→poll→cancel→lease orchestration" framing, a reliability "why" bullet, and a "resilience acceptance fixture" roadmap item.
• Per the workstream header (updated ~2026-06-25) the workstream is now "Implemented (JVM), graduation pending", tracked by epic Observable#87. The JVM url:// path — read contract, runtime proxies, sync protocol, command-lifecycle handles, Compose boundary v1, and the resilience acceptance fixture — is built and merged.
• The command-lifecycle surface is resolved (REMOTE_OBSERVABLES.md §5.1 / §14.8, ~2026-06-25): RemoteCommandDerivedObservable lives in observable-core, exposes val state: DerivedObservable<CommandState> by composition + cancel(); CommandState(status, timestamp?, failure?); statuses PENDING/RUNNING/CANCEL_REQUESTED/ CANCELED/SUCCEEDED/FAILED/NOT_FOUND; no TIMED_OUT ("a lifecycle is never a deadline"), no REJECTED status (it's a FAILED with CommandFailure(reason=…)), no general result (value-returning command = a query, read it back through the projection).
• Ambients are resolved (§14.8 + AMBIENTS.md): wallet + ClockPad are entry-point-established via per-ambient context-parameter types with auto-established neutral roots — NOT context(...) on the projection interface. This matters for §5 below.

---

4. THE OPEN THREAD — client-generated idempotent command ids (not in any doc yet)

This is what the conversation was actively working on when the handoff was requested. It is a net-new extension to the already-resolved command lifecycle. Capture all of it.

4.1 The problem it solves

The resolved lifecycle gives commands reconnect-survival: NOT_FOUND lets a client re-attach to a command it already has a handle for. But there's an uncovered window: "I sent the command and the connection dropped before I got any handle back — did it run?" With a server-assigned id you cannot re-attach (you never learned the id) and cannot safely resend (you might double-execute). A client-generated id that exists before the first round trip is the only thing that closes that window. This is the write-side complement to §3's read-side stale-but-recovering.

4.2 The proposal

Client generates a UUID (ULID) per command before the first send → the server dedups on it → a retry with the same id is idempotent → the operation is robust to transient failures.

4.3 Reasons to be careful (the balanced analysis — these are the real design constraints)

1. Idempotent dispatch ≠ idempotent effect. The framework can dedup the request, but exactly-once effect needs the effect and its dedup record to commit atomically. For externally-effecting commands (provision a droplet, W3Wallet spend, send email, push to GitHub) there's an unavoidable crash window → those stay best-effort. The cultural hazard: once "we have idempotency keys" is in the air, people retry everything and trust it. Scope the guarantee explicitly.
2. Durable, bounded, principal-scoped dedup state is a real subsystem. To survive the server restart we care about, the dedup record must be persisted (fsync before ack), retained for a TTL longer than the max retry window, then GC'd. Too short → duplicates after a long outage (and NOT_FOUND becomes ambiguous again); too long → unbounded growth. And it creates a new fail-open (run without dedup → duplicates) vs fail-closed (reject → hurts availability) choice.
3. Retry-vs-idempotency-key conflation. Reusing an id = "I'm unsure my send landed — give me the canonical outcome (run if new, else return existing)." A genuine retry of a transiently failed op = "do it again." Needs a crisp rule + a transient-vs-permanent FAILED distinction so callers know whether to reuse the id or mint a new one.
4. Id provenance / ergonomics. Three options, can't have all: (a) proxy auto-generates a random ULID per call — transparent, robust to transport blips, but a client restart mints new ids → duplicate; (b) application supplies a deterministic key (build:<sha>) — robust to client restart too, but only the app knows the natural key, so it can't be fully hidden; (c) caller hand-mints — worst (constant → aliasing; fresh-each-time → no protection).
5. Cross-client aliasing = correctness + security surface. Ids must be collision-resistant and dedup scoped by principal (the W3Wallet capability), else client A guessing/reusing B's id reads/replays/forges B's command. Real key = (principal, command-id); principal must be stable across reconnects. Reconcile with W3Wallet's own replay/nonce protection.
6. Single-flight is not optional. Concurrent submits of the same id (client fires twice because unsure) must collapse to one execution with both observers attaching — per-id server-side locking.

4.4 The system-specific win

Most of cost #2 disappears here because the CQRS event log is already the durable store. A state-mutating command appends its effect to the log; key that append by (principal, command-id) as an idempotent/conditional append, store the command's HierarchicalTimestamp in the entry, and a replay returns the stored timestamp (so read-your-writes / §5 stays correct across retries) — no separate dedup table, GC rides log compaction. Strong and nearly free for event-log-backed commands; externally-effecting ones are the residual hard case (#1).

4.5 Carried vs enforced (the crux the user pushed on)

• Carried out-of-band in the request metadata envelope (ServiceHandler.handleRequest(path, params, metadata) already has the metadata: Map<String,String> channel; connectPersistentRpc already threads a metadata map). The function signature stays clean — the id is not a parameter, exactly as wallet/ClockPad are not parameters.
• Enforced at the one layer every server shares — the authoritative durable store (the CQRS event log), via appendIfAbsent(principal, command-id). Never at the transport, never in a single server's memory.

4.6 Multi-server (the user's sharpest question)

• A generic transport/proxy dedup layer is broken across replicas: a retry can land on replica B, whose in-memory "seen ids" never saw A's first attempt → re-executes. Node-local dedup is broken by construction the moment there is >1 server (or one that restarts). This topology is real here — the resolver's reconnect path explicitly discovers alternative peers for a service.
• Works only when the servers share the log/DB: whichever replica handles the retry consults the same log, sees the key applied, returns the stored result+timestamp. Routing to a different replica becomes harmless. Composes with the read side (command yields its timestamp; reads against any replica stay CONSTRAINT_NOT_MET/stale until that replica's watermark catches up).
• If servers are truly independent (sharded, no shared write store) → client ids cannot give cross-server idempotency; do not ship a feature that pretends otherwise. This is the genuine "reason not to" in a sharded topology. Externally-effecting commands stay best-effort regardless of replica count.

4.7 How the server function sees the id (three cases)

Not as a parameter — mirrored from the client side:

1. Default (pure CQRS) — it doesn't. The generated server dispatcher pulls command-id from metadata and wraps the handler in appendIfAbsent(principal, id); the author's method just emit(...)s events and never mentions the id.
2. Semantic key already a parameter (the user's point): e.g. submitBuild(sha, …) — the server reads sha as a normal argument, dedups on it, and ignores the envelope id. This is stronger — it survives a client restart.
3. Escape hatch (external effect / custom store): the author reads the id from an ambient RpcScope and does its own conditional write.

4.8 Converged API shape — optional context(rpc: RpcScope)

The user's landing point, and the recommended shape:

context(rpc: RpcScope)                     // OPTIONAL — declare only if you need request info
fun provision(spec: Spec) {
    log.appendIfAbsent(rpc.principal, rpc.commandId) { digitalOcean.createDroplet(spec) }
}

fun createTodo(title: String) { emit(TodoCreated(title)) }   // needs nothing → plain function

• RpcScope = a read-only, server-side request context: commandId, resolved principal, wallet capability, freshness watermark, caller peer, headers, deadline/cancellation.
• Kept OFF the shared client interface (clients keep POJO-clean signatures; the client can't supply a server-shaped scope; the per-call id is server-observed).
• Forces one architectural commitment: because Kotlin context parameters are part of the signature, a handler declared context(RpcScope) fun provision(...) cannot also be a literal override of a clean fun provision(...) — so server command handlers become a framework-dispatched surface (the generated invoker calls them within the RpcScope context), not literal interface overrides. State this plainly; it revises the "the projection class simply implements the interface" narrative.
• Guardrails: RpcScope exposes, it does not enforce (enforcement stays at appendIfAbsent); keep it a read-only request value, not a service locator / god-object; big testability win — with(FakeRpcScope(commandId="x", principal=alice)) { provision(spec) } needs no transport.

4.9 The auto-UUID vs semantic-key ladder

• L1 — proxy auto-ULID per call (default): robust to transient transport blips; lifetime = the handle; dies on handle GC / client-process restart.
• L2 — server semantic-param key: robust across client restart (re-invoking with same params re-finds the command); server ignores the envelope id.
• L3 — explicit ambient override (withCommandKey("deploy:$releaseId") { … }): cross-restart durability when there's no natural key; still no signature change.

---

5. Reconciliation the next person MUST do (design moved under us)

Since the conversation, the design resolved (2026-06-25) that ambients are entry-point-established via per-ambient context-parameter types with neutral roots, NOT context(...) on the projection interface (§14.8 + AMBIENTS.md). So before writing §4 up:

1. Bundled RpcScope vs per-ambient context types. The resolved ambients decision uses one context type per ambient. The RpcScope proposal bundles server request-info (id + principal + peer + headers) into one server-side context. These must be reconciled: is server-observed request info a single RpcScope bundle, or does it follow the same per-ambient pattern? (Recommendation leaned bundle for server-observed info, since it's asymmetric from the client-provided ambients — but confirm against AMBIENTS.md.)
2. The command-id does not fit "established once at entry point." It is per-call, proxy-generated, and server-observed — unlike wallet/ClockPad. That asymmetry is the crux to resolve.
3. Unify or separate the dedup store from NOT_FOUND bookkeeping. §5.1's "server-side command bookkeeping" (that yields NOT_FOUND) and the client-id-keyed dedup store may be the same store. Decide.

---

6. Concrete next steps (prioritized)

1. [GATED on user] Write the §4 idempotency design into the design doc. Target: REMOTE_OBSERVABLES.md in the Observable repo — as a new §14 open question, or (if the user signs off the shape) a resolved decision + a §5.x subsection. Include §4.1–4.9 and the §5 reconciliation. Add a one-line pointer in the workstream's command bullet and reference epic Observable#87. The user was still "considering" idempotency and had just proposed the RpcScope shape — do not present it as settled without their nod.
2. Decide the open questions in §7.
3. [GATED, larger — implementation] Server-side appendIfAbsent dedup keyed by (principal, command-id) at the event log; proxy auto-ULID into the metadata envelope; RpcScope server context + the optional context(RpcScope) handler surface; the L2/L3 key ladder. Track under epic #87 or a new epic.
4. [GATED — the original prod fix] Migrate kotlin-build-ci BuildRunner.pollUntilComplete from the imperative throwing getBuildRun poll to observing a BuildRun LiveProjection; verify it against the now-built resilience acceptance fixture. This is what actually closes the PR #649 failure class.
5. Docs-alignment check: the workstream's "Current state" text about "no whole-object materialization, no loading contract" is likely stale (a projection/ package now exists in UrlResolver: LiveProjections.kt, LiveProjectionClient.kt, ProjectionServiceHandler.kt, with resolve-immediate / awaitAvailable e2e tests). The workstream header was already flipped to "Implemented"; verify the body matches and fix if not.

7. Open design questions still needing a decision

• Bundled RpcScope vs per-ambient context types for server request-info (see §5.1).
• Dedup retention/TTL; fail-open vs fail-closed when the dedup store is unavailable.
• transient-vs-permanent FAILED distinction so callers know same-id-retry vs new-id.
• Unify NOT_FOUND bookkeeping with the client-id dedup store, or keep separate.
• Is the auto-UUID always on, or opt-in per command/service?

8. Artifacts, links & code anchors

• Merged: PlanRepository PR #51 — plan resilience update (MERGED).
• Triggering failure: UrlResolver PR #649 — the CI infra failure (not a code defect in that PR).
• Epic: Observable#87 — Observables implementation status & remaining work.
• Design doc: REMOTE_OBSERVABLES.md — §3 read contract, §5.1 command lifecycle, §11 failure semantics, §14.8 command-observable surface.
• Plan workstream: workstreams/Observables.md.
• Ambients decision: AMBIENTS.md.
• Code anchors (verified this session): kotlin-build-ci BuildRunner.kt:1367 (bare poll) & SynchronizedBuildTestApi.kt (passthrough); UrlResolver.kt:2679-2742 (reconnect factory); RemoteDerivedObservable.kt:323 (the absorb-and-retry loop); UrlResolver src/foundation/url/resolver/projection/ package; UrlResolver tests/testPersistentConnectionReconnectFailsWhenNoAlternativePeers.kts.

9. Uncommitted / unsaved changes

None. All working trees (UrlResolver, UrlProtocol, kotlin-build-ci, PlanRepository) were clean at handoff. The only committed artifact of this thread is PlanRepository PR #51 (merged). The idempotency / RpcScope work is design discussion only — no code was written — and is captured in full in §4–§5 above so it can be picked up directly.