← Priority list

Handoff: HCF peer-scoped propagation �� protocol half landed; resolver half blocked on a newly root-caused zero-candidate re-resolution defect

2026-07-27: 9 PRs merged (UrlProtocol half complete + 0.0.413/0.0.414 published; both browser-test defects fixed). Resolver gate failing unrelated PRs on a shared netlab service fault; the resolver defect it exposed is ROOT-CAUSED and FIXED in #878 (reviewed, signed off, parked out of the queue with #876/#877/#862 to avoid burning queue time). Causation from #878 to the CI failures is NOT established. Browser assertion still unverified and its expected text is probably stale vs the merged fix �� detail inside.

hf-2026-07-22-hcf-peer-scoped-propagation-fix-ci-blocked

Edit handoff

Title Summary Markdown body

# Handoff: HCF peer-scoped onboarding propagation — protocol half landed, resolver half blocked on a shared-service fault; a fix for the resolver defect it exposed is reviewed and parked

## RE-VERIFY BEFORE ACTING — all state below is a write-time snapshot taken 2026-07-27 21:05 UTC

Everything in this document was live-checked at write time and will drift. Authoritative re-checks:
- PRs: `gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt`
- Merge-queue batches: `gh api "repos/CodexCoder21Organization/UrlResolver/actions/runs?event=merge_group&per_page=5"`
- A failing check's real cause: the **check-run output summary**, NOT the buildtest `test-results` API (see Reusable knowledge — that endpoint silently truncates).

## Mission

Land peer-scoped address propagation so that a node behind a home router can advertise a directly-reachable, cryptographically-signed address, removing the forced relay detour for return traffic. Protocol half is merged and published; the resolver half is code-complete but cannot clear its gate.

## What was found and done (2026-07-27 session)

### Landed (8 PRs merged, all verified MERGED at write time)

- https://github.com/CodexCoder21Organization/UrlProtocol/pull/400 — peer-exchange authenticated responder-self record (the propagation protocol half). Published `foundation.url:protocol:0.0.413` **manually** — nothing auto-publishes on merge in that repo.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/404 — binary-attachment bytecode serving. Published `0.0.414`.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/422, https://github.com/CodexCoder21Organization/UrlProtocol/pull/428, https://github.com/CodexCoder21Organization/UrlProtocol/pull/429 — three CI-flake fixes that de-flaked the merge queue.
- https://github.com/CodexCoder21Organization/W3WalletDaemon/pull/127 — relay nested-param structural conversion. UrlProtocol issue #399 closed; root cause was two-layer (resolver relay decode ≤0.0.595 + daemon stringification), never in UrlProtocol.
- https://github.com/CodexCoder21Organization/W3WalletJvmServerSideCoinCollectorDemo/pull/101 — demo web server emitted absolute redirects built from its own cleartext scheme paired with the TLS terminator's port, so a browser navigation became cleartext aimed at the TLS listener and returned zero bytes. Fixed by configuring the connector for relative redirects. Reproduction flipped 20/20 failing → 0/20.
- https://github.com/CodexCoder21Organization/W3WalletDaemon/pull/129 — daemon refusals were swallowed by the typed sandboxed-proxy client. See "Swallowed refusals" below.

Closed as obsolete: https://github.com/CodexCoder21Organization/UrlResolver/pull/828 — superseded by already-merged #835 (code-verified, all 5 tests present on main).

### Swallowed refusals (merged) — and a live consequence still outstanding

The daemon reports application-level failures **inside the RPC result** as `{"success": false, "code": ..., "message": ...}`, never via the transport error channel — so the transport correctly does not throw. The SJVM client never inspected that envelope: `createCapability` returned the literal String `"null"`, `get*` returned `null` (indistinguishable from genuine absence), `list*` returned empty, `update`/`delete` returned `false`.

Both resolver layers were **verified correct** and are NOT at fault: `UrlResolver.kt` `PersistentRpcConnection.doSendRequest` does throw on a real error response, and `TypeMarshaling.kt` `fromJvmResult` does throw on a sandbox exception.

Two sandbox constraints found empirically and now documented in code + README:
- SJVM cannot construct a client-defined `Throwable` subclass (`NoClassDefFoundError` in its constructor) → use stdlib exceptions with descriptive messages.
- The sandbox stdlib has **no `StringConcatFactory`**; Kotlin on JVM 9+ compiles every string template to an `invokedynamic` against it. **This is why the original client could build no messages at all.** `compileClientKotlin` now passes `-Xstring-concat=inline`.

**Outstanding consequence, verified in code, not yet addressed:** `list_capabilities` over `url://` is refused *unconditionally*. `WalletRpcHandler.remotePermissionFallbackDenial` returns a denial whenever `originAuthenticated == false`, and `UrlProtocolService` always passes `false` — so the `hasPermission` table check beneath it is unreachable on that transport. The demo has therefore been rendering an **empty coin list while reporting "connected"**. After #129 it surfaces an error instead (correct, and what the browser test asserts), but the demo likely wants an explicit explanatory handler rather than a raw error string. `WalletDemoServer.kt` call sites: `:152`, `:683`, `:745`, `:798`, `:848`.

### The current blocker — root-caused this session

Three unrelated UrlResolver PRs failed on network-lab tests within one hour, none of which touch networking. Signature, from the raw job log of merge-queue batch run `30290739996` (job `90059615324`, failed 18:46:14Z):

```
AmbiguousRpcRequestException: RPC request 'uploadFile' to service 'netlab-hosted' has an ambiguous
  outcome because its transport failed after the request write began. ... Original transport failure:
  Persistent RPC connection to service 'netlab-hosted' was closed while requests were still pending.
[test] reconnect for upload retry failed: UrlResolutionException: All 0 peers for service
  'netlab-hosted' failed to connect. Errors: []
TESTS FAILED (1203/1206 tests completed successfully, 3 failed)
```

**RULED OUT — do not re-investigate.** This is *not* the known un-upgraded-consumer instance of the earlier dead-transport defect. Established on both source pins and by fingerprinting the **deployed** jar (HCF was unusable — `~/.config/hardware-control-fabric/` absent on this box — so the running jar was fingerprinted from live stack traces in ContainerNursery logs against `javap -l` line tables of published artifacts):
- Deployed resolver contains `ambiguousWriteFailure`/`ambiguousTransportFailure`, first present in 0.0.671; layout closest to 0.0.677. The running server is *newer* than what main pins.
- Deployed protocol matches the 0.0.402–0.0.409 line set (`NettyStreamController$NettyOutputStream.write` at `Libp2pHostFactory.kt:3230`).
- `NetLabManagerServer/build.kts:54-55` pins resolver 0.0.670 / protocol 0.0.401; threshold crossed in commit `731e76b` (PR #106) on 2026-07-19.
- Every party is past resolver 0.0.638 / protocol 0.0.374. **There is no version-bump fix.**

**ACTUAL MECHANISM.** Transport deaths on these legs are endemic but *survivable* — the retry loops absorb them and 77 uploads succeeded in the failing run. The fatal step is what follows: after the failed peer is evicted, re-resolution returns **zero candidates with an empty error list**. Zero peers AND zero errors means nothing was attempted — the resolver believed it had nothing to try. The defect is the missing guard that should keep a bootstrap/last-known peer available so re-resolution can always attempt something after an eviction.

A second, distinct leg fails the same way concurrently: in a 91-second window of live manager logs (19:14:31–19:16:02Z) the deployed manager emitted 12 `AmbiguousRpcRequestException` traces and 61 `Persistent RPC connection to service 'netlab-worker-<uuid>' was closed while requests were still pending` lines, all on `uploadFile` via `WorkerClient.forwardRequest` (`src/netlabmanager/WorkerClient.kt:144`). Manager→worker, distinct from the CI-client→netlab-hosted leg. Same conclusion: the transport death is the trigger, the zero-candidate re-resolution is the defect.

Two caveats passed through rather than buried: the exact deployed resolver version could not be pinned because 0.0.674–676 and 0.0.678–679 are not published (HTTP 404, absent from maven-metadata) — so the running build is an unpublished/purged version or a republished-in-place artifact; and ContainerNursery's `--until` on `container-logs` is silently ignored, always returning the latest 1000 lines, so no startup banner was reachable to date the deploy.

## Relevant PRs / refs

**Open, all `OPEN` at write time:**
- https://github.com/CodexCoder21Organization/UrlResolver/pull/827 — the resolver half. `BLOCKED`. Branch `fix-peer-scoped-address-propagation`, remote head `e0602fe751a7610e564ab032102a6e9f54f8ca7f` (verified identical to the local checkout — nothing unpushed). Rebased post-#835, adapted to `PeerExchangeReadResult`, pinned protocol 0.0.413 (the responder-self feature is inert below 0.0.413: 8/8 stable vs 4/6 failing on 0.0.409). Hot-path defect already fixed (per-peer single-flight + negative backoff on inbound-RPC refresh). **Final supervisory review PASSED — no review debt.** Needs rebase onto #876 once that lands.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/862 — relay nested-object regression test with genuine pre-fix fail evidence. `CLEAN`, merge-queue batch `30295327912` running since 18:47:21Z (58 min at write time; historical build ~58 min). An earlier batch failed 18:46:14Z on the netlab fault above and was re-enqueued once.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/876 — declares `@Timeout(600)` on `tests/testResilienceHarnessFirstLaunchUsesAtomicEphemeralBind.kts`. `BLOCKED`. **Its own fix is PROVEN**: the run reported 1202 passed / 1205 total with the target test NOT among the failures. Failed only on 3 netlab tests.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/877 — aligns the sole outlier `foundation.url:protocol` pin (0.0.407 → 0.0.409) in `tests/stressTestRelayFailoverDebugCapture.kts`; measured distribution was 1190 files on 0.0.409 + `build.kts:190`, exactly 1 outlier. `BLOCKED`, failed on 2 netlab tests.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/878 — **fix for the zero-candidate re-resolution defect described above. Reviewed and signed off; deliberately parked out of the merge queue.** See "The fix for the blocker" below.
- https://github.com/CodexCoder21Organization/W3WalletTests/pull/30 — browser test asserting a server-side capability creation is denied *and visible*. `OPEN`, deliberately NOT merged. **Verification attempted and returned NO VERDICT** — see "Browser assertion: unverified, and its expected text is probably stale" below.

## The fix for the blocker — https://github.com/CodexCoder21Organization/UrlResolver/pull/878

Reviewed, signed off, **parked out of the queue**. Head commit `3b13dcdeb`. Files: `src/foundation/url/resolver/UrlResolver.kt` (+14/-4 on top of the earlier +82/-2), `README.md`, and two new tests.

**The diagnosis in the section above was one step off, and the correction matters.** The bootstrap records are NOT lost on eviction — `configuredBootstrapPeers` (UrlResolver.kt:1806) deliberately retains them, and its own comment says why: *"pruning a temporarily stale registry entry must not erase the only address the confirmed-delivery jobs can retry."* The defect is that **no part of the resolution path ever reads that map**. `openPersistentRpcConnection`'s candidate loop (5758-5845) draws from `peersForServiceFromLocalState` (registry ∪ `discoveredServices`), relay lookup, and `activelyQueryForService` — and the latter targets `peerRegistry.getAllPeers().take(5)`. Every eviction path empties the registry; `pruneStaleDiscoveredServices` (11322) then drops the `discoveredServices` entry because the provider left the registry. The codebase half-knows this: `connectPersistentRpc`'s reconnect factory consults `configuredBootstrapPeerIds` (6134) only to decide whether to keep *waiting*, never to ask them anything.

**Fix:** `activeServiceQueryTargets()` appends operator-configured bootstraps to the active service query **after** the (unchanged) `ACTIVE_SERVICE_QUERY_PEER_LIMIT` fan-out bound, so a registry that is empty *or* full-but-ignorant cannot crowd them out. The gossip circuit breaker no longer vetoes a configured bootstrap **on the registry-derived fallback path only** — a caller supplying explicit `preferredPeers` (the relay-refresh site at 6014) keeps historical behaviour, since a relay independently marked failing by gossip/peer-exchange/dial-admission must still be skipped there. A resolution that attempted nothing now says so, naming the URL and every discovery source with actual counts, instead of reporting it as peers that failed to connect. No timeout, retry, sleep, or fallback was used.

**Evidence:** `testResolutionAfterPeerEvictionQueriesConfiguredBootstrap` 9/9 FAILED before with the verbatim production string, 8/8 passed after; `testUnresolvableServiceReportsWhatWasSearched` 2/2 → 7/7. Final state 15/15 including three relay-path tests that cover the narrowed call site. **All measured on a wiped build cache** (see the stale-artifact trap below).

**Two properties deliberately NOT changed, so do not "fix" them without new evidence:** (1) nothing rate-limits, backs off, or de-duplicates the active-query path — one query per resolution attempt, uncapped. This was already true: `activelyQueryForService` never calls `circuitBreaker.recordFailure` (only `recordSuccess` at 9273), so it could never arm the breaker against a peer it failed to reach. Measured delta is 2 extra dials per attempt (`DEFAULT_BOOTSTRAP_PEERS` = 2 peers, 1 address each), refused in milliseconds, from a set with three writers and no gossip path so it cannot be grown remotely. (2) Both default bootstraps live on the **same host, 198.199.106.165** — the droplet with documented starvation history. Worst case for one failing netlab test is ~616 extra dials there, spread over the uploader's 5-25 s jittered backoff. A resolution that now succeeds ends the retry loop early, so the net is expected to be a reduction — **that is a prediction, not a measurement.**

**Causation to the CI failures is NOT established.** The reproduction is controlled; nobody observed the netlab client's registry state. The fix addresses a real defect that produces the exact symptom, but "this unblocks the gate" remains unproven.

## Browser assertion: unverified, and its expected text is probably stale

Two attempts to run `assertServerSideMintIsDenied` (https://github.com/CodexCoder21Organization/W3WalletTests/pull/30) both died **upstream of the assertion**, so there is no pass/fail to report:
- Local run of `test_browser_nat_jvm_demo` hit the test's own 2,400,000 ms budget while still chunk-uploading ~130 MB (uploads over `url://` from the /code box are far slower than from a CI runner).
- CI dispatch of `netlab-e2e.yml` (https://github.com/CodexCoder21Organization/W3WalletTests/actions/runs/30296458829, daemon cloned from main at 19:02, after the 18:29 merge, so the fix WAS in the build) aborted at the relay-connection precondition added by commit `aaca70c`, which gates the denial assertion: `relay path unreachable (session 1) ... connectionStatusClass="status disconnected"`.

**That CI run was environment-degraded, not code-degraded.** Versus the pre-fix baseline (run 30279653970), three previously-passing tests failed and every runtime roughly doubled; the job hit its 150-minute cap and was cancelled. `url://netlab-hosted/` held **46 live environments**, nearly all `url-*` from concurrent UrlResolver CI.

**Predicted follow-up (from reading code, NOT observed).** `run-jvm-demo.mjs` (~lines 205-232) expects the denial cause to read `RPC request 'create_capability' failed: PERMISSION_DENIED - <denial>`, citing UrlResolver's `response.error` throw site. But #129 established the daemon **never populates** `RpcResponse.error` — refusals ride inside the RPC *result* envelope — so that path is not taken. The merged fix throws from the daemon's SJVM client as `The W3Wallet daemon refused RPC operation 'create_capability' with code PERMISSION_DENIED: <denial>`, which the resolver rewraps in a `SandboxException`. The test's expected string (commit `b364829`, 2026-07-26) predates the fix (2026-07-27 18:20Z), so **the exact-toast comparison may still fail on framing even once the relay works.** The `expectedRemoteCreateDenial` half DOES match `RemoteOriginGuard.kt` verbatim. Reconcile the framing before concluding the fix failed. Do NOT weaken the assertion to make it pass.

**Substitute evidence, labelled as such:** at merge commit `275f7815`, `SjvmTypedProxyErrorPropagationTest` passes 10/10 including the denied-createCapability case. That proves the refusal is no longer swallowed at the daemon/typed-proxy layer — the mechanism the browser test depends on. It does **not** exercise the relay, the demo's mint wrapper, or the toast.

**Branches:** all work is on remote. `pr827-rebase` in the local lane-b checkout has no upstream but its head is byte-identical to #827's remote head; the lane-d3 commit `f8fe4b2de6ce1576e7282872fa093527033f7a69` is present on `refs/heads/lane-d3/relay-nested-object-param-regression`. Nothing exists only locally.

**Challenges filed this session (9):** including the silent-30s-default-timeout class (`challenges/2026-07-27-1818-a-test-that-declares-no-timeout-silently-inherits-a-30s.md`), the browser/E2E local-verification blocker (`2026-07-27-1743-...`), and the webcron persistent-RPC provisioning failure (`2026-07-26-1747-...`).

## Next steps

0. **FIRST: check whether the shared-service fault has cleared before enqueuing anything.** Four PRs are parked out of the merge queue precisely because it was failing unrelated changes; one burned ~3 hours across three attempts. Check `url://netlab-hosted/` environment count (46 live at write time, nearly all `url-*` from concurrent UrlResolver CI) and look for `testNetlab*` failures on recent unrelated PRs. Enqueuing into the fault wastes an hour per PR and teaches nothing.
1. **Land https://github.com/CodexCoder21Organization/UrlResolver/pull/878** — reviewed and signed off, needs only a working gate. It is the fix for the defect that produces the blocking symptom, so land it first.
2. **Land https://github.com/CodexCoder21Organization/UrlResolver/pull/876 and https://github.com/CodexCoder21Organization/UrlResolver/pull/877** — both are one-file changes with no code risk; they need a gate that is not failing on the netlab fault.
3. **Land https://github.com/CodexCoder21Organization/UrlResolver/pull/862** through the queue.
4. **Rebase https://github.com/CodexCoder21Organization/UrlResolver/pull/827 onto main** once #876 is in, so it picks up the timeout declaration, then drive it through the gate.
5. **Verify https://github.com/CodexCoder21Organization/W3WalletTests/pull/30's assertion** against merged #129 and only then decide on merging it. Do NOT weaken the assertion to make it pass.
6. **Decide on the demo's unconditional-refusal UX** (see "Swallowed refusals" above) — currently a raw error string where an explanatory message belongs.

## Reusable / operational knowledge

- **FOUR tools reported success while doing nothing or returning partial data in this one session.** Read back after every mutating operation; it is what caught three of the four. (a) **kompile served a stale `buildMaven()` artifact** — `build.bash` reported SUCCESS in 14.5 s where a cold build is ~120-140 s, and the test ran pre-edit code. It fails in BOTH directions: after `git checkout --` reverting a fix, a genuine 138 s recompile occurred and the test STILL exercised the fixed behaviour, so a control run affirmatively "proved" the defect absent from unmodified main. Wipe `~/.aibuildcaches/<repo-path>-<build.kts-hash>` before every build+test cycle following a source edit. (b) The buildtest **`test-results` API returns a truncated default 100-row page** as if it were the whole run — it reported 0 failures for a run the check-run summary showed had 3, on a 1205-test suite. Use `?limit=500&page=N`, which also returns `totalCount`. (c) **`gh run view --log` silently truncates** and dropped the actual failure lines, nearly inverting a diagnosis — use the raw job-logs API (`gh api repos/OWNER/REPO/actions/jobs/<id>/logs`) piped through `tr -d '\000'`, and always grep, never cat. (d) **`gh pr edit --body-file` exits 0 without applying the edit** (it dies internally on an unrelated Projects-classic GraphQL deprecation) — use `gh api -X PATCH repos/.../pulls/<n>` and re-read the body.
- **`build-watchman --to-merged` auto-enqueues on checks-green, not only on eviction**, and a fresh instance resets the retry budget an operator believes is spent. Two instances armed on parked PRs would have enqueued them into a failing queue within minutes. Its "N/100" progress numbers are page-1 progress on a 1205-test suite (defect (b) above), producing continuous false "frozen at 100/100" alarms.
- **A foreground Bash command hitting its 10-minute timeout kills background jobs via process-group kill.** Launch watchers under `setsid nohup`.
- **The buildtest `test-results` API silently returns a truncated subset.** For run `aab74a46` it returned 100 rows with zero failures while the authoritative check-run summary reported 1205 tests and 3 named failures — an opposite conclusion. Always prefer the check-run `output.summary`.
- **A test that declares no `@Timeout` inherits a hard 30,000 ms default**, which can be provably impossible for its own fixture (here the fixture's internal waits are 90s/90s/60s). Of 198 undeclared-timeout tests with history, zero average over 30s — the population is shaped by the invisible ceiling, not by intent. Profile before touching a budget.
- **Merge-queue re-batching is normal**, not an eviction: the queue rebuilds a batch when a PR ahead is dequeued. Read the removal reason from the timeline before classifying anything as an eviction.
- **`gh` on the /code box is 2.23.0** and lacks `gh run list --status`; use `--json status` plus jq. **Git identity is unset in fresh clones** — set `user.email`/`user.name` locally in the clone, not globally.
- **`/code/workspace` reuse is unsafe** — several lane-* checkouts are mid-rebase from interrupted jobs. Always clone fresh into a new directory.
- **Never bulk-reap netlab environments.** They self-purge, their names carry no age or run attribution, and deleting live ones breaks other agents' in-flight gates.
- **This repo requires the merge queue**; `gh pr merge` is refused with "Auto merge is not allowed for this repository". Enqueue via the GraphQL `enqueuePullRequest` mutation.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: HCF peer-scoped onboarding propagation — protocol half landed, resolver half blocked on a shared-service fault; a fix for the resolver defect it exposed is reviewed and parked

RE-VERIFY BEFORE ACTING — all state below is a write-time snapshot taken 2026-07-27 21:05 UTC

Everything in this document was live-checked at write time and will drift. Authoritative re-checks:

PRs: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
Merge-queue batches: gh api "repos/CodexCoder21Organization/UrlResolver/actions/runs?event=merge_group&per_page=5"
A failing check's real cause: the check-run output summary, NOT the buildtest test-results API (see Reusable knowledge — that endpoint silently truncates).

Mission

What was found and done (2026-07-27 session)

Landed (8 PRs merged, all verified MERGED at write time)

https://github.com/CodexCoder21Organization/UrlProtocol/pull/400 — peer-exchange authenticated responder-self record (the propagation protocol half). Published foundation.url:protocol:0.0.413 manually — nothing auto-publishes on merge in that repo.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/404 — binary-attachment bytecode serving. Published 0.0.414.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/422, https://github.com/CodexCoder21Organization/UrlProtocol/pull/428, https://github.com/CodexCoder21Organization/UrlProtocol/pull/429 — three CI-flake fixes that de-flaked the merge queue.
https://github.com/CodexCoder21Organization/W3WalletDaemon/pull/127 — relay nested-param structural conversion. UrlProtocol issue #399 closed; root cause was two-layer (resolver relay decode ≤0.0.595 + daemon stringification), never in UrlProtocol.
https://github.com/CodexCoder21Organization/W3WalletJvmServerSideCoinCollectorDemo/pull/101 — demo web server emitted absolute redirects built from its own cleartext scheme paired with the TLS terminator's port, so a browser navigation became cleartext aimed at the TLS listener and returned zero bytes. Fixed by configuring the connector for relative redirects. Reproduction flipped 20/20 failing → 0/20.
https://github.com/CodexCoder21Organization/W3WalletDaemon/pull/129 — daemon refusals were swallowed by the typed sandboxed-proxy client. See "Swallowed refusals" below.

Closed as obsolete: https://github.com/CodexCoder21Organization/UrlResolver/pull/828 — superseded by already-merged #835 (code-verified, all 5 tests present on main).

Swallowed refusals (merged) — and a live consequence still outstanding

The daemon reports application-level failures inside the RPC result as {"success": false, "code": ..., "message": ...}, never via the transport error channel — so the transport correctly does not throw. The SJVM client never inspected that envelope: createCapability returned the literal String "null", get* returned null (indistinguishable from genuine absence), list* returned empty, update/delete returned false.

Both resolver layers were verified correct and are NOT at fault: UrlResolver.kt PersistentRpcConnection.doSendRequest does throw on a real error response, and TypeMarshaling.kt fromJvmResult does throw on a sandbox exception.

Two sandbox constraints found empirically and now documented in code + README:

SJVM cannot construct a client-defined Throwable subclass (NoClassDefFoundError in its constructor) → use stdlib exceptions with descriptive messages.
The sandbox stdlib has no StringConcatFactory; Kotlin on JVM 9+ compiles every string template to an invokedynamic against it. This is why the original client could build no messages at all. compileClientKotlin now passes -Xstring-concat=inline.

Outstanding consequence, verified in code, not yet addressed: list_capabilities over url:// is refused unconditionally. WalletRpcHandler.remotePermissionFallbackDenial returns a denial whenever originAuthenticated == false, and UrlProtocolService always passes false — so the hasPermission table check beneath it is unreachable on that transport. The demo has therefore been rendering an empty coin list while reporting "connected". After #129 it surfaces an error instead (correct, and what the browser test asserts), but the demo likely wants an explicit explanatory handler rather than a raw error string. WalletDemoServer.kt call sites: :152, :683, :745, :798, :848.

The current blocker — root-caused this session

Three unrelated UrlResolver PRs failed on network-lab tests within one hour, none of which touch networking. Signature, from the raw job log of merge-queue batch run 30290739996 (job 90059615324, failed 18:46:14Z):

AmbiguousRpcRequestException: RPC request 'uploadFile' to service 'netlab-hosted' has an ambiguous
  outcome because its transport failed after the request write began. ... Original transport failure:
  Persistent RPC connection to service 'netlab-hosted' was closed while requests were still pending.
[test] reconnect for upload retry failed: UrlResolutionException: All 0 peers for service
  'netlab-hosted' failed to connect. Errors: []
TESTS FAILED (1203/1206 tests completed successfully, 3 failed)

RULED OUT — do not re-investigate. This is not the known un-upgraded-consumer instance of the earlier dead-transport defect. Established on both source pins and by fingerprinting the deployed jar (HCF was unusable — ~/.config/hardware-control-fabric/ absent on this box — so the running jar was fingerprinted from live stack traces in ContainerNursery logs against javap -l line tables of published artifacts):

Deployed resolver contains ambiguousWriteFailure/ambiguousTransportFailure, first present in 0.0.671; layout closest to 0.0.677. The running server is newer than what main pins.
Deployed protocol matches the 0.0.402–0.0.409 line set (NettyStreamController$NettyOutputStream.write at Libp2pHostFactory.kt:3230).
NetLabManagerServer/build.kts:54-55 pins resolver 0.0.670 / protocol 0.0.401; threshold crossed in commit 731e76b (PR #106) on 2026-07-19.
Every party is past resolver 0.0.638 / protocol 0.0.374. There is no version-bump fix.

ACTUAL MECHANISM. Transport deaths on these legs are endemic but survivable — the retry loops absorb them and 77 uploads succeeded in the failing run. The fatal step is what follows: after the failed peer is evicted, re-resolution returns zero candidates with an empty error list. Zero peers AND zero errors means nothing was attempted — the resolver believed it had nothing to try. The defect is the missing guard that should keep a bootstrap/last-known peer available so re-resolution can always attempt something after an eviction.

A second, distinct leg fails the same way concurrently: in a 91-second window of live manager logs (19:14:31–19:16:02Z) the deployed manager emitted 12 AmbiguousRpcRequestException traces and 61 Persistent RPC connection to service 'netlab-worker-<uuid>' was closed while requests were still pending lines, all on uploadFile via WorkerClient.forwardRequest (src/netlabmanager/WorkerClient.kt:144). Manager→worker, distinct from the CI-client→netlab-hosted leg. Same conclusion: the transport death is the trigger, the zero-candidate re-resolution is the defect.

Two caveats passed through rather than buried: the exact deployed resolver version could not be pinned because 0.0.674–676 and 0.0.678–679 are not published (HTTP 404, absent from maven-metadata) — so the running build is an unpublished/purged version or a republished-in-place artifact; and ContainerNursery's --until on container-logs is silently ignored, always returning the latest 1000 lines, so no startup banner was reachable to date the deploy.

Relevant PRs / refs

Open, all OPEN at write time:

https://github.com/CodexCoder21Organization/UrlResolver/pull/827 — the resolver half. BLOCKED. Branch fix-peer-scoped-address-propagation, remote head e0602fe751a7610e564ab032102a6e9f54f8ca7f (verified identical to the local checkout — nothing unpushed). Rebased post-#835, adapted to PeerExchangeReadResult, pinned protocol 0.0.413 (the responder-self feature is inert below 0.0.413: 8/8 stable vs 4/6 failing on 0.0.409). Hot-path defect already fixed (per-peer single-flight + negative backoff on inbound-RPC refresh). Final supervisory review PASSED — no review debt. Needs rebase onto #876 once that lands.
https://github.com/CodexCoder21Organization/UrlResolver/pull/862 — relay nested-object regression test with genuine pre-fix fail evidence. CLEAN, merge-queue batch 30295327912 running since 18:47:21Z (58 min at write time; historical build ~58 min). An earlier batch failed 18:46:14Z on the netlab fault above and was re-enqueued once.
https://github.com/CodexCoder21Organization/UrlResolver/pull/876 — declares @Timeout(600) on tests/testResilienceHarnessFirstLaunchUsesAtomicEphemeralBind.kts. BLOCKED. Its own fix is PROVEN: the run reported 1202 passed / 1205 total with the target test NOT among the failures. Failed only on 3 netlab tests.
https://github.com/CodexCoder21Organization/UrlResolver/pull/877 — aligns the sole outlier foundation.url:protocol pin (0.0.407 → 0.0.409) in tests/stressTestRelayFailoverDebugCapture.kts; measured distribution was 1190 files on 0.0.409 + build.kts:190, exactly 1 outlier. BLOCKED, failed on 2 netlab tests.
https://github.com/CodexCoder21Organization/UrlResolver/pull/878 — fix for the zero-candidate re-resolution defect described above. Reviewed and signed off; deliberately parked out of the merge queue. See "The fix for the blocker" below.
https://github.com/CodexCoder21Organization/W3WalletTests/pull/30 — browser test asserting a server-side capability creation is denied and visible. OPEN, deliberately NOT merged. Verification attempted and returned NO VERDICT — see "Browser assertion: unverified, and its expected text is probably stale" below.

The fix for the blocker — https://github.com/CodexCoder21Organization/UrlResolver/pull/878

Reviewed, signed off, parked out of the queue. Head commit 3b13dcdeb. Files: src/foundation/url/resolver/UrlResolver.kt (+14/-4 on top of the earlier +82/-2), README.md, and two new tests.

The diagnosis in the section above was one step off, and the correction matters. The bootstrap records are NOT lost on eviction — configuredBootstrapPeers (UrlResolver.kt:1806) deliberately retains them, and its own comment says why: "pruning a temporarily stale registry entry must not erase the only address the confirmed-delivery jobs can retry." The defect is that no part of the resolution path ever reads that map. openPersistentRpcConnection's candidate loop (5758-5845) draws from peersForServiceFromLocalState (registry ∪ discoveredServices), relay lookup, and activelyQueryForService — and the latter targets peerRegistry.getAllPeers().take(5). Every eviction path empties the registry; pruneStaleDiscoveredServices (11322) then drops the discoveredServices entry because the provider left the registry. The codebase half-knows this: connectPersistentRpc's reconnect factory consults configuredBootstrapPeerIds (6134) only to decide whether to keep waiting, never to ask them anything.

Fix: activeServiceQueryTargets() appends operator-configured bootstraps to the active service query after the (unchanged) ACTIVE_SERVICE_QUERY_PEER_LIMIT fan-out bound, so a registry that is empty or full-but-ignorant cannot crowd them out. The gossip circuit breaker no longer vetoes a configured bootstrap on the registry-derived fallback path only — a caller supplying explicit preferredPeers (the relay-refresh site at 6014) keeps historical behaviour, since a relay independently marked failing by gossip/peer-exchange/dial-admission must still be skipped there. A resolution that attempted nothing now says so, naming the URL and every discovery source with actual counts, instead of reporting it as peers that failed to connect. No timeout, retry, sleep, or fallback was used.

Evidence: testResolutionAfterPeerEvictionQueriesConfiguredBootstrap 9/9 FAILED before with the verbatim production string, 8/8 passed after; testUnresolvableServiceReportsWhatWasSearched 2/2 → 7/7. Final state 15/15 including three relay-path tests that cover the narrowed call site. All measured on a wiped build cache (see the stale-artifact trap below).

Two properties deliberately NOT changed, so do not "fix" them without new evidence: (1) nothing rate-limits, backs off, or de-duplicates the active-query path — one query per resolution attempt, uncapped. This was already true: activelyQueryForService never calls circuitBreaker.recordFailure (only recordSuccess at 9273), so it could never arm the breaker against a peer it failed to reach. Measured delta is 2 extra dials per attempt (DEFAULT_BOOTSTRAP_PEERS = 2 peers, 1 address each), refused in milliseconds, from a set with three writers and no gossip path so it cannot be grown remotely. (2) Both default bootstraps live on the same host, 198.199.106.165 — the droplet with documented starvation history. Worst case for one failing netlab test is ~616 extra dials there, spread over the uploader's 5-25 s jittered backoff. A resolution that now succeeds ends the retry loop early, so the net is expected to be a reduction — that is a prediction, not a measurement.

Causation to the CI failures is NOT established. The reproduction is controlled; nobody observed the netlab client's registry state. The fix addresses a real defect that produces the exact symptom, but "this unblocks the gate" remains unproven.

Browser assertion: unverified, and its expected text is probably stale

Two attempts to run assertServerSideMintIsDenied (https://github.com/CodexCoder21Organization/W3WalletTests/pull/30) both died upstream of the assertion, so there is no pass/fail to report:

Local run of test_browser_nat_jvm_demo hit the test's own 2,400,000 ms budget while still chunk-uploading ~130 MB (uploads over url:// from the /code box are far slower than from a CI runner).
CI dispatch of netlab-e2e.yml (https://github.com/CodexCoder21Organization/W3WalletTests/actions/runs/30296458829, daemon cloned from main at 19:02, after the 18:29 merge, so the fix WAS in the build) aborted at the relay-connection precondition added by commit aaca70c, which gates the denial assertion: relay path unreachable (session 1) ... connectionStatusClass="status disconnected".

That CI run was environment-degraded, not code-degraded. Versus the pre-fix baseline (run 30279653970), three previously-passing tests failed and every runtime roughly doubled; the job hit its 150-minute cap and was cancelled. url://netlab-hosted/ held 46 live environments, nearly all url-* from concurrent UrlResolver CI.

Predicted follow-up (from reading code, NOT observed). run-jvm-demo.mjs (~lines 205-232) expects the denial cause to read RPC request 'create_capability' failed: PERMISSION_DENIED - <denial>, citing UrlResolver's response.error throw site. But #129 established the daemon never populates RpcResponse.error — refusals ride inside the RPC result envelope — so that path is not taken. The merged fix throws from the daemon's SJVM client as The W3Wallet daemon refused RPC operation 'create_capability' with code PERMISSION_DENIED: <denial>, which the resolver rewraps in a SandboxException. The test's expected string (commit b364829, 2026-07-26) predates the fix (2026-07-27 18:20Z), so the exact-toast comparison may still fail on framing even once the relay works. The expectedRemoteCreateDenial half DOES match RemoteOriginGuard.kt verbatim. Reconcile the framing before concluding the fix failed. Do NOT weaken the assertion to make it pass.

Substitute evidence, labelled as such: at merge commit 275f7815, SjvmTypedProxyErrorPropagationTest passes 10/10 including the denied-createCapability case. That proves the refusal is no longer swallowed at the daemon/typed-proxy layer — the mechanism the browser test depends on. It does not exercise the relay, the demo's mint wrapper, or the toast.

Branches: all work is on remote. pr827-rebase in the local lane-b checkout has no upstream but its head is byte-identical to #827's remote head; the lane-d3 commit f8fe4b2de6ce1576e7282872fa093527033f7a69 is present on refs/heads/lane-d3/relay-nested-object-param-regression. Nothing exists only locally.

Challenges filed this session (9): including the silent-30s-default-timeout class (challenges/2026-07-27-1818-a-test-that-declares-no-timeout-silently-inherits-a-30s.md), the browser/E2E local-verification blocker (2026-07-27-1743-...), and the webcron persistent-RPC provisioning failure (2026-07-26-1747-...).

Next steps

FIRST: check whether the shared-service fault has cleared before enqueuing anything. Four PRs are parked out of the merge queue precisely because it was failing unrelated changes; one burned ~3 hours across three attempts. Check url://netlab-hosted/ environment count (46 live at write time, nearly all url-* from concurrent UrlResolver CI) and look for testNetlab* failures on recent unrelated PRs. Enqueuing into the fault wastes an hour per PR and teaches nothing.
Land https://github.com/CodexCoder21Organization/UrlResolver/pull/878 — reviewed and signed off, needs only a working gate. It is the fix for the defect that produces the blocking symptom, so land it first.
Land https://github.com/CodexCoder21Organization/UrlResolver/pull/876 and https://github.com/CodexCoder21Organization/UrlResolver/pull/877 — both are one-file changes with no code risk; they need a gate that is not failing on the netlab fault.
Land https://github.com/CodexCoder21Organization/UrlResolver/pull/862 through the queue.
Rebase https://github.com/CodexCoder21Organization/UrlResolver/pull/827 onto main once #876 is in, so it picks up the timeout declaration, then drive it through the gate.
Verify https://github.com/CodexCoder21Organization/W3WalletTests/pull/30's assertion against merged #129 and only then decide on merging it. Do NOT weaken the assertion to make it pass.
Decide on the demo's unconditional-refusal UX (see "Swallowed refusals" above) — currently a raw error string where an explanatory message belongs.

Reusable / operational knowledge

FOUR tools reported success while doing nothing or returning partial data in this one session. Read back after every mutating operation; it is what caught three of the four. (a) kompile served a stale buildMaven() artifact — build.bash reported SUCCESS in 14.5 s where a cold build is ~120-140 s, and the test ran pre-edit code. It fails in BOTH directions: after git checkout -- reverting a fix, a genuine 138 s recompile occurred and the test STILL exercised the fixed behaviour, so a control run affirmatively "proved" the defect absent from unmodified main. Wipe ~/.aibuildcaches/<repo-path>-<build.kts-hash> before every build+test cycle following a source edit. (b) The buildtest test-results API returns a truncated default 100-row page as if it were the whole run — it reported 0 failures for a run the check-run summary showed had 3, on a 1205-test suite. Use ?limit=500&page=N, which also returns totalCount. (c) gh run view --log silently truncates and dropped the actual failure lines, nearly inverting a diagnosis — use the raw job-logs API (gh api repos/OWNER/REPO/actions/jobs/<id>/logs) piped through tr -d '\000', and always grep, never cat. (d) gh pr edit --body-file exits 0 without applying the edit (it dies internally on an unrelated Projects-classic GraphQL deprecation) — use gh api -X PATCH repos/.../pulls/<n> and re-read the body.
build-watchman --to-merged auto-enqueues on checks-green, not only on eviction, and a fresh instance resets the retry budget an operator believes is spent. Two instances armed on parked PRs would have enqueued them into a failing queue within minutes. Its "N/100" progress numbers are page-1 progress on a 1205-test suite (defect (b) above), producing continuous false "frozen at 100/100" alarms.
A foreground Bash command hitting its 10-minute timeout kills background jobs via process-group kill. Launch watchers under setsid nohup.
The buildtest test-results API silently returns a truncated subset. For run aab74a46 it returned 100 rows with zero failures while the authoritative check-run summary reported 1205 tests and 3 named failures — an opposite conclusion. Always prefer the check-run output.summary.
A test that declares no @Timeout inherits a hard 30,000 ms default, which can be provably impossible for its own fixture (here the fixture's internal waits are 90s/90s/60s). Of 198 undeclared-timeout tests with history, zero average over 30s — the population is shaped by the invisible ceiling, not by intent. Profile before touching a budget.
Merge-queue re-batching is normal, not an eviction: the queue rebuilds a batch when a PR ahead is dequeued. Read the removal reason from the timeline before classifying anything as an eviction.
gh on the /code box is 2.23.0 and lacks gh run list --status; use --json status plus jq. Git identity is unset in fresh clones — set user.email/user.name locally in the clone, not globally.
/code/workspace reuse is unsafe — several lane-* checkouts are mid-rebase from interrupted jobs. Always clone fresh into a new directory.
Never bulk-reap netlab environments. They self-purge, their names carry no age or run attribution, and deleting live ones breaks other agents' in-flight gates.
This repo requires the merge queue; gh pr merge is refused with "Auto merge is not allowed for this repository". Enqueue via the GraphQL enqueuePullRequest mutation.