Handoff: UrlResolver #608 relay-recovery regression — ROOT-CAUSED + FIXED in PR #752 (relay reservation not woken on transport recovery); awaiting CI-green + user merge

Written 2026-07-06; rewritten 2026-07-11 ~02:30 UTC; RESOLVED 2026-07-11 ~18:00 UTC (effort began 2026-06-11)

⚠️ RE-VERIFY BEFORE ACTING. State below is a snapshot from 2026-07-11 ~18:00 UTC and will be stale. Re-verify with:

• The fix PR: gh pr view 752 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt — if MERGED, this whole effort is DONE (delete this handoff).
• #608: gh pr view 608 ... — its netlab test now lives inside #752, so #608 is likely redundant once #752 merges (close it).
• The bug: #742 (fixed by #752), #743, #609.

✅ RESOLUTION (2026-07-11): the regression is ROOT-CAUSED and FIXED in UrlResolver #752.

Root cause: after the circuit relay restarts (same identity+address), the NAT'd provider re-establishes a healthy transport connection to it, but RelayService's reservation/reconnect task is parked in exponential backoff and is never woken by the independently-recovered PeerExchange transport — so connectedRelay stays none forever and the provider is unreachable. Proven with instrumented hosted-netlab captures ([PEX-CONNECTION] source=reuse closeFutureDone=false to the relay while connectedRelay=none persists 300s; stale-connection-reuse hypothesis REFUTED). Fix (2 layers, upstream-first): (1) upstream jvm-libp2p transport fix published as 1.3.0-codexcoder21-snapshot-10; (2) UrlProtocol2.wakeRelayReservationAfterTransportRecovery — a healthy PeerExchange connection to a configured bootstrap relay now invalidates the reconnect backoff and immediately runs the (idempotent, synchronized) relay-registration path, off the caller thread, coalesced via an AtomicBoolean. Verified red→green: testNetlabNatToNatRelayOutageRecovery failed before, passes twice cleanly after; full CI suite 1071/1072 (the 1 = unrelated testGossipBroadcastStormWithStalledPeersBoundsQueuedMemory timing flake, re-run). Adversarial review (driver): clean. Remaining: #752 CI finalizing → user merge decision (it is a resolver fix + jvm-libp2p dep bump, so explicit approval required). Done by codex gpt-5.6-sol over 4 rounds; the reproducer/instrumentation history is on branch fix-relay-restart-recovery-742.

Everything below this banner is the PRE-FIX investigation record, retained for context. The "merge BLOCKED on the product fix" framing is now superseded by the fix above.

Mission

The user's original request (verbatim):

"We want to have a bunch of comprehensive tests for UrlResolver repository which utilize the netlab to simulate more realistic network scenarios. Especially situations which might arise around two peers, both behind a NAT, wanting to talk to each other, and their relay intermittently going offline."

Later explicit instruction: "ensure https://github.com/CodexCoder21Organization/UrlResolver/pull/608 gets merged" (merge pre-authorized once green). On 2026-07-10 the user also directed an immediate retry ("try again now") during the infra-storm phase.

What was found and done on 2026-07-10/11 (the chain — all verified)

1. Rebased #608 onto latest main (was DIRTY): kept both sides of the NetlabTestEnv.kt additions conflict, realigned all test artifact pins to main's set (foundation.url:protocol:0.0.337, sjvm 0.0.40, jvm-libp2p snapshot-9, + netlab.api:netlab-api:0.0.6). Module compiles clean.

2. The June premise no longer holds — evidence-driven test evolution (3 reframes, each forced by a CI observation):

   • Run 7d418b30: with the provider bootstrapped to the root, removing the relay did NOT break reachability — fresh resolves OK'd "via relay probe" and the session logged OK seq=2..13 straight through the outage. The resolver now forwards service RPCs across any live connection to a mutually-known public peer. → outage unobservable in the original topology.
   • SPOF rewire (provider bootstraps via the relay itself): outage gate then PASSED, but recovery failed — the islanded provider dialed the restored relay (same seed+IP) for 300+s, every attempt still-handshaking>1s/dial-failed, while the root reconnected to that same relay instantly. → filed #742; red repro preserved on branch netlab-islanded-provider-reconnect-redtest (@ 82442788).
   • bld-build 2026-07-10T23:51Z: the outage window is nondeterministic — the same continuity that held in 7d418b30 failed there (fresh resolves could NOT reach the provider during the outage). → neither "must break" nor "must survive" is assertable for the outage window.
   • bld-build 2026-07-11T00:46Z (env url-iro-2128615700439): recovery failed in the root-bootstrap topology because the provider's relay failover adopted the NAT'd session client as its relay (connectedRelay=12D3KooWSRtn… — clients default to relay-capable) and never re-registered with the restored real relay. → filed #743; harness idle/resolve/session roles now honor ENABLE_RELAY, and the test makes every non-relay peer relay-incapable.
   • bld-build 2026-07-11T02:18Z (env url-iro-2357611348077): with that trapdoor closed, recovery STILL failed — see the generalization comment on #742.

3. Conclusion (the current blocker): three recovery failures across three distinct topologies in 24h prove current main cannot restore a NAT'd provider's reachability after its circuit relay restarts with the same identity+address — except when the RPC-forwarding path nondeterministically bridges it. June-era code passed this exact contract repeatedly (#608 green+CLEAN 2026-06-16, plus repeated local/bld-build greens). This is a real regression; the test is its deterministic reproducer.

4. The test as it now stands (head ef72b478, file tests/testNetlabNatToNatRelayOutageRecovery.kts): baseline gates; two relay down/up cycles; outage window drives real traffic and logs the observed outcome WITHOUT asserting direction (dwell gated on observed activity, not sleeps); each restore asserts fresh-client resolve + in-process session recovery via monotonic marker counts; session-restart guard. All evolution documented in PR comments (premise fix, pivot, final semantics, #743 + trapdoor, terminal status).

5. Side item from the old handoff COMPLETED: githubwatchmanserviceserver:github-watchman-service-server:0.0.25 published to kotlin.directory (pom/jar verified 200) and deploy verified — the prod jar (2026-07-07) bundles lambda-api 0.0.14 (fingerprint: lambdaapi/LambdaClientKt.class = 8197 bytes). Nothing left there.

6. Infra forensics (2026-07-10): six consecutive kotlin.build runs died in provisioning with zero tests executed, on three distinct mechanisms — netlab-manager starvation under CN load storms; DigitalOcean account rate-limit (5000/hr) exhausted by the CI's own churn (error names its own reset time — schedule the rerequest just after it); and a 30s-fatal synchronous SJVM webcronserver/WebCronServiceClientImpl.createTimer call inside createBuildDroplet. All documented in challenges/2026-07-10-1807.

Relevant PRs / refs (write-time state — RE-VERIFY)

| Ref | State @ 2026-07-11 02:30 UTC | Notes | |-----|------------------------------|-------| | UrlResolver #608 | OPEN, BLOCKED on product regression | branch netlab-nat-intermittent-relay-tests, head ef72b478. Recovery gate fails deterministically on current main. Merge pre-authorized once green. Do NOT weaken the test; kotlin.build must never land red. | | Issue #742 | OPEN — the blocker | Provider cannot re-attach to a restarted relay (same identity+address). Islanded case dial-loops forever; generalization comment proves non-islanded fails too. | | Issue #743 | OPEN | Relay failover adopts an unreachable NAT'd relay candidate and never re-evaluates. | | Issue #609 | OPEN | Provider never registers with a relay learned AFTER losing its own (new identity). Red repro: branch netlab-relay-identity-replacement-redtest @ 30340e35. | | Branch netlab-islanded-provider-reconnect-redtest | @ 82442788 | SPOF-topology red repro for #742's islanded case. | | Challenge record | merged | The 2026-07-10 provisioning-failure taxonomy + remediation recipes. | | kotlin.build run b5c57812 | left wedged | build.log + test-events frozen since 01:43Z with check in_progress — moot for #608, but a buildtest-server resume-bug data point. |

Next steps (ordered)

1. USER DECISION (blocking): authorize fixing the resolver regression (#742 general mechanism — re-dial/re-attach to a restarted same-identity peer) as a new effort — or park #608 until someone else fixes it. #608's head is the ready-made regression test either way.
2. If fixing: the reproducer is deterministic (#608's recovery gate fails every run that reaches it). Suspect layers: jvm-libp2p connection/dial dedup holding stale state for a dead peer id (still-handshaking>1s loop = a pending-dial entry that never resolves and is returned to every retry), and/or UrlResolver's relay re-registration path (reannounceWithCircuitRelay, relay-candidate selection). June-era main recovered reliably — bisect window is roughly the merges after 2026-06-16 (candidates seen in this effort's history: #679/#687/#690/#698/#731 and the #617 follow-ups). Fix upstream at the owning layer; #608 flips green as the verification.
3. CI mechanics when re-driving #608 (once the product fix exists): rerequest via gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-suites/<id>/rerequest using the current head's suite id (commits/<head>/check-suites, app kotlin-build-ci-test) and verify the check's details_url run id CHANGED; diagnose runs on the host at /root/buildtest-data/runs/<id>/{test-results.json,build.log} — counts {PENDING: 1066} means provisioning never ran the code (rerequest, don't debug the PR); build-watchman 0.0.11 --to-merged (coursier, kotlin.directory) is the right watcher — its lost-webhook-dispatch detection fired correctly and its remediation (warm https://github-webhooks.wasmserver.com/ ×3 + rerequest) worked.
4. Do NOT weaken the test further. "Recovers after restore" is the minimum meaningful assertion; the outage window is already observed-not-asserted for a documented product reason. Anything less asserts nothing about the mission scenario.

Reusable / operational knowledge

• The resolver forwards service RPCs across any established connection to a mutually-known public peer. Any netlab scenario that assumes "removing host X severs reachability" must ensure no alternative forwarding path exists — and even then, verify empirically; this behavior is nondeterministic under churn.
• Client harness roles default to relay-capable. NetlabHarness idle/resolve/session roles honor ENABLE_RELAY (added at ef72b478). Set it false for any client that must never be adoptable as a relay — otherwise relay failover may adopt a NAT'd client (#743) and silently poison the scenario.
• DO rate-limit failures name their own reset time in the error text (resets at <ISO>; ratelimit-remaining=0) — schedule the retry just after it rather than polling blind. The org's CI can consume the entire 5000/hr account budget by itself (10-droplet sharded buildtest runs + netlab envs + cleanup churn).
• buildtest wedge rule works as documented in the watch-build skill: WEDGE = test-events.jsonl AND build.log both frozen >20 min while the check is in_progress (confirmed once on run b5c57812).
• Everything else from the 2026-07-06 version of this handoff (netlab test gotchas, kompile .kts constraints, merge-queue GraphQL enqueue, prod-host diagnostics) remains valid — see git history of this file for the full prior text.