UrlResolver PR 835 emergency union — MERGED; follow-up relay-identity hardening green and awaiting merge

RE-VERIFY BANNER. Everything below is a write-time snapshot taken 2026-07-26 03:00 UTC. PR states, CI results, and host metrics drift. Re-check before acting: gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,mergedAt for each pull request, and probe https://buildtest.kotlin.build/ directly for site health.

Mission summary

Drive the emergency-lane workstream to completion: land the UrlResolver "emergency stabilization union" change, and keep https://buildtest.kotlin.build/ reliably healthy (page loads fast, queued builds start within a minute, builds keep making forward progress).

The core mission is complete. https://github.com/CodexCoder21Organization/UrlResolver/pull/835 MERGED 2026-07-25T23:42:51Z as commit 927f34a534b4b0fa47f2eb0254b4d70546beb371, on a genuinely green gate — its merge-queue run passed 1205/1205 with zero failures. No CI bypass, no admin merge.

This handoff previously said "awaiting final CI + merge" and credited protocol 0.0.406; both are now stale. The blocker that actually held it was two intermittent test failures, root-caused and fixed below.

What was found and done

The two intermittent failures that blocked the merge

Both had the same shape: a safety mechanism silently suppressing the recovery it existed to protect.

1. testWithdrawalRetriedAfterPendingDialStall. A failed connection attempt arms a per-address cooldown (1s growing toward 5min) so a node stops hammering an unreachable peer. That cooldown also silenced addBootstrapPeers(), the explicit, caller-initiated reconnect path — ensureExchangeConnection filtered out every candidate address and returned without opening a socket, so the call returned in ~1ms having done no network work and scheduled no retry. A node that lost its mesh could be unable to re-attach for minutes while appearing to retry. Fixed by clearing the cooldown only for peers a caller explicitly named; background dials keep the cooldown, so dial volume and peer-pruning accounting are unchanged.

2. stressTestRelayFailoverToAlternateRelayWhenRegisteredRelayDies. Only reproducible under 2-CPU pinning (33%; 0/13 on 16 cores). Root cause: after failing over to a live relay, the caller recorded connectedRelayPeerId from its own stale local variable rather than the relay it actually registered with, so the node reported the dead relay forever. Fixed upstream in UrlProtocol 0.0.409.

Measurement table (2-CPU pinned):

no fix                       4 reproductions / 12   (33%)
no fix, second measurement   3 reproductions / 8    (37.5%)
protocol 0.0.408 (wrong fix) 4 reproductions / 12   (33%, no effect)
protocol 0.0.409             0 reproductions / 12

Follow-up: the relay-identity fix was incomplete

An adversarial review found 0.0.409 patched only one of three sites that record which relay a node is connected to. Verified against source directly:

• RelayService.kt:2845 (registerWithRelay, the primary entry point) recorded the relay it intended to reach. A node failing over here reports the dead relay permanently, and because relayConnection is then non-null the reconnect tick never repairs it.
• RelayService.kt:2749-2761 (sendGossipToRelay inline reconnect) never restored the field at all.
• RelayService.kt:6145 (reconnect tick) — the partial existing fix.

Fixed in 0.0.410: all three now read relayConnection?.relayPeerId (the live connection object knows exactly which relay it is attached to). The three doRegisterWithRelay failure paths (~3126/3131/3136) now clear connectedRelayPeerId instead of leaving a relay recorded with no connection. relayConnection, registeredRelayPeerId, connectedRelayPeerId are now @Volatile. Commit 8d62eb15 (a reconnect-backoff gate) was reverted: its measured benefit was zero, and its mechanism is unreachable — both the failure-counter increment and the escalation it feeds live inside the relayConnection == null branch — while its harm is real (reconnect poll can stay pinned at the 600000 ms ceiling, and the failure counter can stay at its escalation threshold, causing premature failover away from a healthy relay).

Local evidence, four-way toggle:

cache purged, no fix       3 failed / 3
cache purged, fix applied  3 passed / 3
fix reverted, cache kept   3 failed / 3
fix re-applied, cache kept 3 passed / 3

Plus 168 relay-related tests passing.

Relevant PRs and refs

• https://github.com/CodexCoder21Organization/UrlResolver/pull/835 — MERGED 2026-07-25T23:42:51Z, commit 927f34a534b4b0fa47f2eb0254b4d70546beb371
• https://github.com/CodexCoder21Organization/UrlProtocol/pull/421 — OPEN, all three checks green, mergeStateStatus CLEAN, head d224cdbf27e22b28fda16efc1252f8c95e4f1f7e, branch fix/abandon-unacknowledged-relay-registration. Carries 0.0.410. Awaiting merge approval. 0.0.410 is NOT yet published — publish only after merge (this repository's changelog records a prior version published from an unmerged branch and silently lost).
• https://github.com/CodexCoder21Organization/UrlResolver/pull/849 — OPEN, both checks failed, branch revert/bld-build-timeout-90, head 1cdb518. One-line change reverting a CI job timeout from 150 back to 90 minutes.
• Duplicate record covering the same original mission: hf-2026-07-22-urlresolver-835-emergency-union-merge-local-gate. Its mission is likewise complete; close it out alongside this one.

Next steps

1. Merge https://github.com/CodexCoder21Organization/UrlProtocol/pull/421, then publish foundation.url:protocol:0.0.410 (publish-maven-artifact skill, foundation.url.protocol.buildMaven). Do not publish before the merge.
2. Diagnose the failures on https://github.com/CodexCoder21Organization/UrlResolver/pull/849 — do not blind re-run. That change is a one-line CI-timeout edit and cannot affect runtime, yet both checks failed with three symptoms. The load carries a serialVersionUID mismatch on foundation.url.resolver.UrlResolutionException (stream classdesc -2677286634181115085 vs local -6415732162446864309), meaning two different builds of that class communicated over TCP. The failing test pins foundation.url.resolver.buildMaven() and current protocol, so a stale pin is ruled out; the live hypothesis is a stale build cache feeding a pre-merge artifact into a forked child JVM. To test: purge the relevant build cache, re-run the single test, see whether the mismatch survives. If it does not, the other two symptoms (testPersistentRpcTimeoutInvalidatesSilentTransportAndRedials failing with "All 1 peers failed to connect", and "Healthy in-process peers did not all receive gossip within 5s under stale-peer saturation") are likely consequences rather than independent flakes.
3. Genuinely stale pins found while investigating (real, but NOT the cause above): tests/perfServiceConnectAnnotated.kts, tests/perfGossipPropagationAnnotated.kts, tests/perfClusterJoinAnnotated.kts pin foundation.url:resolver:0.0.601 while main builds 0.0.661; tests/stressTestRelayFailoverDebugCapture.kts pins foundation.url:protocol:0.0.407 while 1190 other tests pin 0.0.409.
4. Known-unreproduced: testPeerAnnouncementForwardingUsesWithdrawalFilteredServices failed once, never reproduced across 13 runs, passed in the green merge-queue run. Not root-caused. If it recurs, amplify with real concurrency rather than re-running the single test.
5. Two defects deliberately left unfixed (flagged, out of scope for a narrow verified change): in doRegisterWithRelay, if connectToRelay throws, the previous RelayConnection — still referenced by relayConnection — is dropped without being closed (attemptedConnection is still null, so attemptedConnection?.close() closes nothing). And between relayConnection = connection (~3044) and the acknowledgement, the connection is new while the recorded identity is old; a single assignment at the ACK-success point inside doRegisterWithRelay would close that window and make all three call sites redundant.

Reusable / operational knowledge

• scripts/test.bash serves STALE cached bytecode across source edits. This invalidated two separate conclusions in one session. Purge ~/.aibuildcaches/_code_workspace_<repo>-* before every configuration you measure, and run twice.
• The usual stale-cache canary does not work here. Appending invalid Kotlin does not make the build fail — it silently respawns the build worker every ~5s indefinitely with no diagnostic. Use a red↔green↔red↔green toggle instead: it proves the test consumed new bytecode, not merely that a compiler ran.
• Check for orphaned scripts before trusting a working tree. A shell script from an abandoned agent run, re-parented to init, was still copying files over RelayService.kt and deleting the build cache — corrupting measurements and reverting edits. Detect with ps -eo pid,ppid,etime,args | grep -E '<scratchpad script names>'.
• Judge whether delegated work is alive by running processes, not file mtimes. A compiling agent changes no source files for long stretches; conversely an idle one leaves recent mtimes.
• A frontend 503 is not proof a build died. During a ContainerNursery deploy the site returned three consecutive 503s at exactly 30.1s (a proxy upstream timeout) while builds kept executing and passed. Check for an active cn-deploy-*.log being written before restarting anything — restarting mid-deploy is destructive.
• Disk on the host recovers on its own after concurrent-build spikes. It fell 3.5GB→2.4GB then recovered to 8.8GB as runs completed and released build caches. The growth driver is per-run shared-build-cache.tar.gz plus the persistent module-build-cache, not a runaway log. Safe manual reclaim when genuinely tight: rotated *.log.1 files, fuser-checked first. Do not delete rollback jars.
• buildtest run failures with buildSucceeded: true and N-1/N tests passing are ordinary test failures, not infrastructure faults. Infrastructure faults show 0 tests passed.