Finish full-suite validation for UrlResolver relay recovery

Written 2026-10-18 10:34 UTC. Source PR state was read at10:31 UTC; no CI status was queried.

RE-VERIFY: This is a write-time snapshot. Verify the source branch SHA with git ls-remote, and the PR with gh pr view https://github.com/CodexCoder21Organization/UrlResolver/pull/1314 --json state,isDraft,headRefOid,mergedAt. The delegated brief says not to watch CI after opening the PR; no CI or production state was queried for this handoff. Do not treat this document as permission to merge, enqueue, publish or deploy.

Design decisions in force

The owner stated on 2026-10-18:

• “Merging and deploying are mine, not yours; your part ends at an open, pushed PR meeting the bar below.”
• “The connection returns to the direct route without a process restart, through one owned mechanism (not ad-hoc per call), with no request lost or duplicated during the switch.”
• Transport relay failures must be “Classified as a connection/transport failure eligible for the existing reconnect path, NOT as an application/server error. Genuine application errors from the provider stay non-recoverable exactly as today.”
• “Error text” is “Unchanged for genuine application errors; transport failures keep their full detail.”
• “Write the classification rule in one place.”
• “done = failing tests red on main then green first attempt, revert control red, full suite green from pushed head, PR open; then stop, do not watch checks.”
• “May not merge, enqueue, deploy, publish, touch production, raise timeouts, reduce iterations.”

These requirements remain in force. A separate job owns relayed-stream lifetime work in r7; this task must be correct independently and must not change that checkout. Any successor uses a fresh own checkout. The owner's full-green bar was NOT achieved; the PR remains a draft rather than being reported complete.

Mission

The TinySite storage incident left Filedrop on a relay path after its direct reconnect failed. The narrow job was to reproduce and fix persistent url:// direct/relay recovery and transport-error classification in UrlResolver main, with tests before implementation and an open pushed PR. The separate stream-lifetime defect, consuming-service pins, publishing, merging and deployment belong to the owner/other jobs.

What was found and done

1. Proven direct-to-relay defect: connectPersistentRpc captured original-provider relay permission from the INITIAL controller type. An initially-direct connection could not later reconnect to its same provider through an available relay. The regression testPersistentRpcRestoresSameDirectRouteAfterRelay failed on unchanged main at fallback. The one-line permission now follows the existing generic-versus-peer-scoped policy (!peerScoped); concrete peer restrictions stay intact.
2. Proven adapter-delivery classification defect: an admitted RelayProtocol.RelayResponse.error throws through the local relay adapter, which produces RELAY_FORWARD_FAILED. Previously the persistent connection classified that as a provider error and retained the failed adapter. testPersistentRelayTransportFailureRetiresConnection failed on unchanged main at isServerError. Now the adapter marks its own failures, one classifier interprets the origin and code, and the existing generation-retirement mechanism permits the next independent call to reconnect. An uncertain dispatched operation is returned once, never replayed automatically. Existing explicit pre-dispatch RELAY_FORWARD_RETRYABLE semantics remain replayable.
3. Refuted direct-restoration premise on current main: main already observes recovered direct routes and promotes through the existing reconnect owner before writing the next request, preserving written relay siblings. An initial TCP-accept-only test barrier raced libp2p negotiation; after waiting for the public hasActiveConnectionToPeer observation, unchanged main passed the same-address restoration case. No periodic probe/new worker remains in this patch. testPersistentRelayRecoversUnchangedDirectAddress guards existing behavior.
4. Refuted code-only RELAY_ERROR classification: legacy {error: ...} and typed RPC errors can both come from the provider. Treating their code alone as transport would break the required application-error behavior. Independent review caught an early wrongly interpreted typed fixture; the corrected test is testPersistentTypedRelayProviderErrorKeepsConnection. The adapter strips its reserved marker from every provider response and supplies it only on locally generated delivery failures; the persistent classifier trusts it only on RelayBackedRpcController. Separate direct/typed-relay/legacy-provider exception tests verify exact text, server-error meaning, no replay/reconnect and later healthy use. Provider text containing RELAY_FORWARD_RETRYABLE or “is closed” does not change that.
5. TDD evidence: main baseline was 046396a677e14cf950dc7c3d102cbc2c187b75a1. Failing tests were pushed before source edits at https://github.com/CodexCoder21Organization/UrlResolver/commit/9ba35aafd9086592489cba5f43515d4df4f81da9 and https://github.com/CodexCoder21Organization/UrlResolver/commit/1cf4a080d80ea1fd2a22b5d27ba14f843a1d090f . The first runnable actual transport/fallback cases failed for their stated bugs; an earlier test-only compile typo was corrected before execution. Final unchanged-main revert control has 2 FAIL / 4 PASS, precisely the two real bugs plus four existing-behavior guards. Fixed first completed focused run passed 5/5; final pushed head and neighbors passed 16/16 first completed run. Two independent review passes plus a final exact-head test audit have no unresolved findings.
6. Full remote gate not green: https://buildtest.kotlin.build/run?id=bb547490 from exact pushed source head ended with Provisioning deadline exceeded, elapsed615331ms/limit600000ms, currentAttempt10. Independently of that terminal infrastructure error, it recorded 269 PASS / 6 FAIL of2078 expected cases. The six failures are listed below; they must not all be called provisioning failures.
7. Main comparison: one fresh unchanged-main run of exactly those six cases built successfully and ran all cases: 5 PASS / 1 FAIL. testActiveQueryRetainsConfiguredAddressAfterProviderRestart:126 failed on main with the SAME assertion: The separate readiness probe must not leave a reusable connection on the query client. This establishes a baseline failure for ONE case. The other five passing once does not explain their remote failures. No retry or timeout/iteration change was made.
8. Local full gate not complete: a same-head local full attempt reported foundation.url.resolver.buildMaven() (failed in 141ms) then no such file: .../UrlResolver/.git/index.lock during preparation. Earlier owned jstacks show prepareTests workers hashing manifests through captureInputManifest/InputValidationPass.matches/getValidatedRuleClasspath; a disappearing workspace-file handling issue is a lead, not a proven exact ordering or attribution to a Git command. After the reported failure, remaining owned preparation workers were explicitly stopped. CLI result is INTERRUPTED / exit143 / 0 cases run. There is no local full green or completed-case claim. All owned run/watch processes and agents stopped.

Remote case failures: stressTestActiveQueryReportsResponseReadFailure (RPC negotiation readiness), stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation (2/52 announcements missing), stressTestSynchronousPeerExchangeServiceDiscovery (657ms average versus500ms guard), testActiveQueryPrefersRegistryAddressWhenBothAddressesAreUsable (public worker IP advertised instead of expected loopback, SAME port), testActiveQueryRetainsConfiguredAddressAfterProviderMoves (public address connection refusal), and the restart readiness case above. Their test files do not call the persistent-RPC APIs changed here; that argues against direct attribution but is not their root-cause diagnosis. Complete reported messages/traces are in the evidence archive (one remote server record itself truncated its giant thread dump).

Relevant PRs / refs

The archive index explains original-byte SHA-256 hashes and gzip extraction. The chronological findings explicitly retain corrected hypotheses and failed experimental assumptions. Final source/README tests are entirely on the source branch; no stashes or uncommitted source work remain. A clean fresh baseline clone has no changes to save. No application build JAR or dependency cache is archived. Empty API-probe outputs and the discarded experimental source copy contain no additional work and were deliberately omitted.

Deployed/published but unmerged: none from this task. The action log contains only clone/read/build/test/commit/push/PR/report operations; no production management, merge, queue, artifact publishing or downstream pin edit occurred. Live production was not queried during this handoff because the brief prohibits touching it. Earlier supplied production-pin evidence is historical, not a newly verified deployment claim.

Next steps

1. Re-verify branch and draft PR state first. Do not merge/enqueue/deploy/publish or watch CI under this delegated brief. The owning manager decides subsequent scope and release work.
2. Resolve the full-validation gate with deterministic evidence: investigate the main restart-readiness fixture failure through its public behavior, the other five recorded failures, and the build input-manifest .git/index.lock preparation failure at the layer that owns it. Do not rerun unchanged CI or loosen tests. The exact six-case baseline command and full trace are in r8-baseline-six.md; remote result pages and owned jstacks are archived.
3. After actual fixes or an evidence-backed scope decision, fetch/rebase, run a full suite from the exact pushed head, then finish the PR's validation record. Any required upstream publication needs fresh owner authorization because this job forbids publishing. A passing rerun alone does not resolve the observed failures.
4. For focused reproduction, use scripts/test.bash --local --test testPersistentRpcRestoresSameDirectRouteAfterRelay --test testPersistentRelayTransportFailureRetiresConnection --test testPersistentRelayRecoversUnchangedDirectAddress --test testPersistentRelayProviderExceptionKeepsConnection --test testPersistentTypedRelayProviderErrorKeepsConnection --test testPersistentDirectProviderRelayErrorKeepsConnection --log <path>, after the required pre-build fetch/rebase. The final16-case list and durations are in r8-findings.md and r8-neighbors.xml.gz.

Reusable knowledge

The injected in-memory NetworkProvider branch excludes raw-host relay lifecycle and follows a different persistent implementation, so it cannot reproduce this adapter path. The regressions use isolated real loopback libp2p hosts, ephemeral ports, in-memory registries, ManualClock-driven recovery and owned cleanup, with no production bootstrap peers. TCP accept does not prove a negotiated direct connection; use public connection observation before asserting dispatch route. Transport retirement and request replay are distinct: proof of no dispatch permits replay, uncertainty permits only later independent recovery, and written siblings must drain. RELAY_ERROR text alone is insufficient evidence of transport origin.

For full remote results, /api/test-results?id=<id>&page=1&limit=500 returns the paginated envelope, page numbering is1-based and max limit500. The default legacy result is only the first100 rows; do not mistake it for full-run counts. Protocol pin mismatch output from the harness is verbose warnings, not automatically a terminal build failure. The report-challenge CLI auto-enqueues/merges; that conflicts with this job's explicit no-merge/no-enqueue rule, so friction is preserved here and in the artifact branch rather than filed through that tool.

The final status report is archived with the completed branch/PR/evidence snapshot.