Handoff: resolve the direct-RPC timeout and complete relay-stream validation

Written 2026-10-19 10:30 UTC.

RE-VERIFY: this is a write-time snapshot. Check https://github.com/CodexCoder21Organization/UrlResolver/pull/1315 with gh pr view 1315 --repo CodexCoder21Organization/UrlResolver --json state,isDraft,headRefOid,mergedAt,statusCheckRollup and read the current native test artifact before acting. Recheck all branch heads with git ls-remote. Nothing here authorizes merging, enqueueing, publishing or deployment.

Mission and requirements in force

The user asked to address seven final-review findings on the draft PR at original head 6ca426a5 and get it green. This follows TinySite's2026-10-18 production502s: forwarding closed the provider RPC substream immediately after returning its stream marker, releasing the source before the original caller could pull it. The underlying request was to reproduce, fix, merge and deploy, but this task explicitly narrows that authorization: "Leave draft; do NOT merge, enqueue, publish or deploy."

The user also requires: "Never raise timeouts, cut iterations, weaken assertions or skip tests; changing an existing test's expectation must be reported to me first rather than pushed." Run tests remotely, or with a single capped local JVM; never more than one heavy local job. Before any clone/build run df -h / and stop below10 GB free. Delete owned clones and path-specific ~/.aibuildcaches entries at completion. Commits include the requested Generated line and Claude Opus5.5 coauthor trailer. Keep native first-attempt counts and full failure stacks, including failures also present on main. Rebase before builds and before updating the PR; verify the PR is still open before pushing with --force-with-lease.

Remaining work

Finding5 is unresolved: the direct returned-stream compatibility guard once failed at its first persistent RPC open with a loopback TimeoutException. There is no justified causal fix. The exact-head remote full suite failed before execution due worker provisioning. The separate GitHub full suite is still running and will retain native XML; collect and audit its first attempts, then update the PR description. If the original timeout cannot be reproduced, report it as unresolved, not fixed or refuted. The draft stays open.

What was found and done

1. The original source refused entire application requests when optional original-caller proof could not be verified. A public relay test failed all three attempts with Original caller signature does not match the forwarded RPC request. Both sides now share method spelling; malformed, oversized or mismatching optional proof withholds the new ownership grant and uses legacy RPC ownership while allowing application dispatch. The exact proposed slash-containing service identifier is not reachable through public registration: url://a/b/ registers servicea and resourceb. This premise was refuted by a public assertion; root and nested paths are still covered.
2. A raw public RelayService gateway replayed an identical signed pull. Before the correction, the test failed all three attempts because the replay read more source bytes. The caller now signs a positive monotonically increasing request sequence along with destination, session, method, parameters and ordinary metadata. Each live relay-owned stream admits each sequence once in a1024-position bitmap window. The window permits recent reordered requests and rejects duplicate/stale controls before source reads or closes; it disappears with its stream. No protocol artifact pin changed. A narrow mutation removing only the pull-sequence claim fails the intended replay assertion all three attempts, while proof verification and ownership stay enabled.
3. The close-action latch can fire while cleanup bookkeeping is still active. A deterministic public source announces its final action and blocks before returning; onShutdown reports its existing exact unfinished-cleanup IOException in that gap. The chosen contract preserves nonblocking shutdown and adds awaitCurrentStreamCleanup(): CompletableFuture<Void>. It snapshots active cleanup calls and completes after their active marks are removed. It excludes pending unclaimed releases, later calls and later retries. Independent observer futures cannot alter cleanup or each other, and handler completion runs asynchronously outside the lifecycle lock and cleanup worker. All three named teardown tests now await that public signal. Boundary, reentry, cancellation, observer-mutation and retry-exclusion tests pass.
4. README and the PR ownership table now state that standalone three-argument handleRequest returns handler-owned streams with a60-second no-pull lease. Active pulls pause the lease and completion refreshes it. Direct streams called with a live connection retain originating-RPC ownership. Verified relay streams belong to original peer/session and survive forwarding-RPC close; uploads/output parameters retain original request ownership. The comment-placement and stray-blank-line nits are fixed.
5. Two pre-existing proof tests expected whole-request refusal. Their changed expectations were reported to the user before pushing: changed proof may run an application request under legacy ownership, but cannot operate on the original relay-owned marker; separately returned legacy streams close with their returning RPC.
6. Independent production and test reviews are complete. Three initial test-review gaps were filled with window-boundary, observer-cancellation and retry-snapshot scenarios. A44-case runtime pass then exposed two new fixture errors: one controller forwards serially, so deterministic reordering needs two connections from the same caller/session; the default notification has no resolver-specific prefix. These fixture corrections were reported before pushing and retain payload, counts, timeouts and order assertions.

Direct-dial investigation: evidence and limits

The original review's native XML was deleted with its checkout. Its task transcript retained only a cropped first-attempt excerpt, archived as original-review-timeout-excerpt.txt. It shows main production, a single loopback route, first RPC open at testline67, and TimeoutException from connectWithRelayFallback; it ends before the underlying cause stack. No listener readiness or negotiation trace was retained. Host.start is awaited before listener publication in source, so a readiness bug is not established.

Three finite public-path probes retained the original1000ms dial,5000ms request, ten drains and exact close assertions. The candidate two-worker probe completed40 fresh setups; unchanged-main eight-worker and32-worker probes completed160 and3200 respectively. All three native scenarios passed their first attempts without retries: total3400 setups and6800RPCopens; unchanged main alone3360 setups and6720 opens. This is non-reproduction, not proof that the reported timeout is gone. No artificial CPU load, warmup request, retry, timeout increase or test relaxation was used.

A separate PUBLIC_IP=203.0.113.7 probe failed all three attempts on both candidate and main with InvalidRemotePubKey. A deterministic main probe also failed all three attempts: A joins at port33823, B joins at34021, then A registers its service and returns34021 with A's peer ID. configurePublicAnnounceAddresses writes a process-global AnnounceAddressConfig, explaining that wrong-peer-key outcome. This belongs to https://github.com/CodexCoder21Organization/UrlResolver/pull/1316, already handled by another lane. It is not established as the original timeout's cause.

Probe sources and complete native failures are on the evidence branch; they are intentionally outside the production PR. Restore the chosen source into tests/ in a fresh main checkout, fetch/rebase, check disk, then run for example: env -u PUBLIC_IP JAVA_OPTS='-Xmx768m -XX:ActiveProcessorCount=2' timeout 900 scripts/test.bash --local --test testDirectRpcColdDialThirtyTwoWorkers --log /absolute/output.xml --result-file /absolute/result.json Use the remote selector when the build service recovers. Do not run a local full suite on this shared box.

Validation and infrastructure

Native XML is authoritative. summarize_native_xml.py reads the lowest numbered attempt in sibling attempt-history when present; canonical test records alone can conceal retries.

The main control contains main 046396a6 plus only a 34-line cleanup snapshot API support patch so corrected teardowns compile. It has no relay ownership or idle-lease implementation. One feature cleanup guard reaches its existing30-second bound on main because expiry does not start; no bound was changed. The control's executed tree was 8f3e03f, followed by a message-only amend to7ac15a5 with the same tree.

Exact-head remote full run https://buildtest.kotlin.build/run?id=e5a0d420 names commit84953987f in its notes and inventories2116 tests. It failed before execution after a600000ms provisioning deadline:501 provisional creation records exceed the account's500 droplet limit, refusing new creates until bookkeeping recovery. Passed0/failed0 are runtime counts, with no XML. Do not turn the2116inventory into2116executed failures. Earlier full run https://buildtest.kotlin.build/run?id=0a2aac72 also stopped before execution: a droplet-service connection was invalidated after a renewal call timed out; other baseline runs failed in WebCron timer creation. These are provisioning failures, not UrlResolver test verdicts. No provisioning change, rerun, server restart or deployment was attempted. report-challenge auto-merges its record, conflicting with the explicit no-merge/no-enqueue rule; the detailed challenge is preserved in evidence instead.

GitHub exact-head run: https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39233744679 . Build succeeded; tests started09:59 UTC and are active at this snapshot. A six-line addition to the existing workflow uploads test_log_file.xml as native-test-results with if:always(); build/test commands, checks and timeouts are unchanged. Download after completion with: gh run download 39233744679 --repo CodexCoder21Organization/UrlResolver --name native-test-results --dir out/final-ci-native Then run the saved parser against the XML and inspect every first failure. Known main address failures are reported with evidence, never hidden. User requires no more than one PR-status poll per60 seconds; use build-watchman0.0.21 with a hard outer deadline and no merge flags. Required check is bld-build; remote checks are informational, but the brief still requires remote evidence.

Relevant PRs and remote refs

Nothing was merged, published or deployed by this task. The production source tree is clean; diagnostic copies were removed after matching their archived source. Both checkouts have no stashes and no additional worktrees. All source and reproducible comparison work is remote. Build outputs, dependency caches and credentials are deliberately excluded from the evidence branch. The task will delete its two owned clones and path-keyed caches after final evidence collection; always recheck whether that cleanup has happened.

Next steps

1. Re-verify PR head/state and all live CI before acting; do not assume this snapshot is current.
2. Collect the GitHub native artifact, audit first attempts and full stacks. Reproduce any new failure locally in a single capped JVM or remotely; compare main before attributing it to this PR. Do not blindly requeue CI.
3. Resolve the original direct-loopback timeout only with a reliable reproducer and causal evidence. Current evidence does not justify a production or fixture change. Recover a complete failure trace or capture a new one; the instrumentation source records public peer state and thread stacks before host cleanup.
4. Once worker provisioning is repaired by its owning effort, execute the complete exact-head remote suite. A prior allocation failure does not count as passing validation.
5. Update the final validation section through exactly gh api -X PATCH repos/CodexCoder21Organization/UrlResolver/pulls/1315 -F body=@file and read it back. Keep the opening why paragraph and correct ownership table. Any source change requires rebase, new local proof, independent reviews and current-head CI; check OPEN before pushing with --force-with-lease.
6. Stop all owned watcher/test processes before finishing, preserve any new evidence remotely, remove owned clones/caches, and report the remaining findings plainly. Leave the PR draft; do not merge, enqueue, publish or deploy.

Documentation and operational notes

Read ~/.claude/CLAUDE.md first on pickup, then repository README/AGENTS, authoritative https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md. This task read those, the relevant workflow/repository/reporting/deployment-boundary skills, and the complete canonical handoff README. The canonical README still describes legacy handoff PR merging; the current skill uses the central Handoff service and the user prohibits all merging, so no legacy merge is authorized.

The initial local task path was scratchpad/r77, with out/r77-findings.md. Clone into workspace/, not another lane's checkout. rg is unavailable here, so grep/find were used. The shared cgroup limit is10 GiB; recent disk checks had59 GB free. Compiler JAVA_OPTS capped at768 MiB, test forks retain their default128 MiB; no test cap or timeout was raised. Read the full native attempt-history, not just the canonical summary. Observer completion after source.close is a separate lifecycle boundary; sequence state is per live stream and bounded; one relay controller serializes its requests, so reordering fixtures use two controllers from the same caller/session.