Final-review and merge https://github.com/CodexCoder21Organization/UrlResolver/pull/1155; delegate work complete

RE-VERIFY 2026-10-02 14:30:35 UTC (lane 000h-rev-ur1155). This is a write-time snapshot. Re-check gh pr view 1155 --repo CodexCoder21Organization/UrlResolver --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, and use git ls-remote for branch heads. The complete prior Mission, findings chain, and all branch/PR rows follow below as historical snapshots.

Mission and review verdict

Original mission: fix stale-registry relay registration with fail-first reproduction and unchanged original three-second ACK bound. This lane performed test-comprehensiveness review first, then a separate adversarial implementation review, and leaves final review, merge, queue, deploy and handoff completion to the supervisor.

REVIEW_CLEAN 6025ccac6973a609c5ea48cdbb6643a4e36c5d8a. No production regression was proved. PR description links were corrected; no new production source or test change was made, and the assigned PR head was not rewritten. This verdict retains the explicit unproved review questions below; it does not claim a resource-growth or scheduling-independent proof that the tests do not provide.

Verified evidence

1. PR is OPEN at the assigned head. Latest read at 14:28 UTC showed bld-build and kotlin.build (remote) SUCCESS, informational kotlin.build (kompile-remote-build) FAILURE, no comments, newest commit 2026-09-30 20:38:57 UTC. The prior lane established why the informational result is not a failed required gate. No new CI or remote submission was made here.
2. Main-code baseline (only UrlResolver.kt reverted to origin/main): four executed tests, two passed and two failed. AddedDuringRetryBackoff and AddedDuringPass fail for missing ACK within the unchanged 3000ms bounds. ChurnOfTriedAddresses and BeyondCandidateCap pass. The instruction that all four must fail on main is refuted by direct execution.
3. Actual corrective fail-first baselines are confirmed: churn fails on https://github.com/CodexCoder21Organization/UrlResolver/commit/ae17cdaf18fbeb22361b1aa8430111cfd0865fbc because already-tried addresses keep starting passes during the 5000ms window; cap fails on https://github.com/CodexCoder21Organization/UrlResolver/commit/bdb1fa88d672c7605599f7c366e37ef994156376 because an unselected seventh candidate starts a pass 7ms after its add. Full assertion messages and stack traces are preserved on the review branch.
4. Fixed source passed all four changed tests in each of three requested local batches: 12/12 passing, each preceded by fetch/rebase and serialized under the shared lock. An additional documentation-checkpoint batch never started: flock -w 1500 exited 1 after 25 minutes with an empty log (zero test or preparation bodies). Source, tests, build.kts and scripts are identical to the already-passing checkpoint; the exit protocol pushed review evidence only. Main baseline: 2 passes / 2 expected failures; corrective baselines: 0 passes / 2 expected failures. Total executed bodies: 14 passed, 4 expected baseline assertion failures; 1 separate lock-acquisition failure with zero bodies executed. No preparation failure, raised timeout/deadline, reduced iteration, disabled test, or weakened assertion.
5. The private selector preserves old eligibility/configured merge code, tier sorting/stable order and MAX_GUARD_NODES * 2 cap. Identity changes alone do not reset remainingBackoffMs or skip delay: tried-address churn cannot spin that wait. Truly new selected addresses intentionally permit early passes.
6. Tests use public UrlProtocol2 APIs and real isolated loopback sockets, no mocks/reflection. Original stress test is unchanged. No new production exception/effect swallowing or contradicted README contract was found. Diff contains intended source/tests plus deliberately retained review evidence, no generated build output.
7. PR WHY opening was preserved, and full handoff/WUI and historical run URLs were added. gh 2.23.0 pr edit failed on retired projectCards GraphQL field; REST PATCH succeeded. Exact tool error/workaround is in findings. One Handoff title refresh also failed on bytecode read 0 of 4; its second idempotent attempt succeeded, and the full diagnostic is preserved remotely. report-challenge CLI was not invoked because it would enqueue/merge, forbidden here.

Explicit open review questions

• The tried-address HashSet has a task lifetime but no fixed entry cap: six selected peers bounds one selection, not historical distinct keys. The comment is a temporal/input-dependent bound. No memory failure has been reproduced; blindly evicting old keys would recreate the already-tried churn bug.
• Bookkeeping samples candidates separately from the actual pass, and passes can coalesce. A concurrent replacement can make the recorded offered set differ from the dialed set. No resulting public-API regression has been reproduced.
• The cap test's <=20ms probe-pair detector and the tests' elapsed-gap/silence inference lack a public scheduling guarantee. All requested local repetitions pass, but that does not prove those triggers are independent of host scheduling.
• New tests do not directly cover close during this backoff, concurrent replacement between selection snapshots, large distinct history, or eventual loop progress after the churn test's quiet window. Existing close/coalescing neighbours were reviewed as coverage context; they were not rerun by this review lane. These remain questions, not silently claimed completed tests.

Added review branch and artifacts

Repo	Branch	Remote head SHA	PR	What is on it	State
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/000h-rev-ur1155-2026-10-02	e1657639bedf49230d95609b76e445ed76c72f4a	https://github.com/CodexCoder21Organization/UrlResolver/pull/1155 (reviewed; branch not rewritten)	Existing implementation rebased onto main, reviews/000h-rev-ur1155/findings.md, full baseline traces and passing summaries	Durable review record, not a separate fix PR; local verification complete

Findings: https://github.com/CodexCoder21Organization/UrlResolver/blob/wip/000h-rev-ur1155-2026-10-02/reviews/000h-rev-ur1155/findings.md. The rebase includes main's already-merged active-query changes; relay-registration code and all four reviewed tests are unchanged from the assigned PR. Generated jars/cache/log bulk are excluded; relevant complete failure traces and results are retained.

What remains

Supervisor final review and merge decision for https://github.com/CodexCoder21Organization/UrlResolver/pull/1155, including judgment on the explicit unproved questions and the normal integration gate. The review delegate does not merge, queue, deploy, publish or complete this handoff. No production code was deployed/published by this lane, and no test/watch process remains after exit.

---

Previous body retained in full

Finish relay retry review repetitions, then final-review and merge https://github.com/CodexCoder21Organization/UrlResolver/pull/1155

RE-VERIFY 2026-10-02 13:58:33 UTC (lane 000h-rev-ur1155). This is a write-time snapshot. Re-check PR state/head/checks with gh pr view 1155 --repo CodexCoder21Organization/UrlResolver --json state,headRefOid,statusCheckRollup,mergedAt, and branch heads with git ls-remote. All earlier Mission, history and branch/PR rows are retained below.

Review lane checkpoint

Mission carried forward: fix the stale-registry relay-registration flake with reliable reproduction and unchanged original bounds. This lane reviews test comprehensiveness first, then implementation, without merge, queue, deploy, publication or handoff completion.

• Reviewed assigned PR head 6025ccac6973a609c5ea48cdbb6643a4e36c5d8a. No unaccounted recent commit/comment. Required CI was already green, as established by ur1155; no new CI launched or requested here.
• Main-code baseline executed all four added tests, with only UrlResolver.kt reverted. AddedDuringRetryBackoff and AddedDuringPass failed for missing ACK within 3000ms; churn and cap passed. The instruction that all four fail on main is refuted.
• Verified the actual corrective baselines too: churn fails on https://github.com/CodexCoder21Organization/UrlResolver/commit/ae17cdaf18fbeb22361b1aa8430111cfd0865fbc because already-tried churn repeatedly starts passes; cap fails on https://github.com/CodexCoder21Organization/UrlResolver/commit/bdb1fa88d672c7605599f7c366e37ef994156376 because the unselected seventh candidate starts a pass 7ms after add.
• First fixed-source batch passed 4/4 on the locally rebased implementation. 0 requested repetitions remain running under the shared local-build lock. Current cumulative executed bodies: 14 passed, 4 expected baseline failures, 0 preparation failures.
• Selection extraction preserves eligibility/merge body, tier sort/stable order and MAX_GUARD_NODES * 2 cap. Snapshot changes alone do not reset the remaining backoff or bypass its delay.
• Open questions, not established production defects: history has no fixed cardinality cap (only task lifetime); bookkeeping and actual pass select independently; tests infer pass identity from timing (particularly <=20ms probe pairs in cap test), so scheduler-independent triggers and close/concurrent-selection coverage remain review concerns. Full details and behavior/test map are in the checkpoint findings.
• Corrected PR description to include full handoff/WUI and historical run URLs. GitHub source/test head unchanged. gh pr edit hit retired projectCards GraphQL field; REST PATCH succeeded. report-challenge CLI not called because it would enqueue/merge, forbidden by this lane.

Added review branch

Repo	Branch	Remote head SHA	PR	What is on it	State
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/000h-rev-ur1155-2026-10-02	510e0ffc213f5db7f70c820d179a0740dad4309b	Existing reviewed PR: https://github.com/CodexCoder21Organization/UrlResolver/pull/1155	Rebased existing fix plus reviews/000h-rev-ur1155/findings.md and full main fail-first traces	Review evidence only; fixed batch 4/4 passed; no PR rewrite

Remote findings: https://github.com/CodexCoder21Organization/UrlResolver/blob/wip/000h-rev-ur1155-2026-10-02/reviews/000h-rev-ur1155/findings.md.

Remaining: finish two fixed repetitions, push final review evidence and refresh verdict. Supervisor retains final review and merge decision, and decides whether the explicit unproved coverage/resource questions need further experiments. Nothing deployed or published by this review lane.

---

Previous body retained in full

Final-review and merge https://github.com/CodexCoder21Organization/UrlResolver/pull/1155; delegate work complete

RE-VERIFY 2026-10-02 12:50:18 UTC (lane ur1155). This is a write-time snapshot. Re-check with gh pr view 1155 --repo CodexCoder21Organization/UrlResolver --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, inspect the check outputs and main required-check rules, and use git ls-remote for every branch below. The full previous body follows the divider as historical evidence; none of its older state claims replaces this snapshot.

Mission carried forward

The original mission remains to fix stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale, with reproduction before a fix and no loosening of its three-second relay-registration wait. This pickup was narrower: settle the one red check on the existing fix PR, reproduce any relevant failure locally, preserve the evidence and leave final review, merge, enqueue, deploy and handoff completion to the supervisor.

Current finding and actions

1. The PR is OPEN at 6025ccac6973a609c5ea48cdbb6643a4e36c5d8a. The only red check is informational kotlin.build (kompile-remote-build): https://github.com/CodexCoder21Organization/UrlResolver/runs/110114451491. Main rules require bld-build and kotlin.build (remote), both SUCCESS. build-watchman 0.0.21 exited GREEN, 0, against that exact head. No new CI run was submitted or requested.
2. The informational failure names remote route run 9fc96403-f0f4-4b39-bb14-31ff84b6579a, which remained recorded as TESTING. Its getCiBuild status read failed because BuildTestOperationalApi.getBuildRuleResults exceeded the sandbox conversion limit: 10002 values examined, limit 10001, zero values converted; the next value was a Boolean. The runner later hit its unchanged 180-minute overall deadline. The recorded mechanism is result rejection at the conversion boundary, not a relay assertion. No failing test name appears in the check output. The informational run's test count remains unverified; do not label it a zero-test run.
3. Required run https://buildtest.kotlin.build/run?id=178b5ae7 reports 1883/1883 passing, with 13 tests passing after retry. Pages 1, 2 and 4 directly confirmed 1383 individual passing results, including the original stress test (18171 ms). Page 3 returned 503 with a full body saying its cached view was rebuilding within the existing 5000 ms foreground budget. This adds no test failure and is not evidence that main is free of flakes.
4. The existing public-API conversion-boundary test passed 3/3 local invocations against published foundation.url:resolver:0.0.1302, including the post-rebase pre-push gate. Only its dependency changed from the source build rule to the published artifact. The full error assertion, 10001-element input, 120-second annotation and cleanup were preserved. It reproduces the same 10002-versus-10001 rejection before allocation (String leaf in the fixture, Boolean leaf in the CI result). This proves the conversion condition; it is not a complete reproduction or fix of the CI consumer's getCiBuild chain.
5. The targeted original-test attempt on unchanged main 1053ace97e38d3a3ee85294b07a3f8465adf6fd0 failed during foundation.url.resolver.buildMaven, at the existing 20-minute build-script deadline. Its test body never ran. Local counts are 3 passing conversion invocations, 1 original-target preparation failure, and 0 original-target bodies executed. No timeout, iteration count or assertion changed. The complete preparation-failure trace and two advancing compiler thread dumps are on the checkpoint branch. No blind rerun of that source build was performed; its compilation mechanism is not established.
6. Read-only review covered the five-file PR diff, all four added regression tests and the unchanged original stress test. Candidate eligibility, tier order and the six-candidate cap are shared between the pass and early-backoff check. The cap test verifies both no early pass and eventual loop activity. No new defect was established. The prior fail-first evidence is retained below. Supervisor final review remains required.
7. No resolver workaround, raised conversion limit or PR rewrite is justified by the informational status-reader error. The PR's sandbox conversion guard and original stress test are unchanged relative to current main. Main is three commits ahead of the PR base; this lane did not rewrite the already-green PR solely to rerun CI. The supervisor retains its normal merge-queue integration gate.
8. The older handoff checkpoint claim was stale: wip/K25-relay-retry-r2-untried-addresses is actually bdb1fa88d672c7605599f7c366e37ef994156376, not the PR's current head. Earlier rows below remain historical. The existing campaign dependency and blocked-reason metadata were not changed by this narrow lane; the supervisor may reconcile that metadata when closing out the work.

Current remote branches and PRs

Repo	Branch	Remote head SHA	PR	What is on it	State
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/relay-registration-wakes-on-new-registry-address	6025ccac6973a609c5ea48cdbb6643a4e36c5d8a	https://github.com/CodexCoder21Organization/UrlResolver/pull/1155	Existing relay retry fix and four regressions	OPEN; required checks green; informational check red
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/K25-relay-retry-r2-untried-addresses	bdb1fa88d672c7605599f7c366e37ef994156376	https://github.com/CodexCoder21Organization/UrlResolver/pull/1155	Older round-2 checkpoint	Superseded by PR head; historical test evidence below
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake	d9dead42a9805a5868f68e3605f94cd4b83fd42c	No PR	Earlier diagnostic work	Historical only; not the verified fix
UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/ur1155-2026-10-02	b223620e63a040268a147fe124b66030afaccb4a	No PR	Check triage, complete error traces, conversion diagnostic source and compiler dumps	Analysis only; production and existing tests unchanged; conversion check 3/3 passed

Checkpoint commit: https://github.com/CodexCoder21Organization/UrlResolver/commit/b223620e63a040268a147fe124b66030afaccb4a. Recoverable findings and exact reconstruction command: https://github.com/CodexCoder21Organization/UrlResolver/blob/wip/ur1155-2026-10-02/investigations/ur1155-check-triage/findings.md.

Next steps

1. Supervisor: re-verify the exact PR head and required checks, perform final review and make the merge decision. This lane never merges, enqueues, deploys or completes the handoff.
2. Keep the earlier reproduced-before-fixed relay evidence and unchanged waits. The red informational result does not call for another resolver change. Any CI result-reader repair needs a complete consumer reproducer at its owner; no such repair is claimed here.
3. After the supervisor's normal merge and integration process succeeds, it can reconcile the campaign/dependency metadata and complete this handoff. The analysis branch is a diagnostic record, not another fix PR to merge.

Operational knowledge and live-code record

The first progress report upload failed while fetching HandoffService bytecode: direct read closed at 0 of 4 bytes, followed by bounded relay dial-task errors. The second upload and the next progress report succeeded. The report-challenge CLI was not invoked because it automatically enqueues and merges, which this lane forbids; its evidence is retained in the findings.

gh auth setup-git was needed for git operations after cloning. rg is unavailable, so grep was used. The first checkpoint commit lacked an author identity; repository-local identity was set, then the actual analysis commit was verified by matching HEAD with git ls-remote. The working tree and stash are clean. Diagnostic sources use .txt suffixes so normal test discovery does not include them. Generated jars, caches and XML result files were deliberately excluded; error text, test counts, reconstruction commands and useful compiler dumps are preserved remotely.

Nothing was merged, enqueued, deployed, restarted, routed or Maven-published by this lane. Existing published resolver 0.0.1302 was read only for the diagnostic test. No watcher or local test remains running.

---

Previous handoff body — retained in full

Stale registry relay address flake: round-2 fix pushed (CI, reviews, final review and merge remain)

RE-VERIFY 2026-09-30 19:35 UTC (board drain, lane K25, round 2). Everything below the next horizontal rule is the earlier write-time record. Re-check the PR head and its checks before you act.

Mission

Fix stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale. Most recently it failed in CI run https://buildtest.kotlin.build/run?id=b2ead274 (1 failure in 80 executions) with "Client 1: relay registration should use the newer registry address. Expected <peer>, actual <null>".

Mechanism (proven)

When the only known address is unusable, the relay-registration retry loop doubles its backoff after every failed pass (100 ms up to 10 s). A reachable address that later lands in PeerRegistry wakes nothing: PeerRegistry has no add notification, and neither peer exchange, gossip nor a direct addPeer restarts the task. That happens in two windows:

• between passes, during an escalated backoff;
• during a pass. A pass can block for up to 10 s waiting for acknowledgements, and it took its snapshot before the add. The stress test's first client hits the second window.

Fix (head ba1025bf; round 3 limits the early exit to addresses the next pass would dial, via the shared selectRelayRegistrationCandidates)

• The retry task records every candidate address that any of its passes was given.
• The backoff's first check runs before any delay and always recomputes the candidates. Later checks run every 100 ms and recompute only when the registry snapshot changes identity.
• Only an address that no pass has tried ends a backoff early, so churn of already-tried addresses keeps the backoff.

Tests (red-first evidence)

• testRelayRegistrationAdoptsRegistryAddressAddedDuringRetryBackoff: red on main 4/4, green 3/3.
• testRelayRegistrationAdoptsRegistryAddressAddedDuringPass: red 6/6 on pre-fix code (ae17cdaf 4, main 2), green 4/4.
• testRelayRegistrationBackoffIgnoresChurnOfTriedAddresses: red 3/3 on ae17cdaf, green 3/3.
• The original stress test passed 10/10, and 13 relay-registration neighbours passed 13/13.

PR and branches

Item	Link	Head
Fix PR	https://github.com/CodexCoder21Organization/UrlResolver/pull/1155	6025ccac6973a609c5ea48cdbb6643a4e36c5d8a (branch fix/relay-registration-wakes-on-new-registry-address)
Checkpoint	https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/K25-relay-retry-r2-untried-addresses	same head

Nothing was published, deployed, merged or enqueued.

Next steps (remaining)

1. Branch CI green on 6025ccac, including kotlin.build (remote).
2. Re-review against the new head (round-1 review R1155 was on ae17cdaf).
3. Final review, then the merge decision.
4. Complete this handoff once PR 1155 has merged. It is item 3 of the UrlResolver campaign handoff.

---

Stale registry relay address flake: still recurring in CI (1 of 80 executions), not reproduced locally (0 of 32)

RE-VERIFY 2026-09-30 16:15 UTC (board drain, lane K21). Everything below this section is the earlier write-time record. Re-check any claim here before you act on it.

Verdict: KEEP OPEN. The flake recurred in CI after every fix that landed up to 09-29.

Test: stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale (tests/stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale.kts, @Timeout(120), 20 sequential clients).

CI evidence (buildtest.kotlin.build /api/runs pages 1-4 plus /api/test-results?limit=500 for every run)

• Window: 147 UrlResolver-labelled runs, 2026-09-29 05:37 to 2026-09-30 14:00 UTC. The target test executed in 80 of them: 79 PASSED, 1 FAILED. The rest were FILTERED_OUT (targeted runs, 54), executed nothing (12), or did not list the test (1).
• FAILED: run https://buildtest.kotlin.build/run?id=b2ead274 (started 2026-09-29 07:30 UTC). Head https://github.com/CodexCoder21Organization/UrlResolver/commit/200e8e854029962eed41af85efaa0bcdc5ae9276 of https://github.com/CodexCoder21Organization/UrlResolver/pull/1150. Relative to its merge base that head changes only four unrelated test files, and it already contains https://github.com/CodexCoder21Organization/UrlResolver/pull/1117 (ffaac8c79) and the protocol 0.0.551 pin (28e146a25). The run itself was degraded (after a service restart, 6 of 10 shards failed during resume). The target test still executed, on droplet 604590089, and failed on its own assertion in 14,090 ms:

java.lang.AssertionError: Client 1: relay registration should use the newer registry address. Expected <12D3KooWDH4HvbMVwq8dfo8Aq583nCqtzBeqfeC8Sa1ArRZnC9Zj>, actual <null>.
	at foundation.url.resolver.StressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStaleKt.stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale(stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale.kt:86)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)

The run recorded no stdout for the test. This is the same assertion as the 09-24 failures in runs 387fe030 and a4c988cd on main 9553becc.

• git log -S<testName> origin/main finds no commit that adds or removes the name since 09-16. Since 09-16 the test file changed only in 78b23e73c (https://github.com/CodexCoder21Organization/UrlResolver/pull/1106, 09-22) and 28e146a25 (protocol pin, 09-28). Source commits after the failing head (9c6ac572d / PR 1152, 4a7d6259f / PR 1144, b2cf58853 / PR 1147) do not target relay registration.

Local evidence

• 32 of 32 PASSED on origin/main https://github.com/CodexCoder21Organization/UrlResolver/commit/4a7d6259f69b12e6a84373ee9400a9d21d12be3f, using scripts/test.bash --local --test <name>. Runs were one at a time, and each started only when cgroup memory was below 7.5 GiB. Test times ranged from 10.1 s to 17.2 s (mean 13.3 s). A watchdog would have taken a thread dump of the test JVM at 96 s (80% of the timeout); it never triggered.
• Local isolation does not reproduce the flake. That matches the 09-16 record (0 target failures in 50 runs).

Mechanism

• OBSERVED: in the failing runs, getConnectedRelayPeerId() stayed null for the test's whole 3 s wait. For client 1, addBootstrapPeer(stale /tcp/1 address) was followed by getPeerRegistry().addPeer(valid address). This is test setup, before findPeersForService.
• OBSERVED (source, UrlResolver.kt around lines 7200 and 18057): addBootstrapPeer calls startRelayRegistrationRetryTask right away, while only the stale configured address is known. The retry loop's backoff between passes goes 100/200/400/800/1600 ms, and each pass waits synchronously for candidate acknowledgements. A plain peerRegistry.addPeer does not restart the task. The code comment at that site says so for the peer-exchange and gossip paths.
• INFER (not proven): the first pass or passes use a snapshot that holds only the stale address. The valid registry address is picked up only in a later pass. Under full-suite CI co-scheduling, pass duration plus backoff sometimes adds up to more than 3 s, so no acknowledgement arrives inside the test's window. This would be a real ordering defect: registry additions do not trigger a new relay-registration pass. CPU load only widens the window. It is not the cause.
• No deterministic reproducer exists yet. The next step is to force that exact ordering through public APIs: hold the first pass on the stale candidate, for example with a peer that accepts TCP and never acknowledges, then add the valid registry address and assert that an acknowledgement arrives within 3 s. That should fail first on main. Then fix it at the cause, most likely by having a registry addPeer of a relay-capable peer start or coalesce a new pass while no relay is acknowledged. The fix must not raise the test's 3 s bound.

State

• Nothing was changed, pushed, published or deployed in this re-verification. The diagnostic branch https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake (head d9dead42) stays as reference only. It is 09-16-based and its reserved coordinate is below main.
• This is still item 3 of the UrlResolver campaign handoff (dependency above). Complete this handoff only when a fix PR with a fail-first deterministic test has merged.

---

Reproduce the stale registry relay address flake before proposing a fix

Written: 2026-09-16T04:18:18.516917+00:00

RE-VERIFY: This is a write-time snapshot. Fetch the assigned branch and origin/main; compare heads, inspect the recorded cohorts, and obtain fresh original-test failure output. No fix has been established. Do not treat this diagnostic branch as a verified fix or merge it.

Mission: root-cause and fix stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale in its own UrlResolver PR. Historical sightings are linked in the investigation below. The assignment explicitly requires >=50% reliable failure before a mechanism/fix, then a deterministic fail-first test and 20/20 amplified passes. It also explicitly says to stop rather than guess if amplification cannot reach 50%. That stop condition was reached with 0 target failures in 50 completed runs / 16,400 client scenarios.

Remote work and current state

Repository	Branch	Remote head	PR	Contents	State
https://github.com/CodexCoder21Organization/UrlResolver	https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake	https://github.com/CodexCoder21Organization/UrlResolver/commit/da55840b38bdc828728784b748228c51cb9d6679	None	Reserved 0.0.1237; concurrent diagnostic test; complete investigation	Diagnostic runs pass; target not reproduced; production unchanged

Nothing deployed or Maven-published; no merge, enqueue, close, or PR creation was performed. Source changes relative to main are only the reserved coordinate, the diagnostic workload, and the investigation note. No local-only code remains. Raw generated XML/logs were not committed; exact errors, stacks, counts, cohort revisions, and reconstruction commands are preserved in the investigation. The temporary 1,000-client sequential experiment was removed after proving only outer-budget exhaustion; its exact two-line transformation is recorded below.

The local requested findings file is /tmp/claude-501/-code/5ed32826-94fa-44da-ba65-2a91758bd3bb/scratchpad/out/f3.md. The recoverable GitHub copy is https://github.com/CodexCoder21Organization/UrlResolver/blob/fix/stale-registry-relay-address-flake/investigation/stale-registry-relay-address.md . Fresh clone used /tmp/claude-501/-code/5ed32826-94fa-44da-ba65-2a91758bd3bb/scratchpad/workspace/l3/ur; do not depend on that filesystem surviving.

Next steps and constraints

Obtain a fresh original-test failing assertion and operation trace, preferably from a successful manual targeted remote execution once build-service RPC works. Do not blindly rerun CI. Classify the actual assertion before deciding whether the defect is registration, query/refresh, teardown, or another layer. Make the reproducer reliably fail, force that exact condition deterministically through public APIs, then fix the owning production layer. The currently passing samples neither prove nor refute the historical shared-defect premise.

Keep coordinate 0.0.1237 unless a new brief changes it, verify it remains above main, retain protocol 0.0.524, and never add the forbidden older pin. Never increase timeouts, reduce original iterations, weaken or disable original tests, use reflection/mocks, or expose private state for tests. Do not merge, enqueue, close PRs, deploy, or edit other branches. Any eventual PR must open with the three historical sightings and fail-first evidence. The completed assignment stopped well within its 75-minute budget without guessing.

Investigation record

Stale registry relay address flake

Final outcome: unable to reproduce the target flake; no production fix and no PR. Stopped under the brief's explicit instruction to stop rather than guess when amplification does not reach a 50% failure rate. The historical shared-defect premise is neither proved nor refuted by these passing samples. Started 2026-09-16 03:47 UTC; hard stop 05:02 UTC.

Plan:

• DONE: fresh clone, repository guidance, reserved coordinate 0.0.1237 in first commit.
• STOPPED under the explicit no-guessing rule: reproduction threshold NOT MET; 0 target failures in 50 completed runs. No original assertion text available.
• NOT ATTEMPTED because reproduction gate was unmet: deterministic regression and production fix. No mechanism established.
• NOT MET: fix-verification / PR gate. Diagnostic work and findings pushed; no PR and no PR comment because no verified fix exists.

OBSERVED: Brief reports failures on https://github.com/CodexCoder21Organization/UrlResolver/pull/1071, https://github.com/CodexCoder21Organization/UrlResolver/pull/1078, and https://github.com/CodexCoder21Organization/UrlResolver/pull/1080. Historical details were purged. INFER: shared main defect is a hypothesis requiring fresh reproduction. No mechanism established yet.

03:50 UTC ? OBSERVED: Fresh clone and initial version reservation committed/pushed at https://github.com/CodexCoder21Organization/UrlResolver/commit/d613a4bca (branch https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake). Main coordinate is indeed 0.0.1227 and protocol pin is 0.0.524. No AGENTS.md exists in clone; README and testing guidance read. Original test starts 20 clients sequentially, adds the stale bootstrap via addBootstrapPeer, then the fresh record via registry.addPeer, and waits three seconds for relay ACK before its query. INFER: setup registration ordering may fail before the address-merge query is reached. Baseline targeted remote run underway; no assertion captured yet. Plan first item DONE; reproduction IN PROGRESS.

03:53 UTC ? OBSERVED: Both remote CLI submissions remain inside workspace upload after health=OK, with no printed run ID; a client thread dump shows submitBuildChunked awaiting its RPC response. Prepared honest amplification (100 real client scenarios, four concurrent callers, unchanged original assertions/budgets). Started a single targeted local run with JAVA_TOOL_OPTIONS='-XX:+UseSerialGC -Xmx768m -Dfoundation.url.resolver.debug=true' to obtain prompt diagnostic feedback; no local full suite. INFER: plausible failure locations remain setup ACK, premature service discovery, or query itself. Existing query code already unions configured and registry addresses, so a simple missing merge is refuted by source inspection; no fix chosen.

03:56 UTC ? OBSERVED: Remote baseline failed before executing any test: [FAILED] Remote chunked upload for 'ur' using session 'a400521b' failed after 5 attempts. Last error: foundation.url.resolver.UrlResolutionException: RPC request 'uploadChunk' to service 'buildtest' failed: Persistent RPC connection to service 'buildtest' was closed while requests were still pending. The HTTP dashboard still labels that incomplete upload PENDING. The local targeted amplifier is functioning and its first six runs passed (600 client scenarios); each warm test executes in roughly nine seconds. INFER: remote upload failure is separate infrastructure friction, not a reproduction of the requested test. Challenge recorded here instead of report-challenge-cli because that CLI automatically merges another repository's PR and this brief prohibits merging/editing any other branch. No speculative fix.

03:57 UTC ? OBSERVED: First amplification level completed 10/10 passing runs (100 clients/run, four concurrent scenarios; 1,000 scenarios total). Diagnostic workload checkpoint: https://github.com/CodexCoder21Organization/UrlResolver/commit/5996ce607 on the assigned branch. Starting the second level: 500 clients/run, 16 concurrent scenarios, Serial GC and requested 256 MiB JVM heap, original 120-second test timeout and every original assertion unchanged. INFER: no reliable reproduction exists yet (observed failure rate 0%).

03:59 UTC ? OBSERVED: Live worker VM.flags confirms the test runner sets -Xmx128m itself, overriding the heap requested through JAVA_TOOL_OPTIONS; Serial GC is active. Thus both amplification levels actually use a 128 MiB test heap. The requested parent/build JVM heaps differ, not the test heap. Current heads of all three cited PRs pin protocol 0.0.524 too; this does not establish what their earlier failing heads used. Four completed second-level runs passed. INFER: smaller test heap has already been exercised, and a current-head protocol mismatch does not explain the reports.

04:00 UTC ? OBSERVED: Second amplification level completed 10/10 passes (500 clients/run, 16 concurrent scenarios; 5,000 scenarios total). Combined amplification: 20/20 passing runs, 6,000 scenarios, unchanged production code, effective worker heap 128 MiB with Serial GC. Now repeating the unchanged original test 20 times to cover its sequential lifetime ordering independently. INFER: neither the proposed address merge race nor a query-cache defect has been demonstrated. No mechanism sentence can honestly be written yet.

04:04 UTC ? OBSERVED: Original unchanged test passed 10/10 on starting main 9a3efc2cf. Before run 11, required fetch/rebase found new main 91c78457b (https://github.com/CodexCoder21Organization/UrlResolver/pull/1105, preserve pending RPCs when a gossip child negotiation fails). Resolved only build coordinate conflict, retained assigned 0.0.1237, reapplied diagnostic settings, and pushed rebased branch with force-with-lease. Remaining original runs use the new main and are a separate cohort. Remote amplified submission also failed before reporting a test result: [FAILED] RPC request 'getBuildRun' to service 'buildtest' failed: Persistent RPC connection to service 'buildtest' was closed while requests were still pending. INFER: these transport errors give no evidence about this flake; main evolution prevents combining cohorts as one exact-head experiment.

Invariant inventory for diagnosis (no production change proposed):

Resource/state	Creation and owner	Terminal behavior	Public observation / candidate regression
Relay node and local service	Test creates relay; registration owns local service; test unregisters and closes	Service unregister and resolver close	Local service is queryable; registration alone should not populate client service discovery
Client resolver	One real resolver per client scenario, owned by scenario	finally calls close, including assertion failures	Join completion, ACK peer ID, discovered services, findPeersForService result
Configured relay descriptor	Caller installs stale descriptor with addBootstrapPeer	Persists as configuration through registry churn	Registration/query must also consider fresh registry address for same peer ID
Registry relay descriptor	Caller publishes fresh address through getPeerRegistry().addPeer	Registry lifetime/removal owns discovered state	Observe registry peer descriptor and returned provider, never internal fields
Query result	Explicit findPeersForService call	Returned snapshot after query or bounded quiescence	Correct relay provider and exact advertised service; cache/refresh race remains unproven

The mechanism is NOT ESTABLISHED, and the evidence needed to demonstrate one is a fresh failing assertion plus its operation ordering. All completed test runs currently pass; choosing a production fix would be a guess.

04:06 UTC ? OBSERVED: New-main original run 11 never executed the test: the fresh Kotlin compilation hit java.lang.OutOfMemoryError: Java heap space and Compile failure: Non-zero exit code: OOM_ERROR under my 256 MiB parent/build-JVM setting. Stopped the batch during the next compilation. Full failure trace is in /tmp/f3-original-11.log. This is experimental setup error, not the target test failure. Restoring the previously successful 768 MiB compiler setting; the test worker remains at its confirmed 128 MiB cap.

Compiler failure stack (test never entered):

java.lang.Error: Compile failure: Non-zero exit code: OOM_ERROR
	at build.kotlin.jvm.JvmBuildRulesKt$BuildKotlin$1.invoke(JvmBuildRules.kt:1810)
	at build.kotlin.jvm.JvmBuildRulesKt$BuildKotlin$1.invoke(JvmBuildRules.kt:1755)
	at build.kotlin.jvm.JvmBuildRulesKt$inlined$sam$i$java_util_concurrent_Callable$0.call(PrivilegedBuildAssistant.kt)
	at kompile.executionenvironment.BuildscriptRunner.simpleCache(BuildscriptRunner.java:216)
	at build.kotlin.jvm.JvmBuildRulesKt.BuildKotlin(JvmBuildRules.kt:2110)
	at build.kotlin.jvm.JvmBuildRulesKt.BuildKotlinJar(JvmBuildRules.kt:1148)
	at build.kotlin.jvm.JvmBuildRulesKt$buildSimpleKotlinMavenArtifact$1.invoke(JvmBuildRules.kt:1468)
	at build.kotlin.jvm.JvmBuildRulesKt$buildSimpleKotlinMavenArtifact$1.invoke(JvmBuildRules.kt:1467)
	at build.kotlin.jvm.JvmBuildRulesKt$inlined$sam$i$java_util_concurrent_Callable$0.call(PrivilegedBuildAssistant.kt)
	at kompile.executionenvironment.BuildscriptRunner.simpleCache(BuildscriptRunner.java:216)
	at build.kotlin.jvm.JvmBuildRulesKt.buildSimpleKotlinMavenArtifact(JvmBuildRules.kt:2082)
	at build.kotlin.jvm.JvmBuildRulesKt.buildSimpleKotlinMavenArtifact(JvmBuildRules.kt:1447)
	at build.kotlin.jvm.JvmBuildRulesKt.buildSimpleKotlinMavenArtifact$default(JvmBuildRules.kt:1440)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.executionenvironment.BuildscriptRunner.invokeInterceptedLocally(BuildscriptRunner.java:261)
	at kompile.executionenvironment.BuildscriptRunner.visitMethodInsn(BuildscriptRunner.java:255)
	at tools.kotlin.build.runtime.PrivilegedBuildAssistant.visitMethodInsn(PrivilegedBuildAssistant.kt:20)
	at foundation.url.resolver.Foundation_url_resolver_buildktsKt.buildMaven$impl(foundation_url_resolver_buildkts.kt:430)
	at foundation.url.resolver.Foundation_url_resolver_buildktsKt.buildMaven(foundation_url_resolver_buildkts.kt)
	at CmdKt.command(cmd.kt:1)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.executionenvironment.BuildscriptRunner.main(BuildscriptRunner.java:509)

04:08 UTC ? RUNNING status checkpoint: fresh compile succeeded after restoring the parent compiler setting; first three new-main original-test repetitions passed. Combined completed evidence: 20 amplified runs / 6,000 client scenarios on starting main, 10 original runs on starting main, 3 original runs on newer main. No PR, no delegated agents, no production change, no published artifact or deployment. Remaining path: finish original cohort and longer serial workload; if no failing assertion appears, stop under the brief's no-guessing rule and preserve diagnostic branch/findings. Lesson: inspect effective worker JVM flags, because runner command-line flags override parent JAVA_TOOL_OPTIONS.

04:09 UTC ? OBSERVED: Original test repetitions complete: 10/10 passing on main 9a3efc2cf and 10/10 passing on main 91c78457b. All 40 completed actual test executions passed (20 amplified and 20 original; 6,400 client scenarios). The failed compilation and interrupted compilation are excluded because they never entered the test. Starting 1,000 sequential client scenarios in one test invocation, preserving original 120-second timeout and assertions. INFER: no test assertion or causal mechanism is established; the available observations cannot justify a fix or claim the historical flake no longer exists.

04:13 UTC ? OBSERVED: The 1,000-client sequential experiment reached the unchanged 120-second outer watchdog. Its console contains 441 created resolvers, 438 successful target-service query responses, and zero AssertionError lines; the latest dump still shows the main thread polling the next relay registration. This is a workload-size outer-budget failure, not an established reproduction of the original 20-client flake. Per-client assertions did not fail. Do not count this as meeting >=50% reproduction. Full watchdog trace:

java.lang.RuntimeException: Test 'stressTestRegistryRelayAddressSerialAmplified' timed out after 120000ms, stuck at: foundation.url.resolver.UrlResolverKt.awaitRelayServerHandlerQuiescence(UrlResolver.kt:303)
	at kompile.WorkspaceKt.executeTestInSeparateJvm(Workspace.kt:2407)
	at kompile.WorkspaceKt.runTests(Workspace.kt:3570)
	at kompile.LocalBuildWorkspace.runTests$kompile(Workspace.kt:1249)
	at kompile.LocalBuildWorkspace.executeInternal(Workspace.kt:570)
	at kompile.LocalBuildWorkspace.execute(Workspace.kt:508)
	at kompile.cli.CliKt.main(Cli.kt:1232)

Experiment recreation from the original test: copy its file, rename function to stressTestRegistryRelayAddressSerialAmplified, and replace val numClients = 20 with val numClients = 1000; preserve @Timeout(120) and all other code. The temporary file will be removed after diagnosis because the outer-budget failure does not discriminate on the requested contract. Its exact construction is retained here.

04:15 UTC ? OBSERVED: Final amplification level (1,000 clients, 32 concurrent callers, effective 128 MiB test heap, Serial GC) has passed its first four runs. No original per-client assertion has failed in any completed workload. Dedicated diagnostic review confirms the concurrent variant preserves original setup, 3-second ACK assertion, empty-discovery assertions, 5-second query assertion, provider/service assertions, and per-client close; Future.get propagates each scenario failure into the run result. No reflection, mocks, external test service, timeout increase, or original-test edit was introduced. INFER: this remains non-reproduction, not evidence that the historical report was wrong.

04:17 UTC ? FINAL experimental result: final amplification level passed 10/10 (1,000 clients/run, 32 concurrent callers). Overall target failure rate: 0/50 completed runs, 16,400 complete passing client scenarios. No original assertion failure and no deterministic bug trigger were obtained. No production source or original test change. The serial outer-watchdog experiment and compilation failures are separately recorded above and do not satisfy the requested reproduction gate.

Main revision	Workload	Runs	Target failures	Completed client scenarios
9a3efc2cf	Original: 20 sequential clients	10	0	200
9a3efc2cf	Amplified: 100 clients, 4 concurrent	10	0	1,000
9a3efc2cf	Amplified: 500 clients, 16 concurrent	10	0	5,000
91c78457b	Original: 20 sequential clients	10	0	200
91c78457b	Amplified: 1,000 clients, 32 concurrent	10	0	10,000

Worker settings: verified -Xmx128m -XX:+UseSerialGC. Parent/build JVM uses -Xmx768m; an experimental 256 MiB compiler setting failed a fresh compilation, not the test. No timeout was increased, no original iteration count was reduced, and no original test was edited.

Reproduction commands (from the assigned fresh clone):

git fetch origin
git rebase origin/main
JAVA_TOOL_OPTIONS='-XX:+UseSerialGC -Xmx768m -Dfoundation.url.resolver.debug=true' scripts/test.bash --local --test stressTestRegistryRelayAddressAmplified --log /tmp/relay-amplified.xml
JAVA_TOOL_OPTIONS='-XX:+UseSerialGC -Xmx768m -Dfoundation.url.resolver.debug=true' scripts/test.bash --local --test stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale --log /tmp/relay-original.xml

Use --remote when the manual workspace-upload path works. Both attempts here failed at the build-service RPC boundary before returning test results. Do not count failed submissions as test failures or blindly re-run CI.

Final brief audit:

1. Fresh clone, assigned branch, identity, version reservation and first commit: DONE.
2. Honest amplification and measured rate: DONE as an investigation, threshold NOT MET (0%). Exact target assertion cannot be recorded because none failed.
3. Mechanism sentence: NOT ESTABLISHED. Query and registration source already merge configured/registry addresses. Cache and ordering hypotheses remain unproven.
4. Deterministic failing test and production fix: NOT ATTEMPTED under the explicit prerequisite.
5. Fix gate and own PR: NOT MET. PR URL: none. No requested PR comment can be posted without a PR. Diagnostic branch is preserved for continuation, not offered as a fix.

Remaining work: obtain a fresh failing assertion with full output from the original test or a corresponding manual remote targeted run; trace that actual failure, then create a deterministic public-API reproducer before changing production code. Passing samples do not prove the historical report incorrect.

Remote branch: https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake Remote findings: https://github.com/CodexCoder21Organization/UrlResolver/blob/fix/stale-registry-relay-address-flake/investigation/stale-registry-relay-address.md Nothing deployed, published to Maven, merged, enqueued, or closed. Generated build tools/results are excluded from commits. No agents or watchers remain active. The abandoned remote submission records may remain in the build service; their clients have exited and no test result was returned.

2026-09-24 ? folded into the UrlResolver six-test flake-fix campaign

Supervisor adjudication (sweep3, 2026-09-24): this test (stressTestFindPeersUsesRegistryRelayAddressAfterConfiguredAddressGoesStale) is item 3 of the six-test UrlResolver flake-fix campaign tracked by url://handoff/handoffs/hf-2026-09-10-prove-urlresolver-main-is-flake-free-run-one-batch-of-three-parallel-remote-full-suite-runs-at-zero-failures-and-fix-whatever-it-surfaces. It is no longer investigated separately; that handoff is now a dependency of this one.

• The earlier blocker ("no failing assertion available") is resolved: two genuine failures on unmodified main 9553becc in the 2026-09-24 three-parallel full-suite gate ? https://buildtest.kotlin.build/run?id=387fe030 and https://buildtest.kotlin.build/run?id=a4c988cd. Assertion at line 86: "Client 1: relay registration should use the newer registry address. Expected <peer>, actual <null>." The failure is in test setup (relay acknowledgement never observed within 3 s), not in the findPeersForService address merge. Candidate (unverified) mechanism: the first relay-registration pass reads the peer list before the valid registry address is added, and no new pass starts within the window under full-suite load.
• Checkpoint branch with evidence and source pointers: https://github.com/CodexCoder21Organization/UrlResolver/tree/fix/stale-registry-relay-address-flake (head d9dead42a9805a5868f68e3605f94cd4b83fd42c), note at https://github.com/CodexCoder21Organization/UrlResolver/blob/fix/stale-registry-relay-address-flake/investigation/stale-registry-relay-address.md. The branch is 09-16-based; its reserved coordinate 0.0.1237 is now below main ? start a fresh branch from main.
• The campaign worker taking item 3 must read that checkpoint first. When the campaign lands item 3, complete this handoff with that PR as evidence.