UrlProtocol PeerRegistry: recordDialSuccess waits on another lifetime's pending removal from inside a lifetime action

Written 2026-10-21 04:03:06 UTC.

RE-VERIFY: This is a write-time snapshot. Before changing code, recheck main and the proof branch with git ls-remote https://github.com/CodexCoder21Organization/UrlProtocol.git refs/heads/main refs/heads/wip/r738opus-proofs-20261020, inspect src/foundation/url/protocol/PeerRegistry.kt, and recheck the reviewed PR with gh pr view https://github.com/CodexCoder21Organization/UrlProtocol/pull/738 --json state,mergeStateStatus,statusCheckRollup,mergedAt. The tests below are saved historical fail-first evidence; this filing did not run them again.

Design decisions in force

2026-10-21 request: "reject the cross-lifetime wait the way the other paths do, or restructure so the wait cannot need the current thread".

2026-10-21 request: "fail-first proof already exists; fix in the product; keep the proof as the regression test; comprehensiveness and adversarial reviews; PR".

A fix is still required. This handoff was filed without claiming it. The filing task authorizes no code changes, PR creation, merges, publication, deployment, or service restarts.

Mission summary

Fix the pre-existing cross-lifetime dial-success wait in UrlProtocol's PeerRegistry. An independent review of https://github.com/CodexCoder21Organization/UrlProtocol/pull/738 found and reproduced it before that PR merged. It was deliberately excluded from that PR's scope. GitHub now reports that PR MERGED at 2026-10-21T03:04:36Z, merge commit https://github.com/CodexCoder21Organization/UrlProtocol/commit/ba4d6185f4ec6ca34f06186fbc6eda9bbc0ae7aa; that is also the main head verified during filing.

This is a public-API wait-cycle defect. The historical review inferred that it was not on the production relay's then-current path because product callers did not use runIfActive. That inference is not proof of present caller usage or the cause of the broader campaign's production symptoms. The campaign context is https://github.com/CodexCoder21Organization/PlanRepository/blob/main/handoffs/2026-10-19-containernursery-peerregistry-stall-tinysite-publish-timeouts.md ; its operational and PR states are historical and were not rechecked for this focused filing.

What was found and done

1. Read the complete review record at /tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad/out/r738opus-findings.md. A complete remote copy is https://github.com/CodexCoder21Organization/UrlProtocol/blob/37e558e33c42b9c793899f9f1afd048f85621438/lane-r738opus-records/r738opus-findings.md . The remote copy was also read in full during filing.
2. Verified the remote proof branch and fetched the exact test source through GitHub. The suggested wip/R738final-proofs branch contains later, different review proofs, not this test. The actual branch is wip/r738opus-proofs-20261020, remote head 37e558e33c42b9c793899f9f1afd048f85621438.
3. Parsed the saved XML on that remote branch. On reviewed candidate 07a478bbee3688affd08392d5e1b2fd5eaa9b1a4, the proof failed attempts 1, 2, and 3 in 869/784/721 ms. On earlier main 481ec3ba1, it failed attempts 1, 2, and 3 in 871/908/922 ms. These are failed automatic attempts, not a successful run. This establishes that the defect predates the reviewed PR.
4. Inspected current main's recordDialSuccess through GitHub. It still joins pending.membershipFinished after the same-lifetime special case and does not reject a current action for a different lifetime before joining. Source snapshot: https://github.com/CodexCoder21Organization/UrlProtocol/blob/ba4d6185f4ec6ca34f06186fbc6eda9bbc0ae7aa/src/foundation/url/protocol/PeerRegistry.kt . No new test execution was performed against this main head.

Mechanism

In src/foundation/url/protocol/PeerRegistry.kt, PeerRegistry.recordDialSuccess(peerId) finds an in-flight threshold removal in pendingDialRemovals. It avoids the wait if the pending lifetime itself is the action on the current thread, but otherwise calls pending.membershipFinished.join() after releasing the registry locks. Releasing those locks does not release the different lifetime monitor held by PeerLifetime.runIfActive around the caller's action.

The threshold-removal path recordDialFailure -> removePeerMembership -> PeerLifetime.beginRemoval needs the target lifetime's lifecycle monitor. With action A calling success for peer B, action B calling success for peer A, and both threshold removals pending, each action joins completion that needs the other action to return. Other cross-lifetime acquisition paths reject nested action waits using rejectNestedLifetimeAcquisition or the refresh-from-action fast path. The dial-success path omits that check.

Historical stack locations on reviewed candidate: recordDialSuccess at line 1472; threshold path recordDialFailure at 1430 -> removePeerMembership at 2295 -> PeerLifetime.beginRemoval at 505. Earlier main's corresponding locations were 1353, 1312, and 410. Treat those line numbers as historical; function names and the pinned source links are authoritative.

Reproduction and exact failing assertion

Test: testR738opusCrossLifetimeDialSuccessDoesNotWaitOnOtherAction.

Exact path: tests/testR738opusCrossLifetimeDialSuccessDoesNotWaitOnOtherAction.kts.

Branch: https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r738opus-proofs-20261020 .

Pinned source: https://github.com/CodexCoder21Organization/UrlProtocol/blob/37e558e33c42b9c793899f9f1afd048f85621438/tests/testR738opusCrossLifetimeDialSuccessDoesNotWaitOnOtherAction.kts .

Exact assertion text (followed by a full thread dump in the test):

A lifetime action must not join another peer's dial removal while that removal needs the other active action

The test uses only public registry/lifetime APIs: PeerRegistry.inMemory, addPeer, recordDialFailure, recordDialSuccess, getPeerLifetime, and runIfActive. It creates two peers, records one fewer than the dial-failure threshold for each, and holds both lifetime actions. Two threshold threads then start the final failures. ThreadMXBean checks that each threshold thread is BLOCKED in beginRemoval, with its corresponding action thread owning the monitor, before releasing the two cross-peer success calls. It detects both action threads WAITING in CompletableFuture.join inside recordDialSuccess while both threshold threads remain BLOCKED. The test asserts that this cycle does not occur, all four operations complete or reject, and any captured failure is an IllegalStateException.

Observed thread states in the saved review evidence:

• r738opus-action-0 and r738opus-action-1: WAITING in CompletableFuture.join at PeerRegistry.recordDialSuccess.
• r738opus-threshold-0 and r738opus-threshold-1: BLOCKED in PeerLifetime.beginRemoval, called by removePeerMembership and recordDialFailure, each waiting for its corresponding active action's lifecycle monitor. Each cross-peer success waits for the opposite removal.

The test stages the exact wait ordering with latches and ThreadMXBean states rather than machine-dependent load. The blocked monitor entry and join cannot be interrupted. Workers are daemon threads owned by the native runner's isolated JVM; run the test through that runner rather than embedding it in a long-lived process.

Saved complete native XML, including assertion stacks and attempt histories:

• Candidate: https://github.com/CodexCoder21Organization/UrlProtocol/blob/37e558e33c42b9c793899f9f1afd048f85621438/lane-r738opus-records/r738opus-run1.xml .
• Earlier main: https://github.com/CodexCoder21Organization/UrlProtocol/blob/37e558e33c42b9c793899f9f1afd048f85621438/lane-r738opus-records/r738opus-main-481ec3ba1.xml .

After a fresh checkout, README/instruction review, and fetch/rebase on latest main, preserve the proof file on a dedicated fix branch and run it through the local native runner:

scripts/test.bash --local --test testR738opusCrossLifetimeDialSuccessDoesNotWaitOnOtherAction --log /tmp/cross-lifetime-dial-success.xml

Read XML attempt histories; a final success after an earlier failed attempt is not an attempt-1 pass.

Relevant PRs / refs

No repository was cloned or edited for this filing, and no new source artifacts need pushing. All existing proof material needed for pickup is already on the verified remote branch above. The focused findings from this filing are included in this service body. This filing performed no deploy, publication, restart, route update, or other product change; no deployed-but-unmerged state was created by it.

Correct behavior and next steps

1. Re-verify current state first. Read the repository README and any AGENTS.md, the proof and saved XML, and current recordDialSuccess and neighboring lifetime-wait paths. Fetch/rebase on current main before any build. Claim the handoff only when starting the fix; it is intentionally unclaimed at filing.
2. Before writing or changing tests, read https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md (including "Never Calibrate a Reproducer to a Machine" and one-scenario-per-file layout) and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md (including "A Refuted Hypothesis Is a Successful Outcome"). Write the lifecycle/ownership invariant table before product changes: lifetime action ownership, pending-removal completion ownership, terminal states, re-entry, and cross-lifetime waiting. Map relevant rows to public-API tests.
3. Retain the existing fail-first proof as the regression test. Confirm it still fails against the current unmodified product with the exact cycle before fixing it. If current main already differs, report evidence rather than assuming the historical result is current.
4. Fix the product in UrlProtocol. Reject the cross-lifetime wait consistently with the other nested lifetime paths, or restructure the operation so its wait cannot need the current thread. Preserve same-lifetime success, ordinary external success during pending removal, quarantine/outcome identity, and the intended successful-observation semantics. Do not hide the cycle with a timeout or retry, weaken the assertion, raise a test timeout, reduce iterations, skip tests, use mocks/reflection, or expose private members for tests.
5. Run the unchanged proof against the fix and relevant neighboring public behaviors. Keep failure stacks and first-attempt XML. Perform a dedicated test-comprehensiveness review and a separate adversarial code review; address their findings. Cover both the cross-action cycle and the neighboring lifetime/outcome cases identified in the invariant table.
6. Fetch/rebase on latest main, run the required unit suite, commit/push the fix, and open a PR whose description explains this observed pre-existing defect and links the historical review and proof. Verify CI becomes green. Do not merge or enqueue without the user's explicit approval, and do not publish or deploy without a separate request.

Sources read during filing

• Complete local and remote r738opus-findings.md review record linked above.
• Exact remote proof test and both native XML files linked above.
• Current main's recordDialSuccess function and live PR state.
• https://github.com/CodexCoder21Organization/PlanRepository/blob/main/handoffs/README.md . Its legacy handoff-PR merge paragraph is superseded here by the create-handoff skill and this filing's explicit service-only/no-merge instructions.
• The campaign handoff linked in Mission summary, read as historical context.