Handoff: Resolve UrlProtocol snapshot-40 approval holds and four intermittent first-attempt failures

Written 2026-10-23T11:46:07.002709+00:00.

RE-VERIFY: This is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/UrlProtocol/pull/846 with gh pr view https://github.com/CodexCoder21Organization/UrlProtocol/pull/846 --json state,isDraft,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt before changing a branch. Read the current findings and verify every remote head below. No merge, enqueue, publication, deployment, service restart or CI rerequest was authorized in this verification lane.

Design decisions in force

Owner direction, 2026-10-21: "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

The 2026-10-23 verification brief says the two G5 tests "await the OWNER's approval and must stay UNCHANGED" and that the optimistic root-count change must remain its own commit. Both conditions were preserved. The PR body retains the explicit do-not-merge hold.

Campaign constraints from that brief: "never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test." Builds/tests use --local; the runner retries automatically, so count a pass only when attempt 1 passed. The one full suite authorized for this lane is complete and was not repeated. Never publish raw JVM recordings, profiles, dumps, environment dumps or system-property dumps; preserve only reviewed derived evidence.

Mission and remaining work

The lane moved the existing PR from jvm-libp2p snapshot-39 to snapshot-40 and completed its requested targeted verification, one locked full suite, and main comparisons. Remaining work is to resolve the owner-held count/setup decisions and establish reliable reproductions for four intermittent first-attempt failures before treating the full suite as clean. This record does not authorize changing the protected assertions or merging the PR.

Required CI completed before the 90-minute observation cap: bld-build passed; bld-all-tests failed on the two protected G5 assertions. The CI artifact contains 3,174 tests: 3,171 first-attempt passes, 1 retry-only pass and 2 failures, with all 26 build rules passing. testRelayedUrlCallsProgressDuringManyPeerGossipFlood hit its 60,000 ms global limit at stream-controller acquisition (source line 134, cohort 13), then passed in 28,829 ms. This same PR head passed it first attempt in the local full suite in 16,224 ms; a targeted fresh-main comparison at https://github.com/CodexCoder21Organization/UrlProtocol/commit/6e709b751610512e6d13efb515303557b4199c1b passed first attempt in 24,097 ms. These passes do not resolve the intermittent CI failure. Its timeout, iteration counts and assertions remain unchanged.

The existing CI workflow also logged an infrastructure failure in the remote side of its local-and-remote race: run 365b8428 could not establish droplet creation status or verify cleanup after a relay-forward failure. The local side produced the test artifact above. Separately, informational Kotlin suite 108203823662 remained queued with zero dispatched runs and no log. No run was rerequested, deleted or otherwise changed. Watch-only observation ended at 11:36:19 UTC after 89 minutes 26 seconds; no further CI watch is running.

What was found and done

1. Fresh clone and rebase started from PR head https://github.com/CodexCoder21Organization/UrlProtocol/commit/52a5e05b354399c77ba01bd387b5a7b1e98dbdfc. The new commit https://github.com/CodexCoder21Organization/UrlProtocol/commit/935a14d890f41f91dbfc6d25ff3fb2c6aa018408 changes 2,548 files only by replacing snapshot-39 references with snapshot-40. Every direct organization libp2p pin is aligned. The independent diff review proved exact byte replacement, including both protected G5 scripts; no assertion, timeout, iteration count, fixture logic or implementation changed in this lane.
2. Input artifact community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-40 matches the supplied JAR SHA-256 574ebcb307f4da6383a2085cda36e90b2f358554075f468a3e92989bfd881c7b and POM SHA-256 0b0da2a27aa1a072f39c0fcc8a3f638598ca1d8839642c293a961a0e55dc834f. The actual Coursier-resolved JAR has the same hash. The brief names source https://github.com/CodexCoder21Organization/jvm-libp2p/commit/0c3c86ad9ceb1c9d3834da68f239108e14b7f780; GitHub confirms https://github.com/CodexCoder21Organization/jvm-libp2p/pull/70 merged at that commit. An explicit read on a paused child now delivers queued data or stays armed for arrival. This lane published nothing.
3. At the exact new head, three invocations of the carried 30-case selection each produced 28 first-attempt passes, zero retry-only passes, and the same two G5 failures. testGossipServiceQueryResponseFlushesUnderBackpressure, which failed in the prior snapshot-39 runs, passed on attempt 1 in all three snapshot-40 runs and in the full suite.
4. One full suite queued at 08:20:04 UTC, acquired the shared lock at 08:31:22, and completed at 09:39:27: 3,174 tests, 3,169 first-attempt passes, three retry-only passes and two failures. All 26 build rules passed. Exit 1 reflected test failures; no running suite was killed. Shared memory was 8,122,675,200 bytes before execution, sampled at a peak of 10,737,364,992, and 8,171,368,448 afterward. Shared oom=16 and oom_kill=1 stayed unchanged. Preparation and real test execution were both verified through derived process/thread observations; the lack of intermediate XML was never counted as a verdict.
5. Every full-suite first-attempt failure was compared with fresh main https://github.com/CodexCoder21Organization/UrlProtocol/commit/2e9acd78bd951a1ebdc2d9cf61f5f132f9ec0c56. All five actual main executions passed on attempt 1. One earlier timer comparison did not execute because build dependencies could not be downloaded; its native time-zero placeholder is explicitly classified NOT_RUN_INFRA, not a runtime verdict. An isolated timer run passed after an exact dependency download succeeded.
6. Source was pushed to the existing OPEN PR with an explicit lease after another unchanged fetch/rebase. The title/body were patched through the structured GitHub API and read back exactly. The body discloses all retry-only results and the unresolved mechanisms. A generated 30 MB compatibility-resource JAR under build/ was removed before the push; source trees are clean.

Holds and diagnostic gaps

The optimistic registered-root count change remains separate at https://github.com/CodexCoder21Organization/UrlProtocol/commit/31acce21e6608486537383cb0dd4f35cdb8d0771, an unchanged replay of https://github.com/CodexCoder21Organization/UrlProtocol/commit/5dd51e98704980eab92834e1e73a4f69711aea12. It expects two physical roots, not two callback deliveries. Owner approval is still pending.

The two protected tests are admission identity and sibling-root registration. Both expect absence from the network table inside a gated application callback; the newer library registers before calling that callback. Both fail all three automatic attempts in each snapshot-40 invocation and pass on main's snapshot-25. Their assertions remain unchanged. BUMP39's review and unapplied registration patch preserve the earlier ordering analysis.

The three intermittent local full-suite failures and one additional CI first-attempt failure remain real observations, not passing results or fixed defects:

The CI-only case relayed calls during gossip flood hit the 60,000 ms global test deadline while awaiting a stream controller in cohort 13. Attempt 2 passed in 28,829 ms; the same exact PR head passed this case in the local full suite in 16,224 ms. Timeout snapshots showed the cohort-13 capacity victim intentionally held, the test caller in a timed future wait, and the four libp2p network loops in selector waits. These samples establish location and phase, not a mechanism. The existing 60-second limit, 256 identities, 16,384 announcements and all assertions remain unchanged. A targeted fresh-main run at https://github.com/CodexCoder21Organization/UrlProtocol/commit/6e709b751610512e6d13efb515303557b4199c1b passed on attempt 1 in 24,097 ms after 289 seconds of compilation and execution. This single pass does not refute intermittency.

Isolated passes do not refute intermittency. No reliable reproduction of these four mechanisms, guessed implementation fix, weaker assertion, or changed timeout was produced. Complete first-failure stacks and derived case summaries are remote below.

Relevant PRs and recovery refs

Earlier evidence remains on BUMP39 source and BUMP39 evidence. They were inputs to this lane, not additional checkouts modified here.

No code was deployed, restarted or published by this lane. Snapshot-40 was an already-published input. No merge or enqueue was performed. Raw native XML/console, thread samples and the remote process snapshot stayed local; reviewed derived summaries, failure stacks and exact runner commands are preserved remotely. Their omission is deliberate because the lane forbids pushing recordings/dumps that may carry credentials.

Next steps

1. Re-verify live PR/CI state and current dependency publication before acting. Read BUMP40 findings, the per-case main matrix and the review. Preserve the optimistic count commit separately and both protected assertions until the owner decides.
2. Establish deterministic reproductions for the four intermittent first failures using the public APIs and real dependencies. Instrument the timer's frame-processing boundary and the proxy/accepted-socket identities before assuming a cause. Do not replace observed contradictions with longer waits or weaker checks. Fix a dependency defect at its source if evidence points there.
3. Keep the PR draft and its full-suite limits explicit until the holds are resolved. Obtain the owner's decision for the existing count/setup questions. This record grants no merge, production or rerequest authority.

Commands and operational details

Recover the exact 30-case list and runner from lane-evidence/BUMP40-selection.json and BUMP40-run.py; adapt the absolute scratchpad path for a new checkout. Native commands were scripts/test.bash --local --test <name> --log <xml>. The complete execution used the machine-wide /tmp/claude-501/complete-suite.lock; its two-hour acquisition cap was separate from execution, and no executing suite was killed. The runner retried failures automatically; the summaries count only attempt 1 as a pass.

The baseline download interruption lasted about 857 seconds across three native fetch attempts. Six owned connections to 198.199.106.165:443 and TLS-handshake waits were observed, and independent curl initially reported SSL connection timeout; a later exact POM request succeeded. Root cause was not established and no service was changed. HardwareControlFabric was queried read-only using existing client_cert.pem, fabric_key.pkcs8 and server_cert.pem; raw command/property-bearing output was not published. The Handoff CLI accepts list --json, with the command before the option. Its old README's git-merge exception is superseded by the current central-service skill and this lane's no-merge rule.

Main advanced after the five original full-suite comparisons and the PR push to https://github.com/CodexCoder21Organization/UrlProtocol/commit/6e709b751610512e6d13efb515303557b4199c1b. The exact tested PR remains 935a14d; no second full suite was run. A future code edit requires a fresh rebase and verification, rather than carrying the old full-suite verdict onto a changed head.