Classify the remaining flood-test timeout after withdrawal validation verification

Written: 2026-10-22 22:53 UTC

RE-VERIFY: This is a write-time snapshot. Read the current PR and required checks, verify branch heads with git ls-remote, and inspect owned processes before acting. Do not reuse any remembered PID or assume the disposable local disk remains.

Design decisions in force

Owner (2026-10-21), verbatim: "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

BUG8d brief (2026-10-22), verbatim: "The check must accept every peer-id form the product itself issues (derive the rule from the actual id types/providers, not from base58 alone)." / "Refusal on the relay path must skip only the bad frame, exactly like the gossip path, with the same refusal report." These supersede the earlier base58-only implementation described in the previous version of this handoff.

For the large-answer failure: "if caused by this PR, fix; if pre-existing, report with evidence (do not fix here)." Main attempt-history proved it pre-existing; no response-chunk implementation change is included.

Local builds/tests only: scripts/build.bash --local and scripts/test.bash --local --test <name> --log <xml>. Never --remote. Full suites only under /tmp/claude-501/complete-suite.lock, recording /sys/fs/cgroup/memory.current before and during. Count only attempt-1 passes without attempt-history. Never increase a test timeout, lower iterations, weaken assertions, disable/skip/delete tests, use mocks/reflection, or expose private members for tests. Do not add JVM tuning flags.

No merge, queue, artifact publication, deployment, restart, or buildtest deletion/rerequest is authorized. Check PR OPEN before pushing its branch. Update the PR body through gh api --method PATCH ... --input body.json and read it back. Poll GitHub at most every two minutes. Required checks in this repository are bld-build and bld-all-tests; remote check is informational, read a red output once and classify it, never rerun it. Watch required checks for at most 90 minutes. Commit trailer: Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; final PR footer: 🤖 Generated with [Claude Code](https://claude.com/claude-code).

Mission and remaining work

Finish classification of the one native CI first-attempt timeout. Do not declare every failure classified or complete this handoff until that item is established. The requested candidate verification, single full local suite, PR push/body update, and bounded required-check observation are complete. No reliably reproduced PR-caused defect is known from BUG8e; no speculative source fix was made.

Verified source and PR state

PR https://github.com/CodexCoder21Organization/UrlProtocol/pull/857 is OPEN and unmerged at 196e1ee5307a6b33fa04f828718bf3bd072cd136. Latest API mergeable value is True, mergeable_state isclean. The candidate was rebased on then-current main 8ee64cd68b1d58065e6edee995026a310bbc22ac before verification and push. Main advanced afterward to 3faf39426bdb27395332fff57f66c5d01d904db7 (RPC reader ownership change); no untested rebase was pushed after the measured results.

The two BUG8d review fixes are already on the candidate: built-in in-memory IDs remain supported, and invalid relay withdrawals do not end a registered listener or drop the next frame. Canonical libp2p/in-memory IDs preserve exact text. Both relay directions refuse schema errors for one frame, retain authenticated refusal reporting and correlated malformed-request errors, and propagate unrelated failures. Prior fail-first and mutation controls plus independent coverage/code reviews are preserved on the BUG8d evidence branch. Treat these as verify-don't-redo evidence.

Measured verification

• Fresh focused run: 32/32 changed cases passed on attempt 1, zero histories.
• Full suite executed exactly once under direct flock: 3143/3143 passed on attempt 1, zero histories, exit0 at 20:56:44. Lock released as soon as the suite exited. All32 changed cases explicitly present. Large-answer15380ms; flood12318ms.
• Shared memory: before command9,776,771,072 bytes, first observed running9,406,656,512, maximum sampled10,735,190,016; OOM-kill counter remained 1. No suite kill or JVM tuning. Memory JSONL and exact summary retained.
• A nonterminal terminal-memory-record warning was emitted; complete trace and challenge record are preserved. It changed no native test verdict. Do not invoke report-challenge's auto-merge while the user's no-merge rule remains in force.
• Candidate pushed to feature branch with explicit force-with-lease after OPEN check. PR body PATCHed through gh api and read back exactly. WHY and required footer retained.
• Required build https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39853327541 and all-tests https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39853327962 are SUCCESS at exact 196e1ee5. Watchman0.0.21 ran20:58:43–22:19:43 and exitedGREEN0 within90minutes. No queue/merge/rerun/rerequest.
• Native CI has3142/3143 attempt 1 passes, not3143. Large-answer passed attempt 1 in 114781ms. The sole retry is the flood test described below.
• Informational old-head remote failure was read once: infrastructure recovery spent13,630,342ms in RECOVERING_LOCAL_STATE with no upload session/chunk. New-head check data contains no informational red verdict; the old queued zero-run suite was report-only. Do not rerun or delete anything.

Failure classification: large-answer established; flood remains unknown

The large-answer failure is pre-existing on main, proven by native historical-main CI https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39766778628 at fbce5c1ab11da7c75a66cfbc39a3b6166ad69b34. Attempts1/2 failed with incomplete chunks/shared fetch-budget expiration; only attempt 3 passed. Exact test SHA256 59dd38152f45ea014fea86e046d5edb624c12891402346b5adff7d7a91adbd16 matches historical main, tested main 8ee64cd, and candidate196e1ee5. No response-chunk change is included here.

Remaining failure: foundation.url.protocol.testRelayedUrlCallsProgressDuringManyPeerGossipFlood, native CI attempt 1 false62673ms, watchdog60000ms; attempt 2 true22801ms. The full734232-character first console and complete stack are remote. It printed all final assertions/counts: 256 identities,16384 announcements,1024 concurrent relay calls plus16 recovery calls and1 warm call,1041 executions. This summary prints before finally cleanup. Captured jstack found PID126199 already gone, so there is no live frame establishing the exact remaining wait. Do not call this a proven cleanup deadlock.

Distinct connection-open timestamps in the failure span55649ms, largest adjacent gap2139ms; the passing retry spans18969ms. Most of the bound elapsed during workload progress. Across 1610 tests with local durations>=1s, native CI/local median ratio 1.639 and p90 ratio 2.867;18 passing tests exceeded the flood ratio 5.088. These are timing observations, not a runtime mechanism or proof of a main failure.

The unchanged test sends real peer announcements through the RPC dispatcher and valid relay messages. The withdrawal-validation arms are not explicitly exercised, but valid relay messages do enter the new shared parse helper. Static path inspection is supporting evidence and has not been treated as conclusive proof of unrelatedness.

Unchanged main 8ee64cd: 10 fresh local targeted samples all passed on attempt 1, no histories. Its completed CI https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39799885682 passes the same test attempt 1 in24319ms. Historical lookup found four earlier native passes (32499/38348/37401/33459ms) and two cancelled runs without artifacts. An exact-target scan of 28 available cached native XML files finds only this CI failure and a deliberate inline-dispatch mutation control; the mutation is not a main baseline. Clean records are not proof of absence and do not establish the timeout as pre-existing.

The full CI job log adds no live stack. No timeout, iteration, assertion, worker allocation, dependency or implementation was changed to make the failure disappear. Classification remains unresolved. A binary label would exceed the evidence available at handoff time.

Remote branches and evidence

No code was deployed or published by BUG8e. No merge or queue operation was performed. No new source commits were made by BUG8e; the feature branch contains the prior candidate exactly. Source clones, main control, docs and historical-evidence checkout are clean; no stashes. Evidence pushes are verified by ls-remote; orphan evidence ancestry may appear local-only without a tracking ref even though the exact head is remote.

Resume without redoing completed work

1. Claim this existing handoff; verify the current PR/source/required-check state and read lane-evidence/BUG8e.md, native CI XML, first console/stack, timing comparison and historical report. Do not repeat the successful full suite just to recover evidence.
2. Establish the exact condition behind the remaining 60-second watchdog using a reliable public-API reproducer and a live pre-timeout diagnostic capture. The original CI child is gone; its dump cannot be recovered from the existing console. Ten clean main controls and four earlier passes are non-matches. Do not guess a fix or manufacture scheduler pressure with artificial load.
3. If a main reproduction or defensible path exclusion establishes the retry pre-existing, report that with the positive evidence. If PR-caused, write a reliably failing regression before fixing implementation, retain all constraints, and verify the fix independently. Fetch/rebase current main before any new build/push; enumerate the new main RPC reader contract implementations if affected.
4. Update the PR via PATCH/readback with exact native first-attempt counts. Do not merge. Complete this handoff only when the classification and any required PR-caused fix are actually complete.

Local operational notes

Scratchpad $S is /tmp/claude-501/-code/2abf611d-5e04-4919-84a8-d7b72869d40c/scratchpad. BUG8e files under out/ are copied into the evidence branch. Fresh clones: workspace/BUG8e, BUG8e-main, BUG8e-evidence, BUG8e-docs, BUG8e-history. These paths are convenience only; all needed evidence is remote.

All tracked commands and the historical lookup finished. Owned process groups 41709/82114/12211/25059 have zero members in ps; no background watcher/test remains. Do not kill a later reused PID. No build/test artifacts are committed to the source branch. GitHub job-log read at 22:46:58 occurred 33 seconds before the shared two-minute cadence; that process mistake is recorded, and subsequent dependent requests used an explicit UTC gate. Required immediate PATCH/readback is a mutation verification, not CI polling.