Handoff: Identify the first-attempt UrlProtocol gossip-flood timeout

Written 2026-10-23 03:11 UTC.

RE-VERIFY: This is a write-time snapshot. Re-check git ls-remote for both evidence branches, current main, and gh pr view https://github.com/CodexCoder21Organization/UrlProtocol/pull/857 --json state,headRefOid,statusCheckRollup,mergedAt before acting. Native XML and phase journals are the evidence; passing later runs do not classify the old failure.

Design decisions in force

Owner direction dated 2026-10-21: "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

Lane constraints dated 2026-10-23 remain in force:

• "Every first-attempt failure is a real defect."
• "If you cannot reach a reliable reproducer within ~2 hours, stop and report what you tried and observed — do not guess at a fix."
• "Never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test."
• "You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs."
• "Do not add JVM tuning flags."

Build/test runs are local only in this lane. A full suite, once a proven fix exists, requires the machine-wide flock /tmp/claude-501/complete-suite.lock scripts/test.bash --local --test . --log <xml>. The shared cgroup limit is 10 GiB. The explicit lane treats kotlin.build remote as informational and forbids rerunning it; this overrides the general testing document's remote-required policy.

Mission and unresolved question

Find why testRelayedUrlCallsProgressDuringManyPeerGossipFlood timed out on attempt 1 at 60 seconds (native 62,673 ms) and passed attempt 2 in 22,801 ms in the original CI run, head196e1ee5 of the related PR. The original console printed all 16,384 announcement / 1,041 execution counts before the watchdog; its child no longer existed when jstack ran. No causal runtime stack exists.

The test holds a registry peer's lifetime while the registry is full, causing sixteen gossip workers to wait for eviction. It requires relayed calls to complete before releasing that held peer. Across 16 cohorts it sends 16,384 announcements from 256 authenticated identities and checks 1,041 exactly-once relay executions, then closes its resources.

No reliable runtime reproduction or file:line mechanism was established in the bounded investigation. No production fix or PR was created. The failure may be in cleanup, a dependency, or runner completion after the function returns; none is proven. The prescribed stop branch was taken after roughly two hours rather than changing code from a theory.

What was observed and retained

1. Ran 69 original-workload scenarios and 40 amplified scenarios. All 109 passed attempt 1 with zero attempt-history. The four amplifiers each ran 10 times:64 native relay callers (8,209 executions); eight concurrent announcement senders with original counts;64 virtual relay callers (8,209 executions); and concurrent closure of the three relay services after all original assertions. Every 60 s budget and original iteration count was retained. The 64-caller choice follows the provider's 64 active-handler contract, not a host calibration. 2.108 phase journals measure cleanup at 11–315 ms. Longest body was 52,618 ms. These healthy closures do not rule out an intermittent close defect. Code-path inventory includes RelayService.close's receive-worker join and factory Host.stop's admitted-work drain, recorded in findings; they are candidates only.
2. Added WIP-only phase journals and snapshots near 55/59 seconds. Added JDK 21 JSON dumps for virtual threads and a separate externally bounded monitor covering the child JVM after the function returns. Validated the dump command against an owned healthy runner. Corrected the first external monitor to follow forks from JVM worker threads through the process-parent table. The corrected monitor observed 27 test JVMs and attempted zero deadline dumps; no failing stack was captured.
3. One additional command failed BEFORE test entry: artifact fetching from kotlin.directory returned java.net.ConnectException: Connection refused; build preparation failed and native test time was 0. This is separate from the target runtime timeout. A bounded exact-artifact HEAD probe then returned HTTP 200 and local sampling resumed. Full stack, native XML, and challenge record are retained. Server-side reason for the temporary refusal is unknown. The report-challenge CLI was not invoked because it auto-enqueues/merges, forbidden here.
4. Source bases advanced during work: initial b207bde6b; then 3af41e0bf (test ordering); 54ce82ca0 (artifact version); then 8da0a86f1351b93c40d1dc861416009135deb215(response encoding changes including Libp2pHostFactory). Samples after originalAfterFetch sample 2 use 8da0a86f1; exact later main/candidate hashes are in the sample results. The original test is unchanged on those main versions. Preserve these boundaries when comparing outcomes.
5. Shared memory: 227 samples peaked at 10,737,418,240 bytes (the 10 GiB limit). Observed OOM 16 / kill 1 counters were unchanged. No full suite was started or killed here. Both final sample driver and external monitor exited 0 at 03:04 UTC; process audit found zero owned sample/monitor/test JVM background processes.

Relevant PRs and remote work

Related PR context rechecked 2026-10-23 03:11 UTC: source PR isOPEN, head3fb13b7f6f9b55e9ae0dbff3e6bb8e64a1eb8479, merge stateBLOCKED; checksbld-all-tests=IN_PROGRESS, bld-build=SUCCESS. This lane did not change that PR or own its current CI watch.

No deployment, publication, restart, enqueue, merge, or buildtest-run mutation was performed by this lane. Its production-source/build/script/README diff from tested main is empty. The WIP test diagnostics and evidence are intentionally not proposed for merging. Build jars and dependency caches were not pushed.

Next steps

1. Re-verify live repository/PR state and preserve the two branch snapshots. Read findings, native rollup, and sample records. The 109 passes are not evidence that the old timeout was harmless, pre-existing, or caused by its PR.
2. The bounded lane has stopped. A further investigation needs new failing-run evidence or a new, mechanism-focused experiment; repeating the same healthy campaign is not a fix. Keep the 60 s deadline and obtain a live failing stack spanning cleanup AND runner completion. The existing diagnostic code is ready to reuse.
3. Once the specific condition is known, force it through public APIs in a deterministic fail-first test, then fix the owning implementation. Require mutation control, three first-attempt focused runs, one full suite under the lock, a WHY-first PR and required-check watch (90 minute cap), with no merge. None of those fix gates has been claimed complete here.

Operational details and sources read

Fresh checkout was at the brief's scratchpad/workspace/FLOOD; all necessary work is remote. For harness reuse, copy the lane-evidence Python scripts into the scratchpad/out directory: they derive their checkout as its sibling workspace/FLOOD. Use a NEW result label; do not overwrite native results. Tests run via scripts/test.bash --local --test <name> --log <xml>. Native success counts only when attempt 1passed and there is no attempt-history. The external monitor verifies the owning sample-driver identity and has a hard deadline; never target stale numeric PIDs from these records.

Read the project README and relevant relay/gossip/close contracts, prior BUG8e findings, TESTING.md (including extreme-condition and deterministic-reproducer guidance), PHILOSOPHY.md, and the github-repos, fix-flakey-test, status-report, create-handoff and report-challenge skills. Canonical handoff format was read from handoffs/README.md; its legacy merge section is superseded by the current service-based skill and this lane's explicit no-merge rule.