<!-- s141 snapshot start -->
S141 — review the green large-answer fix without merging — 2026-10-23 06:43 UTC
RE-VERIFY: Write-time snapshot. https://github.com/CodexCoder21Organization/UrlProtocol/pull/890 is OPEN and unmerged at 36f679818a7fa0a7d0dbfebd13a2c4646178d1bd. Both own required checks bld-build and bld-all-tests are completed/success on that exact head; full suite at https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39905691126. Native artifact audited: 3173/3173 FIRST passes, 3173 attempts, zero retries/histories/failed first attempts; 26/26 build rules successful. Recheck live PR metadata before acting. No merge, enqueue, publication or deployment is authorized. This shared handoff still includes the other lane's gossip failure and the prior later-refusal signature; do not complete it or its parent from this narrow task.
Design decisions in force
Original acceptance: “a relayed large answer must wait for outbox space, not fail its chunk budget.” “No timeout raised, no load or iteration count reduced, no assertion weakened.” Reproducer must fail first, original victim must pass twenty first attempts on the runner, and an uncommitted product mutation must be caught. Evidence workflows use disposable event commits, never dispatch/cancel against a real PR head. Keep the small per-connection buffers, non-pausing timeouts and per-peer caps.
Mechanism and evidence
RelayService.startKeepalive used the throwing send; a 200ms local write-admission decline counted as remote failure. Two such declines clear the live registration and stop its listener while the remote relay still forwards fetches onto that open unread stream. Ten missing replies hold the shared fetch budget until the original deadlines. Deep capture https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39902829438 shows the actual declined 95-byte ping, failure 2/2, lost provider listener and ten forwards; parse memory is fully free. This refutes setup eating the deadline, ordinary throughput and a proposed 16-MiB Yamux-window explanation. Fresh upstream source inspection confirms its initial window is 256 KiB; no upstream artifact or source fix is needed.
The product fix uses the existing boolean admission overload with the same 200ms limit. False leaves the remote failure streak unchanged and exits through the existing pending-ping finally cleanup; admitted-pong timeout, two-failure rule and actual-close observer remain. No existing test changes in the real PR. The new scenario holds a real provider's public Netty root unwritable, observes actual transport declines and requires registration plus the pending one-MiB answer and a fresh exact answer, with exactly one handler execution each.
- Exact final scenario fails 10/10 invocations and 30/30 native attempts after exactly two declines on unchanged product: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39904712420. Evidence draft https://github.com/CodexCoder21Organization/UrlProtocol/pull/889 is CLOSED without merge. Test bytes are identical to the fixed candidate.
- Fixed scenario passes 10/10 first attempts (8172–8321ms), no histories. Original byte-unchanged testRelayedLargeAnswersWaitForOutboxSpace passes 20/20 first attempts (60783–63287ms), all exit 0, no histories. Temporary restoration of only the old product decision in the runner workspace makes the same new scenario fail 3/3 after two declines. All jobs and native artifacts: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39905344992. Mutation patch is evidence only, never a committed product change.
- Own full suite passed 3173/3173 FIRST attempts, no retries or failed first attempts to attribute. Native log confirms the GitHub runner LOCAL result won after the remote side timed out in PENDING; that service timeout is not a test failure. New scenario, original victim, healthy-pong and closed-transport controls all pass first. Required native watcher exits 0. No more test batches are authorized/needed for this lane. Required build already passed: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39905691196. Candidate tree b2ce4421c8be1c500ef23d353c6ee63da1280566 equals PR merge-test tree (merge ref 23048377479a6740828b8f232411d8084a15b1c6; base 8da0a86f1351b93c40d1dc861416009135deb215).
Recoverable refs
| Remote branch |
State |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/fix/s141-keepalive-admission |
real fix, 36f679818a7fa0a7d0dbfebd13a2c4646178d1bd, open unmerged PR above |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s141-records/investigation/s141 |
full findings, every stack/timing/console, native XML, strict audits and source proof; final records c4d196f4c5ba36323b6607f64bedbe51e7d45f22 |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s141-failfirst |
exact baseline 3dbc568e1807dcd3054bcf68551d868b95b2f9fb, closed draft |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s141-verification |
disposable workflow event d284425653fc65adcce8db82f22478e07d9077ac checking out frozen candidate |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s141-diagnostics |
unchanged-victim observation branch, 4aaa341d6; 9/10 first passes and one genuine failure, not fixed evidence |
| https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s141-deep |
deep observation branch, 47ee59ed5; diagnostic observer and post-success marker only, not product/test changes proposed for merge |
All local commits are reachable remotely; no stash or uncommitted product changes. No local build/test ran: runner verification overrides the shared host's local-first rules. Independent test and code review passes are recorded. No deploy/publish/server mutation command was issued; no new code is live outside these unmerged branches and runner workspaces. Known harness corrections: install git before actions/checkout in the minimal container or it downloads an archive with no .git; exact selector works; success XML omits consoles, so diagnostic capture used a clearly identified post-success marker. The marker is absent from the real fix. Historical corpus has 49 of 50 main archives, 46 victim results and seven failing attempts across five runs; one absent archive is not a pass.
Main advanced after PR creation only in build.kts publication coordinate/comments to https://github.com/CodexCoder21Organization/UrlProtocol/commit/2e9acd78bd951a1ebdc2d9cf61f5f132f9ec0c56. Candidate was rebased on then-current main immediately before PR creation; source proof above names the original merge ref explicitly.
Monitor friction: build-watchman main thread blocked reading an informational App build log body (HttpResponseInputStream.read -> readBuildLogContentDigest -> assessForwardProgress), with no new output for over nine minutes. Full thread dump is in records. Stopped only that own monitor and used bounded native gh run watch at 60-second cadence, which exited 0 on own required suite. Never cancelled a PR-head run, re-requested an App check or reran a suite. Report-challenge auto-merges its record, forbidden by this brief, so this issue is retained here instead.
Next: independent review of the green unmerged PR. This lane's requested green-PR stop condition is achieved; no merge, queue entry, publication or deployment. The broader gossip/later-refusal questions and prior deadline-fix review remain governed by the other sections/owner, not claimed resolved here. Preserve every prior section below and the other lane's work. Final s141 RUNNING report and release will confirm local cleanup; never send DONE for this shared child or parent.
<!-- s141 snapshot end -->
S124 completed narrow deadline-parking task — 2026-10-22 21:58 UTC
RE-VERIFY: This is a write-time snapshot. The initial deadline-test parking assumption is corrected and verified; https://github.com/CodexCoder21Organization/UrlProtocol/pull/869 is ready for review, OPEN and unmerged at 745d842911759c2bd91195efb0db5da393275d83. Both required checks are green. This broader handoff remains open for the gossip failure, large-answer failure and separate later-refusal assertion. No merge, enqueue, publication or deployment.
Design decisions in force
The p124 question was “what holds those 16 handoffs pending (which writer/flush/permit/event-loop task, file:line), and is that a product defect ... or a test that assumes writes complete within a window the contract does not promise?” A test correction must retain all assertions, counts and limits, and still fail when the actual resend gets a fresh deadline. At least twenty first-attempt-clean runner repetitions and required bld-build/bld-all-tests are the gates. Owner model: “small per-connection buffer, backpressure, timeouts that do not pause, per-peer caps.” Product behavior and limits are unchanged; only fixture setup owns frozen time. Prior requirements below remain.
Mechanism, classification and evidence
The sixteen permits belong to a rolling set of accepted off-loop Netty WriteTasks queued on the root event loop before child MuxChannel.doWrite. Libp2pHostFactory.kt main :3998–4000 installs the child-future permit-release listener; AbstractChannelHandlerContext.WriteTask.run :1126 (dispatch :819) runs write+flush. Captures show ordinary multistream negotiation, Mux/Yamux writes, parsing/controller initialization and teardown; delayed writes reach the child after 200–300ms and complete successfully. Some pending records are only 0–6ms old at refusal, so the same sixteen writes need not remain for the full interval. No missing release, stuck parent future or blocked lock was established. Bounded admission permits an unsent caller to fail; the original fixture wrongly requires every setup call to succeed within its 200ms wall-clock admission interval. README :857/:2836 expressly allows that refusal. This is a fixture assumption, not an established transport defect.
Capture: 100 original-workload copies/300 attempts, nineteen genuine all-63 failures and 281 post-success console markers. All 232 delayed completion records on refused callers' roots succeed; 95 take at least200 ms and 92 enter the child at least 200 ms after offer. Capture with the added outbound observer removed still has thirteen parking failures and successful delayed completion records; it also records the known separate later-refusal assertion once.
Deterministic original-fixture control: ten controls/30 attempts, all fail the original all-63 assertion with exactly 62 active handlers and one typed unsent caller. Sixteen real ping frames queue behind a held real loop; the actual admission sleep advances ambient time by 200 ms and releases the loop. No CPU load or machine-calibrated concurrency.
Correction and gates complete
The only existing-test edits are the independent parkingClock declaration/comment and services[2] clock wiring in tests/testDirectRegisteredChunkFetchResendUsesOriginalCallDeadline.kts. Real yielding in sleep permits transport progress while setup time stays controlled. All 63 setup callers still launch together and all 63 handlers must park under the same ten-second entry bound. Every later assertion, payload, count and timeout is unchanged; services[0]'s actual original 30000 ms deadline clock stays independent. Product and dependencies are unchanged. Dedicated test-comprehensiveness and code review passes are recorded in findings; no README update is needed.
Exact-head verification: twenty original and twenty exact forced-condition copies each pass first attempt, 40 total/no histories. The forcing condition still advances ambient time 200 ms behind sixteen queued frames and checks all 16 filler pongs. A workspace-only product mutation renewing accepted chunk resend reply deadlines compiles and fails the unchanged original-deadline ended assertion 3/3; it is absent from the correction. Standard PR merge tree ca892a4ab33cd24d6c3748ef3fd12a0b34f160ee equals correction head's tree; main remains 8ee64cd68b1d58065e6edee995026a310bbc22ac.
Required bld-build and bld-all-tests are SUCCESS. Native full suite: 3134 tests, 3132 first passes, 3134 final passes, 3136 attempts; all 26 build rules succeed. Corrected deadline passes first attempt 7399 ms, no history. Both failed first attempts are attributed explicitly:
- testRelayedLargeAnswersWaitForOutboxSpace fails attempt 1 at test line 262, 30/32 batch 1 replies report chunk-fetch/budget failures; attempt 2 passes. Existing main run, fbce5c1ab11da7c75a66cfbc39a3b6166ad69b34, records the same line 262 relayed-batch failure on attempt 2 and pressure chunk/budget failure on attempt 1 before passing attempt 3. The test is byte-identical. Its cause remains unresolved; no correction claimed. Full native historical archive is retained, including its other failures, and is never counted as correction verification.
- testRelayedUrlCallsProgressDuringManyPeerGossipFlood hits its existing 60000 ms whole-test bound at controller open line 134 (62757ms native attempt), then passes 56268 ms. Same prior signature/line 134 in https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39811280704. Per p124 this lane only records the new observation and does not investigate or modify gossip.
No CI run was repeated. Informational kotlin.build App checks are not gates and were not re-requested. A separate native artifact request for main run 39782188204 found no archive; no attribution is based on that absent result.
Relevant PRs / remote refs
| Repo |
Branch |
Remote head SHA |
PR |
What is on it |
State |
| CodexCoder21Organization/UrlProtocol |
fix/s124-deadline-fixture-clock |
745d842911759c2bd91195efb0db5da393275d83 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/869 |
One-file fixture correction only |
Open, ready/unmerged; both required checks green; 20+20 first passes and 3/3 mutation caught |
| CodexCoder21Organization/UrlProtocol |
wip/s124-write-diagnostics |
3c9022e49569cf320b76d843e7f8732bbddc7210 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/866 |
Minimal future/buffer diagnostics and console-capture workflow |
Closed without merge; minimal capture confirms queued writes; duplicate full suite canceled |
| CodexCoder21Organization/UrlProtocol |
wip/s124-setup-failfirst |
9a815c4e02dc84b5dc29f5fd620b8c0c95563112 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/868 |
Exact setup refusal control |
Closed without merge; 30/30 expected all-63 failures; duplicate full suite canceled |
| CodexCoder21Organization/UrlProtocol |
wip/s124-fixture-verification |
69f4ce87d9b3cdc6e1b9d6b6bca8fbe7988cb414 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/870 |
Forced-condition control and exact-head repetition/mutation workflow |
Closed without merge; 20+20 first passes and 3/3 expected mutation failures; duplicate full suite canceled |
| CodexCoder21Organization/UrlProtocol |
wip/s124-records |
f90341384103a44b059c2686a9fb7cf3ff74bc73 |
No PR |
Native control/capture/minimal/repetition/mutation evidence, strict audits and incremental findings |
Durable evidence, no fix/build claim |
Remaining work for the broader handoff
Review the ready initial-parking correction without merging unless separately authorized. Independently establish the mechanism and deterministic reproducer for the gossip whole-test timeout, the separate later “A later internal chunk fetch must receive a not-dispatched refusal” assertion (original minimal diagnostic copy 072 attempt 3), and the already-on-main large-answer chunk/budget failure. None is claimed fixed by the setup-clock correction. Do not reopen the archived parent handoff or treat this narrow lane as resolving these remaining questions.
Cleanup and evidence
Final native XML, full failed-attempt stacks/consoles, strict audits, metadata, source-tree proof and incremental findings are durable under records. All evidence drafts are closed without merge. The verified check watcher exited GREEN/0; process sweep finds no owned monitors. Cleanup verified at 2026-10-22 21:59 UTC: the entire owned s124-root directory and exact cache entry are absent, after a clean-state/no-stash/no-local-only-commit sweep. Final ready-state metadata, PR description, cleanup proof and completed findings were then committed directly to the records branch and its remote head verified. No owned monitors remain. The broader handoff remains open and unassigned: after the service automatically archived it on a narrow-lane DONE report, it was immediately reopened; fresh completedAtMs=null and claims=[]. The final report is preserved. Do not submit DONE again until the entire handoff is complete.
Identify the two UrlProtocol first-attempt failures before proposing a fix
Written: 2026-10-22 19:51 UTC, lane s117.
RE-VERIFY: All PR/check/branch state below is a write-time snapshot. Recheck GitHub, git ls-remote, and native attempt XML. A final green run after retries does not mean clean first attempts. No flake fix has been established.
Design decisions in force
2026-10-22, original brief: “Write "the mechanism is X, the evidence is Y"; reproduce deterministically (force the shared-state condition, not load); fix the product or the leaking test's cleanup (not the victim's limits). No timeout raised, no load reduced, no assertion weakened.”
2026-10-22, original brief: “Gate: reproducer fails first; after the fix it passes, and the victim passes 20/20 when run in the forcing order.”
2026-10-22, original brief: “If you cannot reach a deterministic reproducer within the time box, report the evidence gathered and stop — do not guess a fix. Do not merge, enqueue or publish. Time box 2.5 h; checkpoint to a wip branch.”
Common brief forbids deployment, reflection, mocks, reduced iteration counts and disabled tests; no subagents. Verification runs on GitHub runners because the shared host cannot reliably execute kompile tests. Required UrlProtocol checks are bld-build and bld-all-tests; App remote checks are informational.
Mission and current conclusion
The user requested execution of p117-protoflakes.md, lane s117. Two tests fail first attempts on unrelated PRs and pass on retry: testRelayedUrlCallsProgressDuringManyPeerGossipFlood (60-second bound), and testDirectRegisteredChunkFetchResendUsesOriginalCallDeadline (All 63 application handlers must be parked). Original full run links and precise timings appear in the complete findings below.
No product fix has been established. Gossip still has no identified wrong mechanism. Pure deadline-copy batches now reproduce the parking assertion with no preceding suite test: all missing handlers correspond to already-failed caller futures, each reporting a 200ms transport admission refusal. Per-offer counters identify an exhausted sixteen-slot write-handoff allowance, with capacity fully recovered by the ten-second assertion. A deterministic public-API control forces that same finite-refusal result on a writable root. Why initial real write completions hold capacity beyond 200ms remains unproven. An independently proven scheduler slot-accounting bug is already fixed upstream; it is insufficient to fix the parking symptom, which persists on the corrected version.
The broader parent handoff url://handoff/handoffs/hf-2026-10-14-land-the-urlresolver-extreme-condition-bug-fixes-fix-the-two-merge-queue-flakes-then-merge-the-reviewed-prs-and-finish-the-rest was already completed/archived on 2026-10-21T06:53:03.522Z when read live. It was not reopened. This is a narrow unresolved follow-on record.
Relevant PRs and refs
| Repo |
Branch |
Remote head SHA |
PR |
Contents |
State |
| CodexCoder21Organization/UrlProtocol |
wip/s117-diagnostics |
be2ef6e0b00e50f691e6968098f59674ae44a48d |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/859 |
Original-test failure-only logging; product unchanged |
Closed without merge; both required checks green; native 3,134 first passes, no fix |
| CodexCoder21Organization/UrlProtocol |
wip/s117-forcing |
ca0310598354f66e261cb0fbeb1fe4b921745a47 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/860 |
Two early resource/scheduler probes; renamed victim batches; product refusal logging; deterministic public-API handoff control |
Closed without merge; bld-build/control green, bld-all-tests canceled; no fix |
| CodexCoder21Organization/UrlProtocol |
wip/s117-records |
2fef62b47475ea2b82e8045103e209a6642e729a |
No PR |
All original/result ZIPs, source snapshots, audit scripts/results, findings, monitor logs and final state |
Evidence only, no PR or build claim |
No deployment or publication by this lane: only WIP branches/draft PRs and diagnostic runner executions. No merge, queue mutation, production restart, or Maven upload occurred. Existing upstream releases were read, not published here.
Complete evidence chain and work performed
S117 findings
Started 2026-10-22 UTC. Scope: investigate UrlProtocol first-attempt flakes in testRelayedUrlCallsProgressDuringManyPeerGossipFlood and testDirectRegisteredChunkFetchResendUsesOriginalCallDeadline. No merge, enqueue, publish, deployment, timeout increases, iteration reductions, or assertion weakening. No subagents per common.md.
Plan
- [completed] Read brief, shared rules, handoff, skills, repository README/instructions, and linked testing/philosophy/CI documentation; collect exact historical failures and timings.
- [completed] Establish full-suite execution model/order and inventory shared resources, lifecycle ownership, and predecessor cleanup.
- [unresolved] Original-defect reproducer: captured deadline admission capacity and forced its finite-refusal contract deterministically, but no wrong lower-layer behavior was proven; gossip timeout not reproduced. All evidence checkpointed.
- [unresolved] No product fix justified: bounded deadline handoff refusal is proven, but why the earlier write completions span that window is unknown; gossip cause unknown. No guessed fix.
- [unresolved] Fix/forcing-order gate not met. Independent reviews performed; evidence draft has both required checks green and 3,134 first passes, but no fix claim. Diagnostic drafts will close without merging.
- [completed] All evidence and the central handoff are durable; all local monitors stopped; all six owned checkouts and their exact cache directories removed and verified.
Documents read
- p117-protoflakes.md and common.md, full contents.
- github-repos SKILL.md, fix-flakey-test SKILL.md, status-report SKILL.md (reading in progress).
- Existing parent handoff hf-2026-10-14-land-the-urlresolver-extreme-condition-bug-fixes-fix-the-two-merge-queue-flakes-then-merge-the-reviewed-prs-and-finish-the-rest fetched; live state will be verified independently.
OBSERVED: Brief reports both victims pass focused loops, but fail inside full suite; shared-state mechanism is a hypothesis, not established.
Progress 17:45 UTC
OBSERVED: Fresh main is 8ee64cd68b1d58065e6edee995026a310bbc22ac. No project AGENTS.md. Read target tests end to end, CI workflows/scripts, relevant README relay/admission/build contracts, TESTING.md required hermetic/TDD/extreme-condition/never-calibrate/public API/clock/forked-test sections, PHILOSOPHY.md, and racing CI procedure. The live testing document still says remote App checks are required; common.md explicitly overrides it for this lane. No scheduling tool is available, so status cadence stays within this turn. Parent handoff is already archived per previous lane reports; do not reopen it for this narrow lane.
OBSERVED: Downloaded original native attempt artifacts for all three requested runs, retained under s117-evidence. Gossip test: run 39788628314 first 62,415ms timeout at whole-test 60s, second 23,982ms pass; run 39811280704 first 62,499ms timeout, second 55,809ms pass. Deadline test: run 39737331931 first 14,713ms fails full assertion "All 63 application handlers must be parked", second 6,092ms pass; run 39811280704 first 13,988ms same assertion, second 5,590ms pass. Full exact failure stacks/consoles are retained.
OBSERVED: Timeout outer stack names executeTestInSeparateJvm. In-run child terminal-memory marker and thread dump independently identify a child JVM. TESTING.md documents one forked JVM per attempt. This refutes shared Java static state, permits, or leaked Java hosts from an earlier test inside a victim JVM. It does not refute host-wide process/port pressure or a within-test multi-host resource coupling.
OBSERVED: Gossip dump on run 39788628314 is cohort 13 at the one-shot stream close (line 140). Run 39811280704 is cohort 15 at controller open (line 134); the captured output later includes the complete success line with all 16,384 announcements and all 1,041 replies. Thus that later attempt made progress through the entire workload after the timer fired; this is evidence against a permanent deadlock, not permission to increase its timeout. Deadline test has no timeout dump, only cleanup stacks. 58 relay forwards are still pending at shutdown in the later run; the assertion doesn't expose missing-call futures.
INFER: Need inspect whether registered target's 63 parked calls monopolize a per-process shared inbound worker budget during their own four-host run. Ordinary RPC allowance is 64 per JVM, but relay handlers have a separate per-RelayService count/executor. No cause established; no fix written.
Progress 17:49 UTC — upstream scheduler finding
OBSERVED: Diagnostic draft https://github.com/CodexCoder21Organization/UrlProtocol/pull/859 is open at be2ef6e0b00e50f691e6968098f59674ae44a48d; full-suite diagnostic run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39825315806 and build run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39825316106 are active. Plain build-watchman is tracked. This is diagnostic evidence only, not a proposed product fix.
OBSERVED: Existing-test edits on this evidence branch: gossip test adds one per-cohort PID/uptime/thread-count println; deadline test saves the exact same 10-second latch result, prints failed-state caller futures and platform thread stacks only when false, then makes the exact same assertion with the same text. No limit, iteration, timeout, or assertion is changed. These are diagnostic edits and are not being offered as a regression fix.
OBSERVED: The pinned kompile-cli 0.0.124 POM resolves manager 0.0.207. Source at its release commit https://github.com/CodexCoder21Organization/kompile-core/commit/40680b32e (Workspace.kt) schedules with outstandingWork++ and admits while outstandingWork < testParallelism; it does not read TestWorkerSlots. Default testParallelism is 4. Later upstream https://github.com/CodexCoder21Organization/kompile-core/pull/390 (commit 8db9c730c7576144689035560451f704a5aefcd7) adds annotation parsing and slot accounting. Thus the gossip victim's existing @TestWorkerSlots(4) is ignored by the local-on-runner half of the race. This is a specific upstream defect, already fixed upstream but absent from the consuming toolchain.
INFER: This supplies a candidate suite-overlap mechanism, not yet a demonstrated cause of either victim's failure. Need hardware-independent scheduler reproduction, plus genuine victim progress evidence. A toolchain bump alone is not an established flake fix.
OBSERVED: Relay stream admission has another candidate coupling: /urlresolver-relay/1.0.0 shares 64 per-root background slots with identify, ping and gossip. The deadline fixture creates 63 exclusive request streams from services[2] to the relay. Transient background streams could prevent all 63 from negotiating. This needs exact refusal/future evidence, rather than assuming the handler allowance is the same as transport allowance.
Resource/invariant table for investigation
| Resource |
Creator / owner |
Closer / terminal states |
Observable contract and forcing experiment |
| Per-attempt JVM |
Kompile launcher owns one attempt |
Launcher reaps child after result/failure |
An earlier attempt cannot retain Java statics in a fresh child; inspect pinned launcher/scheduler source and PID evidence |
| Local test admission slots |
Scheduler reserves declared min(n,pool) |
Release on attempt completion including failure |
Sum must never exceed pool; use public workspace effects to count deterministic admissions |
| Registered-target handlers |
Target RelayService owns each of 64 slots |
Response admission or failed request/close returns exactly once |
Public activeRelayedRequestHandlerCount and handler-entry latches; no cross-test state |
| Background transport slots |
Factory host reserves per root |
Child close releases; root terminal state rejects new work |
64 shared by relay/identify/ping/gossip; hold real background streams and inspect refusal effects/future outcomes |
| Gossip cohort resources |
Test owns 16 sender hosts and held victim lifetime |
Release latch, stop every host, join holder |
Per-cohort timestamps distinguish growing work from permanent wait; no predecessor JVM ownership |
| OBSERVED 17:55 UTC: Initial forcing workflow ran zero tests: CLI rejected probe paths outside a directory named tests. This was my harness layout error, not fail-first behavioral evidence. Scripts moved to investigation/s117/tests and selectors corrected; no test verdict, limit, or assertion changed. Full-suite diagnostic on the other branch remains unaffected. Duplicate automatic all-tests on forcing draft was canceled; its target is only the two forcing probes. |
|
|
|
OBSERVED 17:57 UTC: Both historical gossip dumps contain only the expected shared Netty groups (four libp2p-nio workers, three boss threads), 16 one-shot gossip workers, and a total of 83/85 platform threads. No accumulated sender-cohort event-loop thread generations appear. Netty worker CPU totals are about 23–24 seconds at about 61 seconds elapsed; the snapshots show ongoing stream work or selector waits, not an event loop blocked in registry publication. This refutes the specific within-test leaking-Netty-thread hypothesis in the supplied dumps. It does not identify why whole-test progress exceeded 60 seconds.
OBSERVED: Original deadline failure logs contain 40 and 58 pending forward diagnostics on one caller root, with 33 and 26 interrupted handler stacks respectively during cleanup. Those counts are cleanup observations, not exact entry counts at the assertion. Neither original console contains a background-capacity refusal report. The forcing probe must be compared against this original signature before attributing a cap failure.
OBSERVED: Diagnostic records are durable on https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/s117-records/investigation/s117 at 6f54b9512e9044dd5df714a297b8a7d0ef2b7594. The branch includes all three original attempt ZIPs and the pinned scheduler source/POM evidence; no product code changed.
Progress 17:59 UTC — deterministic probes executed
OBSERVED: Corrected probe run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39826639915 and artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39826639915/artifacts/12446024271 execute both intended tests. Native XML contains 3 failed attempts per scenario, not compile failures.
- Scheduler public workspace probe: all four generated fixture bodies succeed, ledger balances to zero, but peak declared slots is 16 instead of 4 on every attempt. This is a deterministic contract failure in pinned manager 0.0.207, independent of machine speed. Existing upstream fix is published in CLI 0.0.125 / manager 0.0.209; unchanged fixture comparison is being run.
- Background probe: eight deliberately held ping streams on the same root leave exactly 56 of 63 application handlers entered, seven missing, on every attempt. The host reports 14 background refusals and the exact message "Inbound stream '/urlresolver-relay/1.0.0' was refused: the background stream budget on this connection already holds 64 of 64 streams; admitted streams remain open." Finished caller futures carry RelayRequestNotDispatchedException. This demonstrates a deterministic way to reach the victim assertion by exceeding the documented transport cap, but the originals do not report that cap. It is NOT counted as a reproduction of the historical cause, and does not justify changing product limits or the victim.
OBSERVED: No product fix, timeout adjustment, reduced test workload, or assertion change. The upstream scheduler counterexample is the only confirmed wrong behavior so far; the two original flake mechanisms remain unconfirmed.
Pre-registered forcing comparison, 18:04 UTC
Plan: compare CLI 0.0.124 and CLI 0.0.125 on separate GitHub runner jobs, each running 20 copies of each unchanged victim in one ordered batch. Copies differ only in top-level function/file names; original source SHA256 and generation are retained. The pool width stays the toolchain's existing default four, each gossip copy retains @TestWorkerSlots(4), and every timeout/iteration/assertion stays unchanged. This is genuine work through the same public transport APIs; there is no synthetic CPU load and no host-calibrated concurrency knob. Native attempt histories decide first-attempt counts. The counted public scheduler probe is the deterministic contract reproducer. The victim batch tests whether the contract violation actually reproduces the historical symptoms; if it does not, do not claim the scheduler fixes either original flake. No repeated batch tuning is planned. After the comparator, inspect full-suite diagnostics still in flight.
OBSERVED 18:06 UTC: Traced the suggested 64-permit ordinary inbound allowance through Libp2pHostFactory.kt:8929. It is acquired only in the Libp2pRpcProtocol.RpcRequest dispatch arm. The gossip victim sends GossipProtocol.PeerAnnouncement on its one-shot RPC carriers (dedicated one-shot executor), and its relayed application RPC is opaque payload decoded in its supplied registered handler. The deadline victim sends raw relay payloads over UnifiedRelayProtocol; registered request dispatch uses a separate per-service runningRelayedRequestHandlers count and virtual-thread executor. Neither fixture's parked application work acquires that ordinary RPC allowance. Thus this suggested specific permit-stall mechanism is refuted independently of fresh-JVM isolation.
OBSERVED: Plain watcher for forcing draft exited RED because its intentionally canceled duplicate bld-all-tests is terminal; this is expected evidence-harness state, not a test failure. New matrix watcher explicitly includes both comparator job names and will also report RED on that diagnostic draft's canceled duplicate gate after the actual comparator jobs finish. The original full-suite diagnostic watcher remains active on its unmodified head.
OBSERVED 18:07 UTC: First two-version victim batch executes zero tests: the bare JDK container lacks git, so source verification fails with git: not found. I initially added the pinned fetch before reading the exact error; the subsequent correction installs git before actions/checkout (otherwise checkout can fall back to a source archive without .git). These are my workflow setup errors, not victim failures or scheduler evidence. The corrected workflow retains the explicit pinned fetch and unchanged source check. No baseline batch result is claimed from either failed setup.
OBSERVED 18:08 UTC: Version-pair probe run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39827552685 and artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39827552685/artifacts/12447508149 confirm the published scheduler fix. Pinned CLI 0.0.124 again fails all 3 attempts at exact peak 16 versus allowed 4. Only the CLI coordinate and wrapper pin are changed to 0.0.125; the identical test body then passes first attempt in 11,771ms, all four generated tests succeed, ledger balances to zero and peak equals 4. Output explicitly fetches CLI 0.0.125. This confirms a dependency/toolchain contract bug and its already-published upstream resolution; original victim attribution is still pending the unchanged batch/full-suite diagnostics.
OBSERVED: build-watchman reports that the informational buildtest run's /log/raw response lacks Last-Modified, so freshness cannot be verified. It expressly says to keep polling, not declare a stall. Required GitHub Actions suite is still active; no remote run was reset/re-requested/deleted. Read report-challenge skill: its CLI auto-enqueues/merges, contrary to this lane's absolute no-enqueue/no-merge rule, so this friction is recorded in the durable findings rather than invoking that CLI.
Independent test review pass (diagnostic harness)
OBSERVED: Scheduler probe uses public LocalBuildWorkspace and public TestAttemptStarted/Completed effects. The fixture bodies do only check(true); the ledger counts reservations at scheduler admission, so even immediate body completion cannot hide the over-admission. All fixture results must succeed, there must be exactly four, the final ledger must be zero, and the peak must equal four. No reflection, mocks, artificial CPU load, external service calls in the test, or timing threshold drives that verdict. The nested compiler workspace requires real temporary directories; both runtime root and cache are removed in finally.
OBSERVED: Held-background probe keeps eight genuine libp2p PingController streams alive, then runs all 63 original application calls unchanged. Cleanup is the original host/service shutdown and latch release. This intentionally violates the transport background cap and proves only that documented refusal can produce the same high-level assertion. Original refusal evidence is absent, so it is not a root-cause reproducer for the historical failure.
OBSERVED: Unchanged victim comparator regenerates source from the two original files, changes only function names, round-trips that replacement back to exact equality, saves every original SHA256, and verifies src/tests/build.kts/README against pinned main before running. Each CLI version uses a separate GitHub runner with the same image and default four-worker pool. Native attempt XML, command exit, complete output, manifest, and source head are retained. A setup/compile failure or missing XML is not counted as any victim attempt. Existing built-in runner retries remain visible and do not count as first-attempt passes.
OBSERVED 18:10 UTC: After git installation, comparison source verification hit the container checkout's ownership mismatch (fatal: detected dubious ownership in repository at '/__w/UrlProtocol/UrlProtocol'). actions/checkout uses its own temporary HOME for ownership configuration, while subsequent shell git commands do not inherit it. Corrected only the three read-only git calls with per-command safe.directory=$PWD. This second setup failure also executed zero victims. All setup errors and logs are retained; none count toward reproduction.
OBSERVED 18:13 UTC: Corrected comparator https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39828967523 has passed pinned-source verification and batch generation for both CLI versions; both jobs are in the exact-workload step. This is the first comparator run that executes victims. There are exactly 20 unchanged copies per victim, no changed limits/iterations/assertions, no tuned load. Initial full-suite diagnostics remain on https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39825315806, with bld-build green. Duplicate automatic full-suite/build jobs on this evidence-only branch were canceled, never presented as passing checks; the independent comparator builds the actual product it exercises.
Forcing batch baseline result, 18:18 UTC
OBSERVED: CLI 0.0.124 comparison artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39828967523/artifacts/12449020581 has exactly 40 expected victim copies, successful product build, complete 0..19 manifests for each victim, and native command exit 0. Audit checks exact coverage, ordered contiguous attempt history, final/history agreement, no zero-test or build-failure result.
- Gossip: 20/20 first passes, zero failed attempts; first durations 16,469–43,493ms, median 41,051.5ms. The ignored slot annotation is real, but this forcing batch does NOT reproduce the historical 60-second timeout.
- Deadline: 19/20 first passes, one failed first attempt, all 20 final passes. The failure is a DIFFERENT assertion: "A later internal chunk fetch must receive a not-dispatched refusal". No copy fails the original "All 63 application handlers must be parked" assertion. First durations median 4,442.5ms; failed first 13,749ms. This does NOT reproduce the original startup failure, and is not a reliable reproducer of the new assertion failure either.
INFER: The measured scheduler contract bug alone is insufficient evidence to attribute either original flake. No product pin bump is being presented as a flake fix. The fixed comparator and actual full-suite diagnostic still have observations to contribute. Do not retune the batch to this runner to obtain a preferred result.
Fixed scheduler victim comparison, 18:26 UTC
OBSERVED: CLI 0.0.125 artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39828967523/artifacts/12449390112 has the same exact 40 source manifests as 0.0.124, successful product build, complete native histories and command exit 0.
- Gossip: 20/20 first passes, zero failed attempts, 22,619–23,908ms (median 23,640ms).
- Deadline: 19/20 first passes; one ORIGINAL parking assertion failure, "All 63 application handlers must be parked", first duration 14,802ms. All 20 final passes; 21 total attempts.
OBSERVED: Original deadline symptom therefore occurs on pinned main product source with the slot-honoring manager 0.0.209 too. The scheduler upgrade is NOT sufficient to fix this test. This is the first new main-source reproduction of the original parking symptom in this lane, but 1/20 is not reliable enough to write a fix.
Plan update: run 100 unchanged-count deadline copies with the already-written failure diagnostics under CLI 0.0.125's normal four-worker pool. Increase iterations only, not pool width or per-test caller count; all 63 original calls/bounds/assertions remain. This is discovery amplification to capture the exact failed futures/stack condition, not a final regression test. Once the named condition is known, force it deterministically. No further scheduler-as-flake-fix work is justified.
OBSERVED 18:32 UTC: Diagnostic amplification https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39831411400 passed prerequisite setup and generated 100 original-count copies from the already-checkpointed diagnostic source (be2ef6e0b); execution is active. Every copy retains 63 callers and exact original bounds/assertions. Failure-only prints happen before catch/finally teardown, so a caller error captured there is causal evidence rather than a shutdown side effect. Both the actual full suite and this finite diagnostic batch are tracked with plain build-watchman. No local builds/tests, remote resets, CI reruns, merges, publications, or deployments were performed.
OBSERVED: Source inventory counts 1,048 @TestWorkerSlots(4), 121 @TestWorkerSlots(2), and one @TestWorkerSlots(6). The pinned scheduler bug therefore affects 1,170 test declarations, but the unchanged comparison still does not establish either original flake's cause. No metadata changes to victims or toolchain upgrade have been made to a proposed fix.
Diagnostic amplification result, 18:39 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39831411400/artifacts/12449391827 contains exactly 100 successful first attempts, no attempt histories, successful product build, and command exit 0. Durations range 2,396–5,758ms. Failure-only instrumentation therefore captured no missing-caller evidence. This is NOT evidence that the original startup flake is fixed: the earlier mixed batch's one original failure remains, and the condition that differs between batches is not established. No iteration/concurrency tuning or speculative patch follows.
INFER: One remaining evidence-driven ordering hypothesis to check is process lifetime: fresh JVMs rule out shared Java state, but a launcher that reports completion before physically reaping a previous child could allow work to overlap despite declared slots. Inspect the exact published launcher lifetime before claiming or refuting this; the initial source-name evidence alone did not establish physical process exit.
OBSERVED 18:40 UTC: Shared REST core quota is exhausted (0/5000), resets at epoch 1792694570. GraphQL is healthy (remaining 4825); viewer query confirms authentication remains valid. The github-repos search helper misleadingly prints "Invalid GitHub token" for this quota condition. Switched repository discovery/status reads to GraphQL and cached repository search data. No token replacement is needed. This is recorded here because report-challenge's auto-enqueue/merge conflicts with the lane prohibition. Raw artifact download may need to wait for REST reset; existing downloaded evidence is checkpointed. No run is being reset or re-requested.
OBSERVED 18:42 UTC: Read fresh testrunner-jvm README (no AGENTS.md), then exact release source at https://github.com/CodexCoder21Organization/community.kotlin.kompile.testrunner.jvm/commit/f5fee329d0c2f881d875a433effcc92fb48c8162 (published testrunner.jvm 0.0.29, used by BOTH CLI 0.0.124 and 0.0.125 POMs). JvmUnitTestLauncher.executeSync waits for process.waitFor(100ms) to return true before ordinary COMPLETED, reads process.exitValue(), drains output, and reports completion; the enclosing manager worker returns only after executeSync and its finally blocks. Thus a normally completed earlier gossip child cannot keep executing concurrently after its declared scheduler slot is released. The child-reports-result TCP path does not bypass that process wait. Timed-out cleanup is bounded separately and may be investigated from actual timeout evidence; no normal-exit overlap defect found.
OBSERVED: The pure 100-diagnostic batch does not reproduce the original 1/20 mixed-batch failure, so the needed condition remains unidentified. Do not call it fixed, do not tune hardware/concurrency, and do not substitute a metadata/timeout adjustment for cause.
Launch-mode control, 18:48 UTC
OBSERVED: All lane-focused victim comparisons so far used --local; the actual required suite uses --local-and-remote. Read exact CLI 0.0.124 race implementation: local/remote work runs on two daemon parent workers, notification output is buffered, and a failed side reports after a winner. No specific defect found from that read. This is nevertheless a real untested semantic difference from the full-suite launch.
Plan: on one runner, run the same 100 failure-instrumented deadline copies first with --local, then with the actual --local-and-remote flag; no changed test calls, assertions, budgets, or pool width. Native XML/logs/exit codes are separate for each mode. This is a controlled launch-mode experiment, not a request to retry a red required check. Any remote-winning result must be distinguished from local-winning results; do not invent winner attribution if logs do not establish it. No more iteration/concurrency tuning is planned.
Actual full-suite diagnostic result, 18:54 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39825315806 completed SUCCESS on diagnostic head be2ef6e0b00e50f691e6968098f59674ae44a48d. Both bld-build and bld-all-tests are green. Native artifact https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39825315806/artifacts/12450256501 contains 3,134 test results, all first-attempt passes, no attempt histories, zero failed attempts. Gossip passes in 37,118ms, deadline test in 4,496ms. This is one full-suite observation, not a fix or proof of flake removal. Failure-only diagnostics did not fire. The pure 100-copy diagnostic source therefore also did not capture a failed caller.
OBSERVED: Launch-mode control on current forcing head is queued. Superseded head's run was canceled after correcting its upload glob, before it was counted as evidence. No automatic rerun of a required failed job was used.
OBSERVED 18:56 UTC: Full-suite raw job ZIP confirms the local runner result won. Its exact warning: "The remote side of the local-and-remote build race failed during execute: Remote build timed out after 1806584ms (timeout=1800000ms) for runId=a0ca92fe. Last status: TESTING. ... Continuing with the local result." Local native suite still has zero failed attempts. No remote run was reset or canceled manually. This full-suite run does not reproduce either original failure despite the real race timeout. The focused launch-mode control is shorter and a negative result cannot exclude every long-lived-race condition.
OBSERVED: The original full-suite watcher remained alive after both required jobs were verified green through GraphQL; its log was delayed while probing informational buildtest checks. Stopped this no-longer-needed watcher explicitly (only its verified JVM PID), rather than leaving an orphan or waiting on the informational checks. Required-green verdict comes from live GitHub check state plus downloaded native XML. This monitor limitation is recorded, not treated as a test result.
Launch-mode results and newly captured caller failures, 19:12 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39834857871/artifacts/12450753076 contains both exact 100-copy result sets, each final 100/100 passes but four failed first attempts. Local: three original parking failures and one later-refusal assertion failure. Racing: four original parking failures. These are first-attempt failures despite command exit 0; no clean result is claimed. Neither log contains an explicit racing winner message, so winner is not attributed.
OBSERVED: Failure-only pre-cleanup diagnostics show all seven parking failures have active handlers + completed failed caller futures = 63, with missing counts respectively 12, 14, 29, 20, 1, 18, 7. Every completed missing caller failed with RelayRequestNotDispatchedException caused by “Relay message was not admitted within the 200ms budget: stream/connection was closed or remained unwritable, or the per-connection parked-write budget was exhausted”. No background-stream-cap refusal appears. Thus these handlers did not merely miss the ten-second entry wait: their callers had already ended without dispatch. The actual refusal path is sendViaRelayToInternal.sendCommittedRequest -> admitRelayCallMessage using the 200ms bounded Clock admission.
INFER: This narrows the new original-symptom failures to transport admission, but the existing error combines three conditions. It does not yet establish whether failure was budget exhaustion, persistent non-writability, closure, or scheduler delay. Historical originals lacked caller diagnostics, so identical high-level assertion alone is insufficient to assert identical cause. No product fix is justified yet.
Plan: one additional finite diagnostic pass of the same 100 copies, recording the exact admitRelayCallMessage exit reason and connection state at refusal, without changing its decision, budgets, callers, or assertions. This follows the captured transport refusal rather than retuning load. Any proposed fix still requires deterministic forcing of the named wrong condition.
OBSERVED 19:14 UTC: Strict parking artifact audit now verifies exact 000..099 coverage against the manifest, successful product build, contiguous native attempt histories, and final/last-attempt agreement. Both paired modes contain 104 attempts, 96/100 first passes and 100/100 final passes. First-attempt median is 6,253ms local and 6,291.5ms racing; the earlier quick audit median used final durations, so use strict-audit.json for first-duration comparisons. Every missing caller across the seven captured parking failures is typed not dispatched with the same 200ms admission error.
OBSERVED: New reason-only diagnostic head 043be5cdfe46d1887b385e288dfd3fad425034ac is pushed and verified, with unchanged decision branches and limits. It logs park-budget exhaustion versus loop/deadline/close exit, offer count, elapsed injected-clock time, ownership of a parked permit, and channel open/active/writable/autoRead state after refusal. Run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39837592237 is executing the same 100 copied victims once in racing mode. Duplicate bld-build/bld-all-tests were intentionally canceled again. No product fix is being claimed; the explicit bounded-admission contract may allow these refusals.
Exact bounded-admission exit captured, 19:20 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39837592237/artifacts/12451832195 contains 100 exact copies, 95 first passes, five original parking failures, 100 final passes and 105 native attempts, successful product build, command exit 0. The five missing-handler counts are 5,2,13,16,2. All 38 missing callers failed unsent before the entry assertion. Every exact helper exit is loop-ended, closed=false, ownedBudget=false; elapsed injected-clock time 200–252ms after 56–105 offers. Every post-refusal child and root snapshot is open/active/writable with autoRead=true. No park-budget-exhaustion branch executed in these missing callers.
INFER: Repeated zero-wait offers can be declined while the effective writability check does not require a parked slot. This contradicts attributing these five failures to the background cap, a closed transport, or a held parked-write allowance. The remaining checks are handoff-permit availability, ordering/admission predicate, and physical response-read commit guard. Healthy snapshots after the helper exits do not alone prove which check declined offers during the interval. No fix yet.
Plan: classify zero-wait false returns with per-caller counters (handoff-permit, admission-predicate, commit-read-guard, or before-zero-handoff). At the original ten-second failure, inspect public host/root diagnostic semaphore counts before cleanup to distinguish a persistent retained handoff from a transient backlog. Same exact 100-copy workload and no changed verdicts/budgets. New WIP head 3fc7312863db98eba2435b26a916c2fa467c577a; run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39839198068.
OBSERVED: I attempted rebase with unstaged diagnostic edits before this commit; git refused. After committing, re-ran fetch/rebase successfully; origin/main remains 8ee64cd68b1d58065e6edee995026a310bbc22ac and no commits were replayed or conflicts resolved. Remote source head verified by ls-remote. No stale-main build is being claimed.
OBSERVED 19:25 UTC: Latest zero-wait classification run is still executing; no verdict claimed. Review of all five owned checkouts finds no uncommitted files, stashes, local-only commits, or extra worktrees. Only UrlProtocol has lane changes, on the three verified WIP refs. No subagents or local builds/tests were launched. Source logging preserves all old false/true decisions; its per-caller diagnostic ThreadLocal is cleared before offers and in helper finally, and new root-budget prints occur only after the original entry wait has failed. This is instrumentation, not regression coverage or a proposed fix.
INFER: The two later-refusal assertion failures seen in earlier deadline batches remain a distinct unresolved signature within the same original test, not declared unrelated or hidden by final retry success. They may need their own exact causal capture after the parking failure is understood. No change was made for them.
Independent code and attribution review pass, 19:28 UTC
OBSERVED: Reviewed WIP diagnostic diffs separately from the investigation pass. No product true/false admission branch, semaphore acquire/release, close path, request count, assertion, or budget was changed. Diagnostic ThreadLocal state is cleared before offers and on every helper exit; per-caller reason counters require no shared lock. Root inspection reads public host/channel/attribute state only, after the original parking wait has already failed. The extra instrumentation can perturb timing, so measured frequency is discovery evidence, not a claimed baseline rate or proof of correction.
OBSERVED: Generated diagnostic-source.txt and its manifest reflect the extra failure-only root snapshots; function-name reversal verifies each generated copy against that exact diagnostic source, not a falsely claimed byte-identical main file. The comparator from earlier runs separately verifies exact original source bytes. All native failed attempt histories are preserved despite final command exit zero.
OBSERVED: No original-defect fix exists to review. The deterministic scheduler probe validates an already-fixed upstream contract bug, while the occupied-background probe validates a documented refusal. Neither meets this brief's original-defect causation gate. A toolchain upgrade, larger timeout, smaller workload, test slot annotation, setup serialization, or automatic caller retry would not be justified as an established fix from the evidence so far.
Zero-wait rejection check identified, 19:31 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39839198068/artifacts/12453285520 contains 100 exact copies, 97 first passes, two original parking failures and one later-refusal assertion failure, 100 final passes, 103 attempts, successful product build, exit 0. All 30 missing callers (16 and 14) expire after 200–230ms; every rejected offer is classified handoff-permits:0, not the read guard or admission predicate. No parked allowance is owned. At the original ten-second assertion, caller host=2 and both registered-link roots have handoffFree=16, and the caller root also has parkedFree=16. All are open/writable with autoRead=true. Thus the allowance recovered before the assertion; this is not a retained-semaphore leak shown by these snapshots.
The observed causal chain is: sixteen outstanding zero-wait Netty write handoffs occupy writeHandoffBudget -> later initial application sends repeatedly fail handoffBudget.tryAcquire -> their 200ms admitRelayCallMessage budget expires -> RelayRequestNotDispatchedException ends those callers -> their target handlers can never enter -> the fixture's ten-second all-63 assertion fails. The evidence is the per-offer reason counters, completed future stacks, exact missing+active=63 count, and recovered ten-second budgets.
UNRESOLVED: Why the earlier Netty write futures remain incomplete through those 200ms is not established. The finite admission contract permits a typed unsent result; no wrong product behavior or reliable deterministic original-defect reproducer has yet been proven. Checking the published libp2p write-completion implementation before choosing whether further work is justified. No product fix or victim setup change.
Deterministic contract control planned, 19:39 UTC
OBSERVED: Checked jvm-libp2p fork README/FORK.md and source declaring snapshot-25 at https://github.com/CodexCoder21Organization/jvm-libp2p/commit/3dc24d8b66cafa7e6f9a1e26ed4715d3214d8b63. MuxChannel.doWrite retains its child write until the Yamux parent future completes, then queues removeParentWrite/doWrite on its event loop; the semaphore release listener therefore includes that scheduling turn. Yamux PendingWrite combines every parent frame future. No specific wrong completion or retention found. The Maven sources classifier is absent (404), so this is version-declaring repository source, not a verified source-jar/binary identity claim. No upstream code edits or publication.
Plan: one public-API capacity control, not a proposed product regression test: warm one pooled caller stream, open a real raw relay stream on that root, hold its event loop, admit sixteen real zero-wait ping frames until handoffFree=0 with root writable, then call the same indirect sendViaRelay path using the warm stream. An injected ManualClock advances exactly 200ms at that caller's first admission sleep. Assert full typed unsent message, no target application execution, no parked slot consumed, then release the event loop, receive all sixteen real pongs, and verify a fresh application call plus full permit recovery. This forces the exact captured resource condition without artificial CPU load or changing a production budget. A pass would demonstrate the refusal contract, not prove that the historical initial writes were delayed for the same lower-layer reason or establish a product bug.
OBSERVED: Control source head ca0310598354f66e261cb0fbeb1fe4b921745a47 is pushed and verified after successful fetch/rebase on unchanged main. Runner https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39841868135 is active. Shared REST quota exhausted again (0/5000), reset epoch 1792698172 (19:42:52 UTC); GraphQL remains usable. Duplicate standard build/suite cancellation must wait for that reset, rather than replacing credentials or repeatedly calling REST. No App run is being reset or re-requested.
Deterministic public-API control result, 19:44 UTC
OBSERVED: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39841868135/artifacts/12453910864 has exactly the intended one test, first-attempt PASS in 2,234ms, successful product build in 111,434ms, no attempt history, command exit 0. Source head ca0310598354f66e261cb0fbeb1fe4b921745a47. The real sixteen queued ping writes fill handoff capacity while the root is writable. The warm pooled caller reaches its Clock wait, observes handoffFree=0 and no parked capacity charged, advances exactly 200ms, and fails with the full same typed not-dispatched message before event-loop release. Target application count stays one (warmup only). Releasing the loop yields sixteen real replies, restores every handoff permit, and a fresh application call succeeds, bringing exact executions to two. This is a deterministic control of the observed finite-refusal mechanism, not a failing-first regression test of wrong product behavior.
OBSERVED: Latest forcing head bld-build passed while REST quota was exhausted. After quota reset at 19:42:52 UTC, canceled its duplicate bld-all-tests (run 39841868263); no false green gate claimed. No informational App check was re-requested or awaited. The original evidence draft remains both-required-green on be2ef6e0b00e50f691e6968098f59674ae44a48d with 3,134 first passes.
CONCLUSION: Deadline's captured failure mechanism is the sixteen-slot handoff allowance staying full until an initial call's 200ms budget expires, ending that caller unsent and making all-63 entry impossible. The capacity recovers by the entry assertion. No specific defect explaining the original write-completion delay has been demonstrated. Original historical caller outcomes were not captured, so identical assertion alone is not proof of identical cause there. Gossip's historical timeout still has no identified wrong mechanism. No fix, timeout increase, workload reduction, assertion weakening, setup serialization, automatic caller retry, or speculative dependency bump is proposed. The brief explicitly permits reporting the evidence and stopping without guessing when a deterministic original-defect reproducer is not reached.
Brief item audit
- Exact original failures/timings: DONE; all three native archives and full first-attempt consoles/stacks retained.
- Full-suite contribution/shared state investigation: DONE as an investigation; shared JVM state, normal child overlap, ordinary inbound RPC allowance, and leaking cohort Netty groups refuted with source/dumps. Host-wide timing effects and timeout cleanup are not proven absent.
- Gossip deterministic original-defect reproducer and fix: UNRESOLVED; 20/20 first passes in both toolchain batches and one clean actual full suite. Historical dumps show progress, not a specific wrong code path. No guessed patch.
- Deadline deterministic original-defect reproducer and fix: UNRESOLVED; observed handoff-refusal mechanism controlled deterministically, but no wrong lower-layer behavior proved. Three later-refusal assertion failures also remain unresolved, not declared unrelated.
- Fix passes reproducer then victims 20/20 in forcing order: NOT CLAIMED; no fix exists.
- PR and required checks: evidence draft has both required checks green; forcing draft has diagnostic controls only, not passing full-suite gates. Close both drafts without merge. No real fix PR.
- Existing-test edits: diagnostic gossip adds one cohort println; diagnostic deadline stores the exact existing ten-second wait, prints futures/platform stacks on failure, then asserts the unchanged result/text. Later generated copies add failure-only root snapshots. Original tests on forcing branch remain unchanged; the new control is a separate scenario. The failure-instrumented deadline copies still fail on unchanged product behavior; no claim that added diagnostics fixed either test.
- Read docs: complete brief/common, required TESTING/PHILOSOPHY sections, racing CI procedure, all six source-repository READMEs and relevant fork notice, and applied skills listed above. No subagents, local builds/tests, merges, queue operations, deployment, or publishing.
Terminal evidence/PR state, 19:47 UTC
OBSERVED: Both evidence drafts are CLOSED without merging. https://github.com/CodexCoder21Organization/UrlProtocol/pull/859 remains at be2ef6e0b00e50f691e6968098f59674ae44a48d with bld-build and bld-all-tests SUCCESS. https://github.com/CodexCoder21Organization/UrlProtocol/pull/860 is at ca0310598354f66e261cb0fbeb1fe4b921745a47 with bld-build SUCCESS, control SUCCESS, bld-all-tests CANCELLED. The latter is not full-suite-green or a fix PR. Remote App checks remain informational and unmanaged. Native artifacts and live closed/check-state JSON are retained.
OBSERVED: gh pr edit failed with “GraphQL: Projects (classic) is being deprecated ... (repository.pullRequest.projectCards)”; closing succeeded. Updated both closed descriptions via REST PATCH with exact JSON body files, confirmed state/head/body length. No need to reopen or edit code. report-challenge's automatic merge/enqueue conflicts with scope, so this tooling issue is recorded here.
OBSERVED: Explicitly stopped the three completed-diagnostic watcher JVMs as their native results became terminal (known PIDs only), including the final monitor. ps shows no remaining lane monitor/test/build child; only Codex infrastructure and the active short verification command remain. All six checkouts were swept: no uncommitted files, stashes, local-only commits, or extra worktrees. Five read-only clones and their exact cache directories have been removed. UrlProtocol will be removed after the final evidence/handoff checkpoint is pushed and verified. No artifacts/JARs/caches are being committed; native result ZIPs and compressed source/log evidence are intentional investigation records.
Durable handoff, 19:49 UTC
OBSERVED: Created url://handoff/handoffs/hf-2026-10-22-identify-and-fix-the-two-urlprotocol-main-first-attempt-failures in the central Handoff service, ORANGE/unassigned, not completed or blocked. This is a narrow follow-on to the already archived parent, not a reopening of its broader mission. The body includes every WIP ref/verified head, exact native first-attempt evidence, the finite-refusal control and its limits, all refuted hypotheses, unresolved questions, no deployment/publication, and concrete next steps. No work depends on a local tree.
OBSERVED: Before this final metadata commit, remote records head is 1af44a5c5f4a23b8e52130771e8aad018bb8a89c; diagnostic source heads are be2ef6e0b00e50f691e6968098f59674ae44a48d and ca0310598354f66e261cb0fbeb1fe4b921745a47. The final metadata head will be verified and written into the service body before cleanup. No original-defect fix is claimed.
Final cleanup verified, 19:50 UTC
OBSERVED: After verifying the final checkpoint 1e53776500465ac6f36978ebd0e8ada2e7b89d93 with git ls-remote, deleted the remaining UrlProtocol checkout and its exact ~/.aibuildcaches entry. workspace/ is empty: all six owned clones/caches are gone. ps shows no local lane monitor/build/test process; only Codex infrastructure and the active short verification command remain. This final cleanup note is saved to the existing WIP records branch through the GitHub contents API after checkout removal, then its new remote head is verified and recorded in the central handoff. No merge, queue operation, publication or deployment. Original-defect investigation remains unresolved; no guessed fix.
Next steps
- Reuse the passing capacity control on the forcing branch. It deterministically shows that a writable root can return the exact typed unsent error with sixteen real handoffs outstanding and the injected admission clock advanced by 200ms. It is a contract control, not a failing-first test of wrong behavior. Do not replace this conclusion with a speculative fix.
- Trace the sixteen original outstanding write futures under the unchanged 63-caller setup: distinguish child queued-write delay, Yamux parent frame completion, Noise/TCP completion, and the queued MuxChannel completion turn. In the captured failures the counter recovered by ten seconds; do not pursue a persistent semaphore leak without new evidence. No ordering/read-guard or parked-write refusal was observed in the classified failures.
- Once a specific wrong completion/scheduling condition is identified, force it deterministically through public APIs, write an initially failing regression test, and fix its owner. If upstream, reproduce both consumer and library layers; do not hide it in a downstream timeout/retry. The current evidence alone does not establish a product bug: finite admission can correctly report an unsent request.
- For gossip, the original dumps show progress in real stream open/close and eventual complete totals, but no specific wrong mechanism. Collect per-operation progress from a genuine failing run. Shared Java state, normal earlier child overlap, ordinary inbound RPC allowance, and accumulating cohort Netty groups are refuted by the recorded source/dumps. The ignored slot annotation is a real already-fixed upstream defect but not proven to cause this timeout.
- Three later-refusal assertion failures inside the same deadline test remain a separate unresolved signature, not unrelated failures. Capture their exact state rather than relying on final retries.
- Once a genuine fix is established, verify the failing-first reproducer flips and victims pass 20/20 in the forcing order, then obtain green bld-build/bld-all-tests on a real fix PR. No merge, enqueue, publication or deployment is authorized by this brief.
Reusable operational knowledge
Use GitHub runners, not this shared host. A bare eclipse-temurin container lacks git; install it before checkout, then use per-command safe.directory for later git reads because checkout uses a temporary HOME. Diagnostic selectors must live under a directory literally named tests. Native XML and its attempt histories are authoritative; diagnostic job SUCCESS and command exit zero may conceal first-attempt failures followed by built-in retries.
Shared REST exhaustion occurred twice, with authentication still valid in GraphQL. Do not replace tokens on the repository helper's misleading invalid-token report; wait for the reset and use GraphQL for status. At least 60 seconds between PR status reads. After the second quota reset the duplicate standard suite was canceled, while its build had already passed. App remote checks are informational, not re-requested, reset, or waited on.
Plain build-watchman kept probing informational buildtest after required checks were terminal, and hit REST quota 403 on the final run. Native/check-state proof was used and exact known monitor JVMs stopped. gh pr edit failed on deprecated projectCards; REST PATCH with JSON body files updated both closed descriptions correctly. report-challenge auto-merges/enqueues, contrary to lane scope, so friction is recorded here instead.
Cleanup: all local lane monitors stopped, all six trees swept clean with no stashes/local-only commits/worktrees; five read-only clones/caches removed. The final UrlProtocol clone/cache has also been removed; workspace/ is empty. The final cleanup note was written to the existing WIP records branch after removal, producing 2fef62b47475ea2b82e8045103e209a6642e729a, then verified by git ls-remote. This body records that final head. No checkout or local source cache remains. No local builds/tests were started. No code was deployed/published; no PR merged or queued.