Finish the UrlProtocol snapshot-25 validation and record the snapshot-32 follow-up
Written 2026-10-21 19:30 UTC. RE-VERIFY: write-time snapshot. Recheck https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 with gh pr view --json state,headRefOid,statusCheckRollup,mergedAt; fetch main and verify all remote refs before acting.
Design decisions in force
2026-10-21, p121: land the admission/shutdown changes on snapshot-25; move snapshot-32 and required fixes into a linked follow-up. Owner: "I think we should do the things that the web server does." Owner: "I think the shared buffer size should also have a cap per connection, and the shared buffer should have a maximum bounded size but also should shrink if it isn't being heavily utilized." Retention: "Keep policy, rely on other fixes". Never merge, enqueue, publish, deploy, ask user, raise test timeouts, reduce iterations or weaken assertions. No full local suite; GitHub bld-all-tests is the authorized full gate.
Current r121 work, 2026-10-21 20:00 UTC
PR remains OPEN/unmerged at aee31f8456121877ea04fedcdea1f1047efbbe83. Build39659272454 SUCCESS; AllTests39659272681 still active, no XML yet. Candidate snapshot-25 head8a521a7118c09fb554d7459f809db2c4cf83c51d has ALL26 previous failures passing first attempt:26 attempts,zero histories/failures, no resource abort. Its product source matches aee31f845 byte-for-byte. A47-file serial run is active, including the11 new physical-pause scenarios and the32-specific registration expectation before they move/revert. Repeated file paths (one scenario per file) are proved serial from exact manager0.0.207 source; bare names/directories could run4-wide, and0.0.124 ignores TestWorkerSlots.
Created and GET-verified:
- https://github.com/CodexCoder21Organization/UrlProtocol/issues/795 — snapshot32 follow-up with historical r120 table,26 failures, exact11 moved scenarios, preserved branch and upstream reports.
- https://github.com/CodexCoder21Organization/jvm-libp2p/issues/61 — explicit physical-close retry capability. Standalone published32 repro shows same injected failure twice, one physical attempt, channel stillopen; raw close succeeds. Existing cached-failure ownership is intentional.
- https://github.com/CodexCoder21Organization/jvm-libp2p/issues/62 — paused-child read demand. Standalone public Mplex repro showsread() no events; automatic-read resume delivers the same queued DATA and read-complete. Full assertion/positive control retained.
Both standalone examples compiled/ran serially under the shared JVM lock and8.5GB/8GB guards. No source fix or publication/deployment was performed upstream. Snapshot25 recreates child config percall; therefore the25 guarantee is suspended decoder dispatch and bounded parser-owned/accepted callback charges, not persistent child autoRead/physical pauses/total transport-byte bounds. The current PR README must say this; its prepared text links issue795.
| Repo |
Branch |
Remote head |
PR |
Content |
State |
| UrlProtocol |
https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r121-libp2p-snapshot-32-followup |
aee31f8456121877ea04fedcdea1f1047efbbe83 |
no separate PR |
Exact snapshot32 source/tests before split |
Preserved, linked from issue795 |
| UrlProtocol |
https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r121-snapshot-25 |
8a521a7118c09fb554d7459f809db2c4cf83c51d |
candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Pin25 in build and24 test declarations; no source delta |
26/26 first native passes;47-file boundary run active |
| UrlProtocol |
https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r121-verification-evidence-20261021 |
follow branch for current head |
no PR |
verification/r121 task/findings/commands/XML/logs/issue bodies |
Incremental verified checkpoints |
Remaining work
Finish47-file touched/neighbor run; classify every failed history. Move only the11 newly added32 physical-pause scenarios intact with the bump; revert only the existing delivering-root test's32 pre-callback expectation(2→1) and correct README bound. Verify corrected fixture/four review regressions/manual-read boundary; mutation-check the fixture correction and restore exactbytes. Rebase, verify expected OPEN remote head, lease-push candidate to existing PR and PATCH/GET verify why-first body with issue links/counts. Watch exact-head required CI, read/audit full native first attempts and exact names, update issue795 with final counts, then update/release this handoff and clean owned processes/cache/workspace. No full local suite; no merge/enqueue/publish/deploy/timeout or assertion relaxation. r121 task deadline21:53UTC; local cleanup boundary21:43UTC.
Inherited history and refs
The following snapshot is preserved for provenance and superseded by live r121 state above.
id: hf-2026-10-21-finish-urlprotocol-shutdown-callback-reproduction-and-review-verification-then-update-the-open-pr
url: url://handoff/handoffs/hf-2026-10-21-finish-urlprotocol-shutdown-callback-reproduction-and-review-verification-then-update-the-open-pr
title: Finish rebased UrlProtocol validation and diagnose remaining native failures
summary: PR updated at aee31f845; CI running, final-head native validation incomplete and two startup histories unresolved. All r119-owned processes, checkout and cache cleaned.
created: 2026-10-21T13:44:42.623Z
completed: null
dependencies:
Finish the UrlProtocol review fixes and native validation, then update the open PR
CURRENT19:12UTC: PR https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 is OPEN/unmerged at aee31f8456121877ea04fedcdea1f1047efbbe83 after an expected-head lease push; description change was verified. Build https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39659272454 and All Tests https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39659272681 are in_progress on that exact head. No CI/native full-suite pass claimed. Expected test-name discovery refreshed to2941 after main0ed6401dc. Final rebased focused check stopped at19:13 during compilation without XML; earlier15/15 revised-guard passes belong to0723768ae, not the new base. The preceding selected run had46final/45first passes out of48 with8failed attempts across3histories; one manual-read bug fixed and two startup histories remain unexplained. No full local started because another full suite was present before the authorized cutoff. Source saved to https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-rebased-final-20261021 . No merge/enqueue/publish/deploy. Remaining verification and cleanup are described below; earlier milestones are historical where superseded.
Latest r119 validation, 2026-10-21 18:15 UTC
Refined source ed3b98f249bde8614e45d67bcacc5ebd96d80ede has20/20 first native passes across five separate four-scenario invocations. Final inverse-F1 is6/6 intended red; final decoded-handler inverse is3/3 intended red; both restored exact final source. Final inverse-F2 is still compiling with F1 retained. PR is freshly OPEN/unmerged at000b25266 and remains unchanged. Next: finish that control, run touched43+5 neighbors, the single full-local attempt if preconditions permit before18:26, then lease push/body verification and required-CI/native artifact audit through the19:16 conservative task cutoff. No gate is waived; any unfinished gate stays explicit. Earlier source/controls below are historical where superseded.
Current refinement, 2026-10-21 17:52 UTC
This supersedes older first-fixed-state text below. F1/F2 at dad6de26f passed15/15 first attempts; isolated F1 inverse6/6 and F2 inverse3/3 intended failures were completed/restored. Before PR update, a new public decoded-handler test showed the first F1 pipeline-head EMPTY_BUFFER creates101 interactions for100 requests (3/3 native failures). Refined F1 wakes only installed RPC decoders after the idle observer and leaves decoded handlers alone. Candidate ed3b98f249bde8614e45d67bcacc5ebd96d80ede is on wip/r119-framed-wakeup-20261021; first refined four-scenario build is active with no green claim yet. F2's entire scope remains byte-identical. Main5e3318589 changes only the host-clock fixture; cleanly adopted. PR remains OPEN/unmerged at000b25266 at17:52UTC. Remaining: five refined four-scenario invocations, inverse controls, touched43 plus admission neighbors, full-local attempt/identical-head CI in parallel, lease push/body verification, native CI audit, cleanup. Conservative18:26 local/19:16 terminal cutoffs remain. No new external decision.
| Repo |
Branch |
Remote head SHA |
PR |
What is on it |
State |
| UrlProtocol |
wip/r119-framed-wakeup-20261021 |
ed3b98f249bde8614e45d67bcacc5ebd96d80ede |
candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Refined F1, unchanged F2,4 new tests |
Compiling first final focused invocation; prior dad6 validation retained separately |
Latest r119 milestone, 2026-10-21 17:35 UTC
F1 exact reviewed control is6/6 intended red attempts; main control is2/2 first-attempt green. F2 final public-executor test is3/3 intended red. F1/F2 fixes at dad6de26fa4a3749ab222db269c677230189b0eb have now passed all five separate fixed native invocations: each3/3 first passes, aggregate15/15,15attempts,zero failures/histories. Source is saved on wip/r119-review-fixes-20261021. Isolated inverse-F1 is currently the only source mutation; the controller restores exact bytes in finally before inverse-F2. No test limits or assertions changed.
PR snapshot17:27:OPEN/unmerged at000b25266; candidate not pushed to it yet. Dedicated test/code review notes are in verification/r119/review.md; touched selection42 and four existing admission neighbors are next. Expected CI names2909 is discovery only. After selected checks, start the single guarded full-local attempt, then push the identical verified head and run CI in parallel; full-local still stops at the authorized18:26 checkpoint if unfinished. This overlaps final integration verification without using CI for diagnosis or waiving a gate. Conservative terminal cutoff19:16UTC remains.
| Repo |
Branch |
Remote head SHA |
PR |
What is on it |
State |
| UrlProtocol |
wip/r119-review-fixes-20261021 |
dad6de26fa4a3749ab222db269c677230189b0eb |
candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
F1/F2 fixes and3 regression tests |
15/15 fixed first native passes; inverse/touched/full/CI gates pending |
Current review work
F1: both wire decoders stop on autoRead=false, but request-admission resume only issues channel.read(). The complete tail of a coalesced 110-request write remains in decoder cumulation with no decode event. New real-wire JSON and attachment tests hold handlers until100 pending, then release without sending more client bytes. Both fail3/3 native attempts on the reviewed implementation rebased to main9635bf080. Exact000b25266 control is now also6/6 intended red native attempts; main9635bf080 control is compiling. No weak assertions or timing changes.
F2: returning withdrawal capacity under the scope monitor invokes process-wide listeners, which can enter other scopes. New public-path test uses real Netty LocalChannels and UnifiedRpcProtocolBinding, with a real DefaultEventLoop whose submission boundary enters the releasing host.stop on another thread. A holds one withdrawal; B holds six and waits on its seventh; returning A's charge forces the listener boundary. Corrected fixture fails3/3 native attempts exactly because stop cannot enter until the listener returns. Earlier ClosedChannelException from reusing a one-way child was a fixture error, not a bug result. Planned fix decrements peer accounting under lock and releases shared ownership outside; detaches failed-lane cleanup before closing; starts lazy workers outside lock so rejected dispatch cannot synchronously clean up under it. Keep worker ownership until release notifications finish.
F3: public GossipHandler callbacks run inside work.apply, after runInterruptible enters and publishes lane.action. A callback-entry barrier cannot force stop in the prior action-creation-to-entry gap. No public executor or action-creation hook exists. Keep existing stochastic test and state its 1/3 prior mutation result; no reflection or hook.
F4: not implemented under p119's explicit scope allowance. A synchronous effect from decoder cleanup would put owner callbacks back on the network thread; delivering one through a closed scope needs distinct bounded reporting ownership and an atomic accepted-versus-waiting transition. A shutdown snapshot alone can falsely report an accepted frame whose decoder wait has not yet been returned. This changes lifecycle/reporting semantics beyond the small review fix; preserve this explicit decision in the PR.
Current refs
| Repo |
Branch |
Remote head SHA |
PR |
What is on it |
State |
| UrlProtocol |
wip/r119-review-failfirst-20261021 |
baf088149cab293bcff969708f3d756269996799 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 (candidate only) |
Three new regression tests and rebased inherited work |
F1 two formats3/3 red each; corrected F2 reentry3/3 red; no implementation fix |
| UrlProtocol |
wip/r119-verification-evidence-20261021 |
148358d2abf23a31974addcb8852de7212fa1125 |
no PR |
verification/r119 has task, findings, invariant table, native records, scripts and exact candidate drafts |
Incremental evidence; no jars/caches |
Remaining: exact reviewed/main controls, implement and five first-attempt runs, isolated inverse controls, dedicated reviews, touched39+new tests, guarded single full-local attempt if permitted (stop at18:26UTC), expected-head lease push/body verification, required CI and native artifact audit, final handoff/cleanup. Hard task stop19:16UTC. No deployment or publication occurred. Normal candidate builds rebase first; exact historical controls are explicitly required by p119. One owned heavy invocation at a time, continuous cgroup8.5GB/disk8GB guards. Findings: verification/r119/r119-findings.md on evidence branch; do not rely on local directories surviving.
Prior execution record (historical, superseded by current review findings above)
Finish required CI and native artifact audit for the UrlProtocol shutdown fix
Written 2026-10-21 16:25 UTC. RE-VERIFY: this is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 with gh pr view --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, fetch main, and verify remote branches before acting. The r114 state below supersedes every inherited r109 pending instruction and older verdict below. No user decision is pending.
r114 current state and remaining work
The fix and requested targeted/mutation validation are DONE. The reviewed source was pushed with an explicit expected-head force-with-lease to the existing PR branch. PR remains OPEN/unmerged at 000b252663899c02fea97ea7c3a089b5ad944d73, based on main15bc62a6c2489ebd3c6ae727597da7da9d5d903d. REST PATCH changed the why-first description and a fresh GET verified exact body equality. Rebase explanation: https://github.com/CodexCoder21Organization/UrlProtocol/pull/749#issuecomment-6320934151 . No merge, enqueue, publish or deploy occurred.
Remaining: wait for required bld-build and bld-all-tests on this exact head, download its native test-attempt-history artifact, audit first-attempt results and exact test names, investigate any real failure locally, update the PR with final counts, and complete this handoff only when green and nothing remains other than review/merge. Do not merge/enqueue. Final CI snapshot16:28UTC: bld-build SUCCESS (completed16:27:12UTC), bld-all-tests IN_PROGRESS. The full-suite gate is not green:
- bld-build: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39634411232
- bld-all-tests: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39634411249
Updated from a fresh PR status query at16:28UTC. Native CI counts remain unknown; 2,889 expected names is discovery only. Historical Action39562685044 took67m19s and is not current validation.
The single full-local attempt started15:27UTC after a clean machine-wide preflight and reached native execution with changing test processes. It was unfinished at the explicit p114 checkpoint and its owned process group2344645 was stopped16:21:28UTC. Duration3258seconds, SIGTERM exit-15, no native XML and no full-local verdict. Resource guard never tripped: peak4998533120bytes, disk minimum21010984960bytes. Residual group sweep was empty. p114 expressly authorizes CI as the full-suite fallback; no second local full run was started. This is an incomplete gate, not a claimed passing suite.
Mechanism, evidence and review
OneShotGossipScope.drain formerly published a LAZY child as lane.action before it entered runInterruptible. Host close could cancel that unentered child; accepted work.apply was skipped while finally refunded the reservation. The fix captures the child Job in its body and publishes it only inside runInterruptible, after the interruption handler is installed and immediately before work.apply. Pending accepted work remains drainable; entered work remains interruptible. The pre-edit invariant table is in r114-findings.md. Independent test/code review found no concrete remaining defect. The private entry interval has no public deterministic gate; the public test uses genuine concurrent callers, and p114 expressly accepts the retained3/3 red baseline plus five fixed invocations. No private seam, new timeout, sleep, reflection or mock was added.
- Baseline15A/1B:3/3 first native attempts missed accepted callback IDs. Expected976, actual975/975/973; no extras. Original XMLs matched remote r109 evidence byte-for-byte. Across automatic retries:7attempts =3 missing-ID failures,2passes,2existing runner deadline failures. One deadline dump is pre-stop send setup(line121/137), with held callbacks(line40); the other has exited the test body by dump capture (DestroyJavaVM and GlobalEventExecutor.takeTask), so its earlier phase is unknown. Neither deadline is counted as a callback-loss reproduction.
- Fixed: five separate invocations of fifteen-peer, sibling-stop and self-stop; each5/5 first passes, aggregate15/15,15attempts, zero histories/retries.
- Touched tests on exact 000b252663899c02fea97ea7c3a089b5ad944d73:39/39 first passes,39attempts, zero histories, no missing/extra/duplicate files.
- Lost-entry mutation: first invocation passed, second failed first attempt with missingA-5-13 then passed its automatic retry. Aggregate2invocations/3attempts,1intended failure,2passes. Source restored exactly.
- Isolated generation, FIFO, terminal and root-fixture controls each failed3/3 native attempts for the intended behavior. Generation: old queue overtaken/maxActive2. FIFO: later root consumes capacity before A-7. Terminal: existing B close-future get(line293) expires after A-7 enters, demonstrating never-terminal head. Fixture: exact expected1/actual2. Every hash restored exactly. Inherited baseline15/15 and clock/binary6/6 red controls remain retained below.
- Fixture provenance: isolated73e528617 was cherry-picked as7f84956d0, not an ancestor of candidate24a4e5b83. Fixture blob2f7e331b1d6291f002396b290123eee1b33db9ce is identical in isolated73, candidate24a and current 000b252663899c02fea97ea7c3a089b5ad944d73. Snapshot32 registers roots before invoking the public handler; snapshot25 did the reverse. The correction was already adopted and was not duplicated.
Relevant r114 PRs and remote refs
| Repo |
Branch |
Remote head SHA |
PR |
Contents |
State |
| UrlProtocol |
wip/r80-admission-followup-20261020 |
000b252663899c02fea97ea7c3a089b5ad944d73 |
https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Reviewed fix and all inherited tests |
OPEN/unmerged; required CI in progress |
| UrlProtocol |
wip/r114-shutdown-callback-20261021 |
fd8ff3535220548f96ab21b69b7fe37fb4fd3351 |
no separate PR |
Fixed milestone on earlier main |
15/15 focused first passes; factory byte-identical to rebased fix |
| UrlProtocol |
wip/r114-rebased-shutdown-20261021 |
000b252663899c02fea97ea7c3a089b5ad944d73 |
same existing PR |
Fix rebased onto main15bc |
39/39 touched first passes and all controls complete |
| UrlProtocol |
wip/r114-verification-evidence-20261021 |
f0ece104624f249c116742468c009273241cd987 |
no PR |
XML, full stacks, source hashes/patches, review, findings, helpers under verification/r114 |
Verified checkpoint; follow branch for final cleanup evidence |
All refs were read with git ls-remote at16:24UTC. The evidence branch receives final cleanup checkpoints; verify its current remote head. Source working tree/stash were clean. Source/evidence are separate branches. No artifacts, jars or dependency caches are committed.
Resume commands and operational notes
Clone fresh under workspace; read README/AGENTS and linked Testing/Philosophy. Fetch/rebase before any build. Use build-watchman0.0.21 plain --pr 749 --check bld-build --check bld-all-tests; never --to-merged/--until-merged. Poll PR status no more than once per60seconds. The owned r114 watcher is stopped before exit; start a new tracked watcher on pickup.
At16:24 build-watchman reported an old queued custom check suite107364303889 with zero dispatched runs. Its prescribed three receiver warm requests returnedHTTP404 and one rerequest POST returnedHTTPsuccess {}. Required Actions were independently observed IN_PROGRESS with start times16:23:59/16:24:00; this does not prove the rerequest caused them to start. Do not blindly rerequest again. This dispatch observation is preserved in ci-dispatch-recovery.json/watch.log; report-challenge auto-merges and conflicts with the explicit no-merge rule, so no challenge auto-merge was performed.
Fetch evidence helpers from verification/r114 into a fresh evidence directory. collect-ci-evidence.py RUN SHA validates completed run head, downloads test-attempt-history, audits XML first attempts and exact names; use RUN39634411249 and 000b252663899c02fea97ea7c3a089b5ad944d73 only after it completes, or the verified replacement run on that same head. audit-native.py, expected-ci-tests.json and baseline files are self-contained in r114 evidence. Adapt local root/repo paths; source checkout cleanup is required by this brief. A final green with retries is not a clean first-attempt result. Never weaken counts/deadlines or use CI to debug. Fix failures locally and compare main before changing unrelated tests.
Deployed/published but unmerged: NONE from r114. Command/output records contain only code/evidence pushes, PR body/comment edits and CI observation/recovery; no production operations, Maven publication, merge or queue request. Existing snapshot32 is inherited, not a release by this lane. The service handoff is authoritative; no git mirror edits were made.
Inherited context — historical r109 snapshot, superseded where stated above
Design decisions in force
Owner wording from p99 (2026-10-21): “I think we should do the things that the web server does.” This requires bounded per-connection intake and per-peer/process limits, backpressure for managed withdrawal streams, and progress deadlines that keep running during memory waits. Follow-up verbatim: “I think the shared buffer size should also have a cap per connection, and the shared buffer should have a maximum bounded size but also should shrink if it isn't being heavily utilized.” Up-to1000 retained identities remain as requested: “Keep policy, rely on other fixes”
Task rules verbatim: “No new timeouts/sleeps, no fewer iterations, no reflection/mocks. Never stop to ask the user. Checkpoint to WIP branches often.” “no merge/enqueue/publish/deploy”. Every commit ends Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. Run selected test files locally, one owned heavy build/test at a time, continuous cgroup guard under8.5GB and disk above8GB. Buildtest was explicitly down in the original brief; Actions is the full-suite gate. Never use CI to debug or retry a flaky result until locally understood. No new product fix without a reliably failing public reproducer (>=50% per run; measure at least10 executions, target near100%), then consistent green with <=1% unchanged-baseline probability. Read the linked Testing Architecture and Philosophy before code/tests. Current create-handoff SKILL uses the central service, so do not follow the legacy README merge instructions; the explicit no-merge rule applies throughout.
Mission and why
p109 asks: “Continue: finish the targeted runs (first attempts from native XML), run the post-fix mutation controls (revert each fix, its test must fail), run the PR's touched tests, then push to the PR branch (check OPEN; normal push or --force-with-lease with a PR comment if a rebase rewrote history), update the PR description (why first; list each finding and its fix/test), and watch bld-build + bld-all-tests with build-watchman0.0.21 plain --pr mode; report bld-all-tests first-attempt counts from its artifact.” The larger user request was to investigate why https://my.externalbrain.wasmserver.com/ failed to load and fix its underlying problem; https://github.com/CodexCoder21Organization/UrlProtocol/pull/725 moved callbacks off network event loops, and this follow-up makes their bounded admission correct.
The150-minute r109 session started11:23 UTC and must stop13:53 UTC. A new accepted-callback loss surfaced in native touched verification, so the PR could not honestly be updated as clean. No user decision is pending; this is unfinished engineering work.
What was found and done
- Inherited fixes are on latest candidate, rebased onto main
c72bf541e: managed withdrawal stream and physical-root backpressure instead of dropping; distinct announcement/withdrawal refusal slots; per-peer direct-handler scope across roots and generations; ordered waiting-frame identities across replacement roots; unchanged idle deadline during waits; terminal head refusal exits FIFO; completed binary decoder returns unused parser permits/turn before waiting; parser ownership returns before child close. Snapshot32 was already published and selected because snapshot25 did not propagate child read pause to transport. Nothing was published by r109. Effect sharing concern was refuted against exact0.0.6 bytecode: execution repositories are ThreadLocal; registrations complete before sharing, and caller callbacks own their synchronization. No effect workaround was added.
- Preserved inherited dirty exact-binary-byte assertion before doing anything else, then rebased. Four targeted native controls passed4/4 first attempts with no histories.
targeted-four.xml, audit/resources in evidence; peak3.475GB.
- Exact35 touched-file native run returned34/35 final passes but33/35 first-attempt passes,38 total attempts,4 failures,2 histories. This is NOT clean.
testR105WireWithdrawalsWaitForProcessCapacity first missed an accepted A withdrawal after host.stop and retried pass. testReview725QueuedWorkKeepsItsDeliveringRootAcrossLiveReplacement failed3/3 at expected intermediate network membership1 vs actual2.
- Root fixture finding is confirmed by exact libp2p25/32 bytecode:25 invokes public connection handler before internal registration;32 registers then invokes. Changed only the intermediate expected count1->2 and message, preserving both ping readiness, the callback gate, old-root retirement and later work-order assertions. Corrected test passed its first native attempt in
sibling-red-root-green.xml (3.158s). Isolated fixture commit is listed below; adopted into latest candidate.
- New callback-loss mechanism supported by source and exact coroutine1.7.3 bytecode:
OneShotGossipScope.drain publishes a LAZY child Job in lane.action before starting/entering runInterruptible. close() snapshots/cancels it as if running. Cancellation before the body starts skips accepted work.apply(), but lane finally releases its reservation and advances. This is a concrete implementation gap consistent with missing accepted IDs. Proposed change is to capture the action Job inside its launch, publish it as the first operation INSIDE runInterruptible immediately before apply, and remove pre-start publication. Before publication it is pending and must drain; after publication the interrupt handler is installed, so close interrupts an entered callback. Do NOT apply that change without reliable native red. No product fix was made in r109.
- Public probe evidence:16-peer self-stop (64 withdrawals each) passed1/1 first attempt, so it is a useful passing control, not a red reproducer. Four independent sibling setups x2 rounds:4/10 first attempts failed,6/16 total native attempts failed; below threshold. Shared-server8A/8B probe (512 withdrawals, all16 worker slots real/gated):1/4 first attempts failed,1/5 native attempts failed, all final pass; also below threshold. All failures were exact missing accepted callback IDs, not harness setup failures. Current15A/1B probe retains16-worker boundary,960 withdrawals, all admission child closes observed before release, B stops only after every A's first callback. Only first stop performs close, hence15 actual draining lanes may be a stronger discriminator. Its latest measured JSON is below. All prior tests are retained unchanged; no sleeps/artificial CPU load/private seams/reflection/mocks.
- Post-fix mutation baseline restored product/build files to8b14406ff while retaining new tests:5 scenarios failed15/15 native attempts at intended assertions (distinct refusal visibility, cross-root peer limit, managed peer pause, paused close, parse-before-child-close). Next clock/binary mutation2 scenarios failed6/6 native attempts: paused idle stream did not close after its ManualClock deadline; binary retained8388608 vs actual1048574 bytes. Source restored by exact hashes after each. Generation/FIFO group was stopped after716.96s of compilation without native XML, peak4.778GB; no runtime verdict claimed. Terminal/root-fixture post-fix control was not started. Prior failing-first FIFO/generation/terminal evidence exists in r105, but does not substitute for unfinished r109 controls.
- Dedicated initial code/test static review was clean at older c789 candidate with limits in
review.md; later independent action-entry-review.md checks exact coroutine cancellation, reentrant stop and interruption semantics. action-entry-invariants.md records ownership rows for accepted work/reservations, pre-entry actions, entered actions, and shared host.stop future. Both new probes were drafted mechanically and reviewed by parent; native measurement belongs to parent. No subagent product edits occurred.
- No PR update/push/watcher happened in r109. PR was OPEN/UNSTABLE/unmerged at8b14406ff0b90cefa8fec813809de09650dd8f31 in the13:33 snapshot (refresh final status in evidence). The old full-suite Action https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39562685044 is historical baseline only:2850 final passes,2848 first-attempt passes,2852 attempts,2 failed histories (R90 parse-before-close and PeerRegistryCapacityBatchRemovalCallbackRejectsLaterVictimReadd). Preserve those counts, do not call them clean for current candidate.
Current15A/1B measurement (may be updated before this handoff is final):
{
"invocations": 3,
"firstAttemptFailures": 3,
"nativeExecutions": 7,
"failedExecutions": 5,
"failureCauses": {
"missingAcceptedCallbacks": 3,
"testDeadline": 2,
"other": 0
},
"details": [
{
"record": "jobs/r109/fifteen-sibling-red-01.xml",
"tests": 1,
"passed": 1,
"firstAttemptPassed": 0,
"attempts": 2,
"failedAttempts": 1,
"histories": 1
},
{
"record": "jobs/r109/fifteen-sibling-red-02.xml",
"tests": 1,
"passed": 1,
"firstAttemptPassed": 0,
"attempts": 2,
"failedAttempts": 1,
"histories": 1
},
{
"record": "jobs/r109/fifteen-sibling-red-03.xml",
"tests": 1,
"passed": 0,
"firstAttemptPassed": 0,
"attempts": 3,
"failedAttempts": 3,
"histories": 1
}
],
"reliableReproduction": false,
"reason": "Only7 native executions; two deadline failures are not counted as missing-callback reproduction. Fourth invocation did not start because strict audit rejected unrelated failure kind. No product fix."
}
Relevant PRs / refs
| Repo |
Branch |
Remote head SHA |
PR |
What is on it |
State |
| UrlProtocol |
wip/r109-concurrent-stop-reproducer-20261021 |
0f7564f1264ea2ba554c97b65219a7ebaba909f9 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
8A/8B shared-server probe plus inherited candidate and fixture;1/4 first failures,1/5 native attempts failed. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-fifteen-peer-stop-reproducer-20261021 |
24a4e5b837b74cb785974042373e2f745d08c55e |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Latest candidate plus15A/1B discriminator; native measurement recorded below; no new product fix. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-inherited-test-20261021 |
45d14753b49a0325b906cb63ee5f4386edd0dac1 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Inherited binary exact-byte assertion before rebase; superseded by latest candidate. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-rebased-candidate-20261021 |
c789c12b8401a99bc1d86af7e058576d4be94082 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Earlier rebased inherited fixes; superseded by latest candidate. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-root-registration-fixture-20261021 |
73e528617d58c64e581588c8251d18219a6541ba |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Isolated1->2 connection-count fixture correction; adopted into latest candidate; corrected native first attempt passed. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-sibling-stop-reproducer-20261021 |
eae144de932cac4c31ec98cd2e6f79e396e58f69 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
Four independent sibling setups repeated twice;4/10 first failures,6/16 attempts failed; unreliable red. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-stop-drain-reproducer-20261021 |
7e02200a1390e36238686cdf30eb721bb0e17280 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
16-peer self-stop control; native1/1 first pass, no failing repro. |
Unmerged WIP; see measurements |
| UrlProtocol |
wip/r109-verification-evidence-20261021 |
d0a77c2d7ef96ca5f611d9b64e7e51174915dd78 |
no new PR; candidate for https://github.com/CodexCoder21Organization/UrlProtocol/pull/749 |
All r109 briefs, notes, full native XML/stack audits, exact mutation patches, resource records and helper scripts; no build outputs. |
Unmerged WIP; see measurements |
Latest source candidate is the fifteen-peer branch, current full SHA in table. Earlier WIP branches are deliberately preserved, superseded where stated. Inherited evidence is https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r105-verification-evidence-20261021/verification/r105 and first-round evidence https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r97-verification-evidence-20261021/verification/r97 . Every r109 helper/artifact is under verification/r109 on the evidence branch: p109/p99, out findings copied asr109-findings.md, native XML, full attempt stacks, logs, resource guards, hashes, patches, reviewer notes, draft why-first description and CI artifact scripts. Build outputs and dependency caches deliberately excluded. Source tests live on source branches, not the evidence branch.
Deployed or published but not merged: r109 deployed/published/restarted nothing and issued no merge/enqueue command. It only made test commits/WIP pushes and read-only PR checks; no live code needs reconciliation. Snapshot32 is an existing published dependency, not an r109 release. Main was fetched/rebased before every normal build; latest source work is unmerged. No new PR created; do not close or merge the existing PR.
Historical r109 next steps — superseded by the r114 state above
- Re-verify PR OPEN, remote heads, main, and any other session's ownership before touching branches. Clone fresh under workspace per global rules; r109 clone/cache is slated for cleanup. Fetch latest candidate branch from table and rebase latest main before any build. Read latest evidence
r109-findings.md, original p109/p99, and all review/invariant notes. Mainc72 added2 byte-array tests and transport7-line change; no GossipFactory overlap at last rebase.
- Finish reliable callback-loss reproduction. Run selected
tests/testR109FifteenDrainingPeersKeepAcceptedCallbacksDuringSiblingStop.kts using native XML; inspect every failed stack for exact missing-ID assertion. Measure at least10 executions and >=50% red rate; preferably near100%. If insufficient, investigate/use genuine public callers and precise ordering; never a CPU spinner, changed timeouts, fewer iterations, mocks or private seams. The private publication-to-entry interval has no deterministic public callback hook; reviewer found no legal deterministic gate. Do not substitute a structural claim for evidence.
- Only after reliable red, fix the action-entry publication ordering described above, retaining all ownership/accounting/order cleanup. Verify same reproducer flips consistently, P(passes under unchanged baseline)<=1%; rerun self-stop, sibling probes and existing host-stop/close/queue ownership neighbors. Independent comprehensive tests and code review must cover all invariant rows, not just the missing-ID case. Mutation-revert the new fix after green and require red again.
- Finish post-fix mutation groups generation/FIFO and terminal/root-fixture. Helpers
run-group-controls.py generation-fifo terminal restore original source hashes in finally; first fetch/rebase. Group audit must require all selected scenarios/attempts fail for intended assertions. baseline and clock-binary completed as above. mutation-backpressure earlier hit existing1200000ms buildMaven deadline before runtime (not a red verdict); grouped baseline supersedes its intended runtime coverage.
- Run final exact touched-file list after latest rebase and fixes, with first-attempt XML audit; current committed source has39 touched files, expected2875 full-suite names.
update-test-selections.py computes diff from original8b head and2850 baseline; regenerate against final HEAD. This is discovery expectation only. Task expressly restricts selected local tests and authorizes Actions full suite; do not run whole local suite on shared host. Resource guard continuously reads memory/disk every0.1sec, stops own group at8.0GB (below8.5 limit) or disk<=8.5GB; use -XX:ActiveProcessorCount=4 -Xmx1536m, fork tests retain existing128m. Never increase test/build deadlines or lower iteration counts.
- Check original r97 PID1902169 exited (already absent r109); never touch unrelated PR https://github.com/CodexCoder21Organization/UrlProtocol/pull/748 watcher. Before PR push fetch/rebase latest main, run selected verification and check PR OPEN/head. PR branch
wip/r80-admission-followup-20261020 still8b at snapshot. History rebased, so only if no concurrent branch changes use force-with-lease and add a PR comment explaining rebase. No merge/enqueue.
- Update why-first PR description using preserved
pr-description.md, replace every PENDING count honestly, include fixture reason and new callback fix once proven, list all findings/fixes/tests, preserve historical baseline. Old gh pr edit may fail projectCards; use REST PATCH structured JSON. Do not submit placeholder clean claims.
- Start tracked watcher:
coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlProtocol --pr 749 --check bld-build --check bld-all-tests. Plain --pr only, NEVER --to-merged. Read watch-build skill. Poll GH PR status at most once60sec; no blind requeue debugging.
- Download
test-attempt-history from final HEAD's completed bld-all-tests Action. collect-ci-evidence.py RUN SHA validates head, extracts XML, audits first attempts and compares expected names. Report final/first pass counts, attempts, failed histories, duplicates/missing/extra; a final green with histories is not clean. Fix any real failure locally. Push every milestone and evidence. Then report completion and clean owned clone/cache/watchers. Never silently defer any of these pending steps.
Operational knowledge and limits
- Helpers assume the old absolute scratchpad plus repo path; adapt root/repo/cache paths for fresh clone. They are executable reference evidence, not a mandate to reuse deleted directories.
run-targeted.py stashes dirty/untracked files for fetch/rebase; avoid concurrent editing during it. run-native-guarded.py omits rebase for intentional mutants, used only after controller rebases. audit-native.py counts native <test> histories, not aggregate console green.
- Cold product compilation took8–20min; warmed per-file invocations took30–100s. Two no-XML compiler cancellations/deadlines are infrastructure observations, not runtime test failures. Full compiler/runner thread dumps are preserved; no compiler root cause claimed. An inherited orphan88-file run produced bootstrap60000ms handshake diagnostics and no XML; it was stopped and counted as nothing. Exact35-file selection corrected an accidental R67 prefix selecting unrelatedR670 cases.
report-challenge CLI necessarily auto-merges, conflicting with no merge; it was not run. Challenge details are preserved rather than violating the instruction. Handoff service health was OK via0.0.8. No tracked handoff existed before this record.
- Cgroup oom_kill stayed9 throughout r109; no memory guard trip. Other lanes have JVMs on this machine; stop only owned process groups. Do not delete shared Coursier or another clone's caches. Cleanup exact owned clone and its path-derived
.aibuildcaches only after source/evidence remote verification.
Historical r109 native audit and remaining gate
At13:48 UTC all owned native runners had exited. Latest15A/1B audit has3/3 first attempts failing the exact accepted-ID assertion,7 total native executions:3 missing-ID failures,2 passes,2 existing30000ms test deadline failures. The strict series rejected the two other failure kinds and did not start invocation4. Do NOT count those deadlines as target reproductions or call this a reliable clean sample. One timeout names send helper line121; the other had no extracted main-thread hint, full dumps retained. Next experiment should send each peer's unchanged64 messages concurrently across15 genuine clients (retaining all960, per-peer order, barriers and default deadlines) and diagnose any remaining setup/runtime timeout before measuring enough clean trials. No product fix applied.
Current PR read-only snapshot13:44:OPEN, unmerged, head8b14406ff0b90cefa8fec813809de09650dd8f31; mergeStateStatusUNKNOWN. Nothing was pushed to its branch, no description update, no required-CI watcher/run started for current candidate. Source candidate24a4e5b837b74cb785974042373e2f745d08c55e is remotely verified. Evidence head before final handoff text refresh d0a77c2d7ef96ca5f611d9b64e7e51174915dd78. Final brief audit is in evidence r109-findings.md; unfinished items are explicitly named, not waived.
Cleanup verified13:51 UTC
After verifying final evidence remote SHA 82621ed9183170b75b12805f1ec533cf6eab4f9c and source24a4e5b837b74cb785974042373e2f745d08c55e, clean git status/stash/local-only branch sweep and no owned runner groups, removed only workspace/UrlProtocol-r99 and its exact path-derived .aibuildcaches directory. Other lane checkouts, JVMs, shared Coursier cache and the unrelated watcher were untouched. No active goal, subagent, scheduler or owned watcher remains. Re-clone the latest source WIP branch for pickup; helpers need fresh-path adaptation.
Final r114 cleanup verified
At 2026-10-21T16:27:21.402486+00:00 source/evidence working trees, stashes and local-only commit sweeps were clean and remote heads matched. Removed only workspace/r114 (including its evidence worktree) and its exact path-derived .aibuildcaches directory. Full-run and watcher process groups were empty. No shared cache or other lane was touched. Final evidence checkpoint f0ece104624f249c116742468c009273241cd987 is remotely verified; its final-checkout deletion facts are recorded here in the authoritative service body after the checkpoint. No scheduler was available/created; no active goal or subagent remains. The handoff remains open because CI/native artifact audit is unfinished.
Resume after r119 task limit
Fresh-clone the PR branch and re-verify its live head before writing. Current expected head aee31f8456121877ea04fedcdea1f1047efbbe83, main0ed6401dc012c0230d31bc914241d3082e071b58. Read verification/r119/r119-findings.md, review.md, p119.md, and all native XML/audits in the evidence branch. The final branch includes F1 buffered wake, safe decoded-handler guard, F2 out-of-monitor cleanup, and the extra manual-read guard narrowing. No public API/test hook/reflection was added.
Unfinished: five focused first-attempt invocations on the final rebased head; additional combined F1/manual-read inverse with exact restoration; clean43 touched plus5 neighbor validation; root-cause the two startup/deadline histories without raising limits or blind reruns; follow required CI and audit current-head native first-attempt artifacts. Native test-name manifest is2941 discovery only. Do not reuse the old2910 manifest. collect-ci-evidence.py can audit completed All Tests for the exact pushed head, but read it before use. The prepared combined inverse script was based on the preceding main and MUST be adapted to preserve the newly merged forwarding code before running; it was not executed on the new base. Previous6/3/3 controls belong to ed3 and are historical. F3 remains unachievable at its exact pre-entry boundary through public callbacks; F4 remains unchanged under the explicit scope allowance, with the lifecycle reason in the PR/review.
Never merge/enqueue/publish/deploy under p119. The old orchestrator watcher is not r119-owned; ignore it. r119-owned local native/watch processes stop at the bounded cutoff and only workspace/r119, its exact cache, and out/r119/pycache are removed after all durable records are pushed. Complete this handoff only when CI/native verification is genuinely green and only review/merge remain.
Final r119 source refs (all full URLs)
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-framed-wakeup-20261021 — ed3b98f249bde8614e45d67bcacc5ebd96d80ede
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-manual-read-20261021 — 0723768ae6bc6c3366595f7544a83a3eb2f600e9
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-rebased-final-20261021 — aee31f8456121877ea04fedcdea1f1047efbbe83
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-review-failfirst-20261021 — baf088149cab293bcff969708f3d756269996799
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-review-fixes-20261021 — dad6de26fa4a3749ab222db269c677230189b0eb
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-verification-evidence-20261021 — 597896fab7b53d93a24e048c5298ef6f0a015797
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r80-admission-followup-20261020 — aee31f8456121877ea04fedcdea1f1047efbbe83
- https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/r119-rebased-final-20261021 — aee31f8456121877ea04fedcdea1f1047efbbe83
FINAL bounded-run outcome19:13UTC: rebased-focused stopped during compilation without XML after246.4seconds; no final-head native verdict exists. Watcher exited130 at its bounded deadline; its child group and the native group were terminated. CI snapshot is in verification/r119/ci-final-summary.json; current required workflows remain unverified. Cleanup is complete: workspace/r119, exact cache and Python cache removed; every tracked process group absent. Actual result is verification/r119/cleanup-final.json.
FINAL19:14UTC: all owned cleanup completed and verified. Evidence remote head fd5b6dd991b4ff06dc7da29db221d1a89e235c4e: https://github.com/CodexCoder21Organization/UrlProtocol/commit/fd5b6dd991b4ff06dc7da29db221d1a89e235c4e . r119 stopped under its bounded task limit and releases its claim. The handoff stays open because required CI/native audit, final-head repeat/mutation/selected validation, and the two startup histories remain unresolved. No deployment or publication is present on any branch. All saved branches are review-only Git work.