Finish the deferred-cap proof and validate the UrlResolver review fixes

Written 2026-10-23 11:12 UTC, lane FIX1499.

RE-VERIFY: All state below is a write-time snapshot. Recheck https://github.com/CodexCoder21Organization/UrlResolver/pull/1499 with gh pr view https://github.com/CodexCoder21Organization/UrlResolver/pull/1499 --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, and verify the branch heads with git ls-remote. The PR source is still original f971; review fixes are only on the WIP branch. Do not merge.

Continuation checkpoint, 2026-10-23 11:29 UTC

The final decisions below resolve the old unanswered questions. Item 4 is now tracked at https://github.com/CodexCoder21Organization/UrlResolver/issues/1513 and remains unfixed, as authorized; no hook or visibility change was added. The merged-snapshot assertion is unchanged. Rebase onto main 6895c45 succeeded without conflicts; all 25 patches compare equal in git range-diff. Candidate 32b1a31c4ebbe40580a357442170b402a705ae1a is pushed at https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/FIX1499B-rebased-20261023 . The PR source still remains original f971. Focused validation is in CLI preparation, and full suite/push/new CI are pending. Continuation findings are at https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/FIX1499B-evidence-20261023/lane-evidence .

Local validation checkpoint, 2026-10-23 11:37 UTC

Candidate remains 32b1a31c4ebbe40580a357442170b402a705ae1a. Four focused scenarios pass attempt 1 with no retry history: merged-snapshot unchanged assertion (3.403s), automatic descriptor waiting (5.169s), unchanged held-peer stress (16.364s), and current-main withdrawal stress (66.000s). The full local suite is queued under the machine-wide lock with a three-hour lock-wait cap and no execution kill; execution gets memory samples every two minutes and is allowed to finish. It has not started yet. No PR-source push has happened. Derived focused results are saved at https://github.com/CodexCoder21Organization/UrlResolver/blob/wip/FIX1499B-evidence-20261023/lane-evidence/FIX1499B-focused.audit.json .

Design decisions in force

The owner said on 2026-10-21: "Do we have comprehensive e2e tests? Including scale tests as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos" / "If there are gaps, have sol write the missing tests." / "Then, do another round, Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation."

The lane brief on 2026-10-23 says: "The head commit f971d45d changes assertions in three existing tests and is awaiting the owner's approval; leave it exactly as it is and keep the PR's 'do not merge: awaiting approval of f971d45d' note. Do not change any other existing assertion." It specifically permits restoring a precise never-opened expectation if history shows the loosening was in this PR, and says: "if a precise expectation is impossible, record it as an open question in your final message instead of changing it."

The same brief says: "Absolute rules: never increase a timeout, never reduce an iteration count, never disable/skip/delete a test, never weaken an assertion, no mocks (no mockito), no reflection, never make a private member public for a test. A failing test is a defect report: fix the cause. Error messages state what was wrong and the actual value."

It says: "You may not merge, enqueue, publish artifacts, deploy, restart services, or delete/rerequest buildtest runs." It requires local builds/tests only, first-attempt passes only, at most one GitHub status poll every two minutes, and PR body updates through gh api --method PATCH ... --input body.json followed by readback. Required checks are Actions jobs as defined by this repo; remote kotlin.build is informational. Read a red informational log once and report its cause; do not rerun it. Stop CI watch after 90 minutes.

It says: "Full-suite time budget: a full suite takes 1–2 hours of execution AFTER it acquires the lock, and the lock wait itself can be 1–2 hours. Two suites today were killed by the lane's own outer deadline mid-run, wasting the shared slot. Never wrap the suite in a deadline shorter than 3 hours of execution measured from lock acquisition, never kill a suite that is executing, and size your lane plan so the suite can finish." Full suite command is flock /tmp/claude-501/complete-suite.lock scripts/test.bash --local --test . --log <xml>. Record /sys/fs/cgroup/memory.current before and every few minutes. Shared memory limit is 10 GiB. No JVM tuning flags.

The lane limit is eight hours from 05:32 UTC, ending 13:32 UTC. The owner has not answered a lane-extension question. No suite was started, queued, or killed by this lane.

Push the first commit and every milestone; preserve findings under lane-evidence/ on a WIP branch at least every 30 minutes because this machine has already lost its local disk once. Never push JVM recordings, profiles, heap/thread dumps, environment dumps or system-property dumps. Commit messages end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; PR descriptions end 🤖 Generated with [Claude Code](https://claude.com/claude-code).

Required reading: the complete Testing Architecture, especially extreme conditions, public API only, and never calibrate to a machine; Engineering Philosophy; the repository README; and the original review below. These were read in this lane. No root AGENTS.md exists in the repository.

Continuation decisions, 2026-10-23

The FIX1499B brief supersedes the earlier unanswered questions:

1. "Item 4 (cap regression in the pop / held-check / reinsert interval): do NOT add any test-only scheduling callback, hook, or visibility change. If you can find a deterministic trigger through the public API within 45 minutes of focused effort, use it with a failing-first test and mutation control. Otherwise leave item 4 unfixed in this PR, and record it as a follow-up: open a GitHub issue in UrlResolver describing the interval, the two audits, and why no public trigger exists, and link it from the PR description. Do not ship an untested fix for item 4."
2. "The merged-snapshot never-opened test: leave its assertion exactly as it is."
3. "The approval hold on commit f971d45 (three bounded-record expectations) stays; the PR description keeps "do not merge: awaiting approval of https://github.com/CodexCoder21Organization/UrlResolver/commit/f971d45daae96078705fc62dccf01874b35d9fa7"."

FIX1499B claimed this handoff and is reading evidence in a fresh clone. Historical results below remain historical; no continuation tests or source push have happened yet.

Mission and remaining work

Finish eight findings from the independent review of the outage fix. The original outage involved repeated peer announcements accumulating targeted snapshot replay records and memory. This lane adds failing-first public integration tests and implementation fixes; it must not replace an unexplained failure with a timer or loosen the original stress test.

Six numbered behavior/coverage items have validated WIP changes, and the banner correction is live. Item4 (deferred promotion cap race) is still open. The original merged-snapshot never-opened assertion also needs a precise expectation decision; the brief explicitly permits leaving this as an open question. A full suite, final PR source push, and new-head required CI watch have not happened. The original PR approval hold remains.

What was found and done

1. Cancellation terminal cleanup released queue bytes without releasing each unsent replay generation. A Clock gate before the actual send forces cancellation before a normal dial result can win. Both wholly unsent and successful-prefix/unsent-suffix scenarios fail all three attempts on the old implementation, then pass. Cleanup now completes per-message outcomes and releases resources before invoking callbacks; callback errors are aggregated so one error cannot strand later resources.
2. Capture set the whole batch's enqueued flag before enqueueing every message. Queue refusal therefore retained replay rows for messages never accepted. Enqueue now transfers ownership one message at a time; capture finally releases only messages not actually owned by a queue. Tests cover all-rejected and an accepted prefix followed by rejected suffix, plus publication that occurs before a supplied store throws.
3. A value-equal replay state was treated as the old generation. The store now records each admission's exact state identity, and conditional removal matches identity. Each admission, including reuse of a publication object, creates a distinct state. Released-state admission/pruning avoids an unbounded historical destination map. The source records publication identity before a throwing caller can hide it.
4. OPEN: promotion pops a deferred descriptor under one lock, checks held-peer membership outside it, then locks again. Concurrent ingress can fill the freed deferred slot and the active limit, after which putIfAbsent reinserts beyond the deferred cap. No fix was applied without a reliably failing public test. The candidate design is one atomic selection/check/removal/merge/admission transaction, with launches and warnings outside the lock; this is a design, not verified code.
5. Capture diagnostics counted only payload after capture returned. The captured deque now increments as each JSON message is built and decrements on handoff/final cleanup, so its maximum includes already-built payloads.
6. The held-peer stress timeout exposed loss of delivery intent when a service record was refused before admission or inherited an earlier batch record's failure without invoking its own send. Exact original lines are UrlResolver.kt:24580/24585 (batch handoff flag), :26073–26075 (4096-count refusal) and :27580–27584 (failure copied to unsent suffix). The request had already retired and nothing could recapture the missing service. The fix retains a bounded, payload-free descriptor in the existing deferred map, wakes it on real queue-capacity release, and recaptures services. It distinguishes an unattempted service from an actual attempted failure. A cycle identity stops automatic recovery after an actual service send failure while letting a later explicit request start again. Automatic descriptors wait behind an active capture of the same peer; new external requests still merge and force a full catalog pass. Connected peers count as held; ready deferred peers get progress before blocked first contacts. Close and synchronous launch failure release exact ownership.
7. Added exact-accounting, forced-merge public test rather than editing the three approval-held tests. History shows https://github.com/CodexCoder21Organization/UrlResolver/commit/f592b7f9f0e889b4c33a5e57ca10bdf69c4309ee loosened both never-opened assertions in this PR. testRealRelayReregistrationRetriesRecordWhoseStreamNeverOpened now closes the old root and observes exact FAILED before introducing the replacement; this precise expectation passes even on f971. testMergedSnapshotRetriesRecordWhoseTransportNeverOpened publishes its replacement during capture, before the queued send, so its current schedule allows a first successful send. Its assertion remains unchanged; a precise expectation for that coalescing schedule is still an explicit question.
8. Corrected the live PR banner from two tests to three using PATCH/readback. The exact f971 approval hold is preserved.

Validation and limits

The WIP head compiles and 23 unique focused scenarios pass on attempt 1, with no XML attempt-history. The original unchanged held-peer stress has seven consecutive independent first-attempt passes: 19.155, 19.104, 16.609, 19.240, 16.674, 17.651 and 15.482 seconds. No timeout, iteration count or original stress assertion changed. The three approval-held refresh files are identical to f971 apart from dependency annotations inherited from rebased main.

Twelve isolated mutation controls each fail all three attempts at the intended condition: cancellation cleanup, whole-batch handoff, value equality, capture accounting, ready priority, live membership, publication-before-throw ownership, callback finalization, released-state replacement, forced coalescing, capacity-intent retention, and the automatic active-peer wait predicate. The last public B/C schedule gates real bookkeeping calls: B's second remove leaves its original capture active and its descriptor pending; C's first automatic publication proves promotion processed B first. Removing only the predicate leaves zero instead of one deferred descriptor. Fixed code delivers both records to both real receivers and releases all counters.

Independent test and code reviews were completed; their source notes are in evidence. One review found the missing active-peer wait test, which was added, mutated, and validated. Two explicit older bookkeeping fixtures were also executed and pass, and all interface implementations were inventoried. One helper incorrectly caught its own successful SystemExit and returned 1 after every mutation had printed VALIDATED; the XML results are valid, source was restored, and only that helper catch clause was corrected.

No full suite has been run for these review changes. The original PR's historical full-suite result is not a result for the WIP head. No new-head CI watch has run because the WIP source has not been pushed to the PR branch.

Item4 testability question

The public API testing rule says "Visibility should never be increased purely for testing" and bars added internal test bridges. The exact promotion interval has no existing Clock/store/queue callback: PeerRegistry and RelayService are final concrete map readers, registry insertion snapshots caller getters, relay-client descriptors skip the held check, and resolver Host/RelayService instances are private and not caller-injectable. Clock, bookkeeping publication and public queue pauses occur outside this interval. The audit is evidence of no seam found, not a proof none can exist.

The owner was asked whether to permit a narrow scheduling test gate as an explicit exception, or leave this item open until a public deterministic seam is identified; no answer arrived. Do not infer approval from elapsed time, introduce reflection, expose private state, or tune thread counts until a race happens on this host. A second independent review has now traced both f971 and current code and also found no existing public gate. Its evidence is in FIX1499-cap-boundary-review.md.

Relevant PRs and refs

The original independent review and probes are recoverable at wip/REV1499-review-20261023. The final why-first PR body was PATCHed and independently read back exactly at about 11:10 UTC; it explicitly separates WIP review results from the original PR head. Main WIP base was last rebased to 0b63446b4; re-fetch/rebase before every further build and before any PR push, then revalidate changed code. All local task checkouts were swept: main FIX1499 and read-only kompile-cli clean/no stashes/no extra worktrees; baseline clean and its sole local-only-reported commit is verified present at the remote ref above (remote tracking ref was not created by its explicit refspec push).

Deployed or published but not merged

This lane deployed or published nothing, restarted no service, and made no production file/process changes. The only service calls were read-only health/process facts and Handoff metadata creation. No artifact publishing CLI or deploy CLI was invoked. The production resolver version was not verified and is not claimed here.

Write-time original-head CI

The PR's required Actions job bld-build is SUCCESS. This repository defines that single job, which performs the build and full test suite. The informational remote check reports 2,439 passed and 6 failed, then failed to retrieve the detailed attempt results with an RPC handler timeout. Its linked https://buildtest.kotlin.build/run?id=6e47ca6e now says "Build run not found: 6e47ca6e". The six test failures cannot be classified as code versus infrastructure without their missing details. Full returned error stack is in lane-evidence/FIX1499-original-remote-check.md. No informational run was deleted or rerequested.

The owner-decision dependency is tracked at url://handoff/handoffs/hf-2026-10-23-decide-whether-the-deferred-cap-regression-may-use-a-narrow-scheduling-gate (human page).

Next steps

1. Get an explicit decision on the narrow scheduling gate, or identify an existing public deterministic schedule that the independent audit missed. Write a test that fails on f971 first, then implement the atomic deferred promotion transaction and demonstrate the cap invariant. Add an isolated mutation check. Keep close/re-entry ownership and the automatic recovery predicate intact.
2. Resolve the merged-snapshot never-opened precise expectation, or carry its explicit open question forward as the brief allows. Do not alter any other existing assertion or remove the f971 approval hold.
3. Rebase onto latest main, run focused neighbor checks appropriate to any conflict, and run the full local suite under the machine-wide lock with memory samples. Allocate at least three hours of execution after lock acquisition, plus possible lock wait; get a lane extension or continue in a fresh invocation instead of killing an executing suite. Count only attempt-1 passes.
4. Recheck PR OPEN, then push the final validated head to its original source branch using --force-with-lease. Preserve WIP and evidence branches. Update the body why-first through PATCH --input and read it back.
5. Watch only the required Actions checks with build-watchman, at most once every two minutes and for at most 90 minutes. Do not warm/retry/enqueue/merge. The read-only watch mode must be used; in the inspected 0.0.21 CLI, --until-merged is the strictly passive mode despite its name, while the ordinary PR path can remediate. Re-read the watch-build skill before invoking.
6. Perform final background-process cleanup, confirm with ps -eo pid,args -ww, and report what remains. Never kill an executing suite. Nothing may be merged, published, deployed or restarted by this lane.

Reproducers and operational notes

Fresh clone goes under $S/workspace/<lane>. Local findings are $S/out/FIX1499.md; recover the evidence branch if local disk disappears. The exact-head baseline is a separate clone and intentionally predates current main; never mistake it for the final candidate branch.

Example focused invocation after the required fetch/rebase:

COURSIER_MIRRORS="$S/out/FIX1499-coursier-mirror.properties" scripts/test.bash --local --test testTargetedSnapshotSurvivesSharedQueueCountRefusal --log "$S/out/shared-count.xml"

The test deterministically fills 4,096 global message slots using genuine small broadcasts, observes the target service's SHED_GLOBAL, releases pressure through public registry/broadcast operations, and requires delivery without another snapshot request. The earlier f971-production-on-current-build baseline fails all three attempts after every pressure/cleanup precondition. The supplemental entire-checkout f971 build succeeds and all three attempts fail the same intended missing-delivery assertion (12.802/12.739/13.025 seconds). Other exact test selectors and mutation scripts are preserved in the evidence branch, with result-audit.json and held-stress-first-attempt-proof.json.

Public kotlin.directory dependency GETs stopped returning headers during this lane even though TCP/daemon health answered. Do not call that slowness a diagnosed root cause. No production restart was authorized. A task-local, read-only loopback Maven tree mirror serves immutable already-cached POM/JAR/checksum files; Coursier's supported COURSIER_MIRRORS overrides each newly-created workspace generation's repository locations. It makes no cache or production writes. It rejects duplicate paths whose bytes differ, traversal, generated memory repositories, and the resolver 0.0.1333 self-artifact. It may serve older resolver862 needed by the Handoff CLI.

Mirror helper and notes are evidence files. Reconstruct its named .from/.to/.type=tree configuration locally and set a fresh loopback port; its prior listener will be stopped before this session ends. Cached annotations0.0.10 was absent from this lane's surviving cache roots; 76 complete cached copies found by bounded exact-layout lookup agreed on hashes, and four immutable files were copied only into task-local staging. Those JARs/caches are deliberately not pushed. If a fresh host lacks the artifacts and the real Maven host still fails, that preparation issue must be resolved separately; do not count it as a test result.

Hypotheses refuted by exact traces: outer first-contact permits alone, provider freshness, and slow close do not explain the deterministic missing service. It is rejected before its own send (or gets a prior message's result), then has no surviving pending intent. Automatic service-only recovery avoids regenerating the peer catalog and must stop after an actual own-send failure.

Raw JVM/thread dumps, trace buffers and Maven binaries remain deliberately unpushed because they may contain credentials or are build/cache artifacts. Only reviewed derived summaries and source-only patches are remote. The report-challenge skill auto-merges, which this lane forbids, so ecosystem friction is recorded in these findings instead of invoking that tool.