Finish PeerRegistry incremental eviction validation and open its PR

Updated 2026-10-19 20:08 UTC. RE-VERIFY: this is a live work snapshot, not a completed landing. Re-check the remote branch heads, pending approvals, current native result records, and remote run state before acting. The A3b lane hard stop is 2026-10-19 20:12 UTC.

Terminal A3b state

The full-suite client wrote INTERRUPTED, exit143, with0/0 counts during shutdown at20:08. Those are not usable aggregate suite counts; progress snapshots prove named tests executed. No full-suite pass is claimed. All owned local runners, their children and watchers were stopped at20:08; see a3b/owned-cleanup.json. Required remote CI remained IN_PROGRESS at20:07. No PR was opened. Thirteen of fourteen regression streaks and the complete independent head review are done; churn, seven unapproved legacy replacements, full-suite counts and required green remain unmet. Both questions are unanswered and both proposals unapplied. The source checkout is clean. Later paragraphs preserve timed historical observations; this terminal paragraph supersedes any earlier statement that the local suite or watcher is still active.

Mission summary

Finish Lane A3 capacity eviction in UrlProtocol: a single PR replacing per-add full greedy capacity ranking with an incrementally maintained eviction index. Work must be O(log n) amortized per update/removal, measured by comparisons/candidate visits, and retained index memory O(n). Capacity policy (b) was already chosen and documented by A3; public bootstrap ranking stays unchanged. Stop at an open PR with required kotlin.build (remote) green. Do not merge, enqueue, publish an artifact, deploy, restart services, or touch production.

A3b compiled the predecessor implementation and corrected demonstrated comparison overhead. Twelve non-churn regressions each have five consecutive first-attempt native passes at the ce4726ee checkpoint; their code and test content are unchanged at the current head. The new descendant-tie scenario has five consecutive first-attempt passes at 11eded65. Thus, 13 of the 14 scenarios have that streak; the 100,000-operation churn test remains unresolved at its unchanged 30-second limit. Seven unchanged existing tests still fail under the new capacity contract or retired work counters. Reviewed proposals for both issues remain unapplied pending explicit user approval. The first full-suite run is active with no terminal results or counts yet. The current required check is IN_PROGRESS with run https://buildtest.kotlin.build/run?id=d5ca022c; no execution verdict is available. The tracked standard watcher currently cannot find it in the known-stale HTTP projection, which does not establish executor failure.

Design decisions in force

The user's brief says: “Do not restructure locking, threading or reader snapshots; those belong to lanes A1 and A2.” A3 changes only eviction choice and its maintained data. A1 owns enqueue-only mutations and A2 owns immutable structurally shared snapshots; their later merge order is A3, A2, A1.

The brief also says: “NEVER raise a timeout, lower an iteration count, weaken or delete an assertion, or skip or disable a test.” It says that when an older survivor assertion changes under policy (b), “list it in your findings and the PR and leave it failing for the supervisor to decide.” Neither proposed exception has approval in this snapshot. Do not treat unanswered async questions as approval.

User requires using this existing checkout and warm workspace-path cache; no source copies or fresh source workspaces. Run at most one local kompile invocation at a time. Queue admission is waiting, not a failed test. Never impose a preparation deadline shorter than preparation needs; only the lane wall-clock limit applies. Do not requeue failed tests as a debugging strategy.

What was found and done

The production incident involved Netty event-loop callers entering PeerRegistry.addPeer while another thread performed quadratic greedy ranking at the 1,000-peer capacity under the publication lock. A3 had already chosen a separate incremental capacity policy and wrote five public regressions with native first-attempt failures on main. Its fixed source was never compiled before its lane expired. Those baseline failures are accepted evidence; do not redo them.

A3b started from ca8c6aeb211c4fafda3a79b414967ac625c5fc06 and fetched first. Initial remote submission failed before a run ID with repeated RPC write dropped: stream/connection unwritable after 5000ms awaited-write delivery wait (peer not reading) inside RemoteBuildWorkspace.submitBuildChunked. This was submission failure, not queued admission; only that client was stopped. All subsequent local tests used this checkout and its original cache.

The first actual native run passed 3 of 5 tests. Mixed-address work, 100,000-operation memory bounds, and transitions/restart passed on the first attempt. The policy reference timed out while rebuilding temporary independent expected-node keys. Mixed batch work was 2625 against its unchanged 2551.2407768734943 bound. Restoring temporarily selected victims with insertAbsentEntry removed 67 comparisons, but 2558 still exceeded that bound. The final scoped correction uses one peer-ID TreeMap.put for ordinary upsert, takes the prior immutable entry from its return value, and removes/reinserts only the group memberships. Every actual comparator invocation remains charged. No work bound changed. Independent reference inputs cache immutable per-peer grouping keys while retaining every original full scan, trial and assertion.

After these changes, ten of eleven scenarios passed on their first attempt. All six added failure/restart tests passed: failed refresh version, rejected inputs, callback failure cleanup, cold restart after missing observation, accepted descriptor ID read, and real relay wire restoration after one-shot filesystem failure. These tests use public APIs, real isolated libp2p hosts where needed, FakeFileSystem, no mocks/reflection or test-only API.

The 100,000-operation memory test hit its default 30-second limit on three native attempts. Its self-dump placed the main thread in unchanged PeerSnapshot.updated copying membership for a non-front victim. An unchanged diagnostic invocation with external JFR attach passed first attempt in 21.029 seconds; the earlier initial implementation run passed in 26.929 seconds. In the sampled diagnostic window, 204/294 main CPU samples were snapshot work (69.4%), 25 index work (8.5%), and 4 scoring work (1.4%). These samples locate the dominant cost; they are not exact whole-body CPU percentages. No claim was made that variability is fixed. Snapshot architecture is A2's scope, so A3b did not rewrite it or guess at an index optimization unsupported by the profile.

A concrete unapplied patch adds import Timeout plus @Timeout(60) only to this NEW/unmerged memory test, retaining all 100,000 operations and assertions. The testing guidance permits initial sizing of an unmerged test, but the user's stricter prohibition takes precedence. Explicit approval was requested around 18:23 UTC and remains pending. If denied, retain 30 seconds and report the gate unmet unless an authorized code change actually addresses the mechanism.

Fresh independent review of the full current head found two missing cases, now added and source-reviewed: live missing-record tier-0 eviction with lifetime/service cleanup, and a first descriptor with 10,000 mixed addresses whose numeric minimum IPv4 is last. Both passed their first attempt. A 51-test native neighbor run passed 44 tests: all twelve selected non-churn new scenarios plus 32 of 39 unchanged existing ranking/eviction tests. The twelve non-churn scenarios then completed four further native runs, all at first attempt, for five consecutive passes each at ce4726ee. Their code and test content are unchanged at current head. Independent source and test review is complete for the current head, including the descendant-tie scenario.

Seven older tests failed all three native attempts. Two expect the old greedy final victim; five expect positive work from removed ranking/fallback routes, while the new route reports zero. The missing-record test also expects the opposite old victim after its failing work assertion, so those later assertions were not reached. None of these tests was changed. A seven-file/five-rename proposal retains all seeds, input draws, trial counts, concurrency gates and timeouts, keeps old greedy witnesses, checks exact new-policy survivors and differences, and replaces retired route work with exact zero plus positive bounded index work. Root and fresh independent review checked the proposal. Review caught a reference-only block-tie error: tied blocks must compare the candidate from the most populated address group, not the weakest peer anywhere in the block. The corrected proposal has the A/B => B5 counterexample and is independently source-clean. Explicit approval was requested around 18:45 UTC; no answer is assumed.

A separate new public registry scenario now guards that exact tie through actual capacity eviction: two four-peer /16 blocks, three peers in each largest exact address, eight equal confirmed network peers below positive caller fillers. Whole-block minima would choose A; selected-descendant minima correctly evict B5. It asserts exact survivors, lifetime and service cleanup, index entries, positive index work within the 32 log2(capacity) bound and zero legacy ranking. This new scenario is committed and pushed at 11eded65. It passed on the first attempt in all five native rounds; there is no retry history.

Fresh current-head review and inherited reader-index finding

A dedicated fresh pass of the complete production diff and all fourteen tests signed off exact source 11eded65c at 19:29 UTC, with no new A3 defect. The signed report is preserved in evidence as a3b/rA3b-head11-review.md. Runtime/approval/CI gates remain explicit and unwaived.

That pass also found the same service-sidecar ordering in current source and main: serviceIndex/indexedServicesByPeer are written before a later descriptor address read or injected clock call. If that call throws, peers.compute retains the old descriptor while the service side effects have already occurred. The new capacity entry and accepted sequence are below the fallible call and remain unchanged. This is a source finding, not a native failure proven on either head. A public reproducer proposal is preserved separately and has not been compiled or run.

The user assigns readable peer, service and address indexes to A2 and confines A3 to eviction choice and its maintained data. The inherited reader-index finding is therefore recorded for that designated owner; no A2 source changes or unverified extra tests were added to this frozen A3 head. The scoped review does not claim global registry failure atomicity.

Current policy and complexity

Tier 0 is missing/zero score; tier 1 positive unconfirmed network; tier 2 positive confirmed network; tier 3 positive caller history. Zero overrides protection. Select largest family, largest IPv4/16 or IPv6/32 block, largest exact IPv4 or IPv6 upper /64 address. Every peer has one canonical address: minimum numeric IPv4 else minimum IPv6 upper /64 else no-address. Population ties compare the cached selected descendant by local score, witnessed tenure, accepted sequence, reverse peer ID.

The index uses a fixed four-root array, peer-ID TreeMap, per-level key TreeMaps and population TreeSets. Mutable group population/victim caches are detached from parent sets before mutation and reattached after recomputing. Each entry/update touches a fixed number of logarithmic map/tree operations. Empty groups disappear; live node count is peers + addresses + blocks + families + tiers ≤ 4n + 4. Temporary batch selection removes chosen entries, chooses subsequent victims against reduced populations, then restores entries before ordinary removal/publication. It has no historical lazy deletion queue. Comparison diagnostics include key maps, ordered sets, head visits and root slots.

Existing scoreShapeLock ownership and publication/reader architecture remain. An investigated incomplete-index hypothesis was refuted: add holds concurrentAddPublicationGate.readLock through compute and index registration, while selection takes its writeLock, excluding incomplete accepted adds. Accepted sequence and installed object identity guard stale restoration; metadata and fallible reads occur before the accepted sequence advances. Persistence format is unchanged. A missing live observation is tier 0; saving/restarting recreates a cold record via normal load, preserving persisted caller history, exactly as README documents.

Relevant PRs and refs

A fresh branch PR query at19:34 UTC returned no A3/A3b PR. No source from this lane has been merged, published, or deployed, and no service has been restarted.

Related work remains separate. A3 does not fully supersede PR 724, PR 694, PR 725, PR 701, PR 682, or PR 1318. PR 724 has separate chunked addPeers/onPeersReceived ingestion, visibility, provenance, and accepted-prefix failure behavior. PRs 694, 725, and 701 cover off-event-loop registry publication and lifecycle action ownership, one-way gossip callbacks, and direct publication waiter wakeups, respectively. PR 682 partially overlaps A3 in capacity selection, but its bounded admission and lifecycle contracts remain distinct; its existing capacity tests need integration review against policy (b). PR 1318 changes UrlResolver's age-based departed-peer checks, which A3 does not implement. Do not push their branches or claim full supersession without a current diff review. No comments or closures were made by A3b.

Next steps

1. Re-verify source/evidence heads and current native streak completion. Read full lane findings and independent review reports in evidence a3b/.
2. Obtain explicit answers on the two prepared proposals. Do not apply either without approval. If approved, apply exact reviewed files, review any new delta, and run changed tests natively. For approved churn sizing, obtain five consecutive first-attempt passes with all 100,000 operations unchanged. If rejected, leave the corresponding gate unmet and report why.
3. Complete the planned full-suite invocation from an exact committed head. If head changes after counts, they are not final-head counts; a new final-head run is needed. Any newly observed failure needs its own cause/reproducer, and must be shown on main before being called unrelated. Never blind rerun.
4. Fetch/rebase on origin/main before builds and immediately before PR; run required tests from resulting exact head. Put real full-suite pass/fail counts and all approved existing-test changes in the PR. Opening sentences quote the user's production motivation and plainly state policy (b) contract change. End body with the required Claude Code attribution; commits carry the specified coauthor.
5. Open one PR only when the brief's bars are met. Required kotlin.build(remote) must run and pass; fleet-wide kompile-remote-build red is explicitly informational. Do not merge/enqueue/publish/deploy. Stop and terminate every owned background process before the final report.

Reusable operational knowledge

Working directory is the exclusive rA3 checkout under /tmp/claude-1000/-code/c501fb07-e4c8-4951-b454-fd406f2af343/scratchpad/codex/rA3. Findings are in sibling out/rA3-findings.md under ## A3b. Native records use /tmp/rA3b-*.xml/json/stdout, with meaningful files copied into the A3b evidence branch. Local cache is /home/u/.aibuildcaches/tmp_claude-1000-code_c501fb07-e4c8-4951-b454-fd406f2af343_scratchpad_codex_rA3. Do not replace the checkout path or copy sources to scratch; cold preparation costs many minutes and hundreds of MB. scripts pin kompile-cli 0.0.124.

Targeted example: scripts/test.bash --local --test testPeerRegistryIncrementalCapacityDescendantTie --log /tmp/result.xml --result-file /tmp/result.json. Native attempt-history records retries; a successful test with no attempt-history is attempt1. Finite requested validation batches stop on every nonzero exit, missing verdict or retry, rather than looping until green.

For remote status, scripts/test.bash --remote --attach e4a2d0aa reads an existing run without submitting another; it returned authoritative PENDING. The HTTP /api/runs projection explicitly had outOfDate=true and omitted that run, so build-watchman's not-found warnings did not establish executor failure. The required check for ce4726ee was rejected before execution because the deferred queue was full at 64; tests did not run. Do not confuse rejected with queued, and do not rerequest until admission conditions change. The next pushed source head may get a legitimate new check for its actual changed test source.

The current-head watcher is bounded to end before20:12; stop its recorded process tree at lane end, without cancelling the actual queued remote run. The old-head build-watchman 0.0.21 --run e4a2d0aa monitor session 18805 was stopped at 19:02 UTC; it was monitoring only. Its tracked exit was 130 (INTERRUPTED); it monitored the old pending run and did not cancel or delete the remote run. The official run e4a2d0aa remains PENDING. At19:40 UTC, the required check for current head 11eded65 was IN_PROGRESS with run https://buildtest.kotlin.build/run?id=d5ca022c and no completed stage or execution verdict. Standard watcher session24447 monitors this run; the HTTP projection does not yet list it. No requeue is authorized or inferred from that missing projection. The earlier required check for ce4726ee was rejected before execution because the deferred queue was full at 64. Continue reading current-head status at intervals greater than 60 seconds until an actual run ID or PR exists for standard monitoring. Descendant validation session 31588 completed with exit 0. The first full-suite invocation, session 71787, began at 18:52 UTC from 11eded65, using --local-and-remote --test tests. At 19:31 UTC, the runner reported live child processes under its independent test watchdog. It remains active; no terminal XML, JSON, or test counts are available. Do not use broad process-name matching; stop only owned, recorded process trees. Before final reporting, re-check owned processes and remove ignored launchers/scratch after evidence is pushed. Preserve the warm cache.

Do not upload raw JFR: it may contain ambient environment events. Only execution/allocation stack events were exported and pushed. Full test stdout/XML, failure stacks and runner attempts are preserved separately. Disk had 29 GB free at 18:49; if No space left on device occurs, record and retry that step once, not a test verdict.

A central record now exists at url://handoff/handoffs/hf-2026-10-19-finish-peerregistry-eviction-validation-and-open-its-pr (WUI: https://www.handoff.wasmserver.com/handoffs/hf-2026-10-19-finish-peerregistry-eviction-validation-and-open-its-pr). Structured health returned OK at 19:20 UTC. The first and only create call returned AmbiguousRpcRequestException after its request write, so it was not replayed. An exact-title read at 19:25 found the committed record and matching summary. The list response returned an empty body field; get then failed with the same connection-close/EOF result, so the stored body was not verified. The claim succeeded at 19:29 UTC for codex-A3b-2026-10-19. The full body/status update returned {"updated":true} at19:34, without replaying creation. The subsequent formal report upload failed during service-bytecode fetching, before reaching the report API; its text and full response are preserved. Exact CLI responses are preserved in evidence. Production restoration remains forbidden. This body and all work are also preserved on the evidence branch.

Required docs read for source/test reviews: repository README, DocumentationRepository CODE_REVIEW.md, architecture/TESTING.md, PHILOSOPHY.md; predecessor reading inventory and linked PRs are retained in original evidence. No repo AGENTS.md was found. Native runtime evidence, not source review alone, determines every remaining gate.

Latest central-service verification

At19:34 the full body and blocked-reason update was acknowledged with {"updated":true}. The19:35 formal report upload failed during service-bytecode fetching before the report API call, with direct closed-channel and relay admission failures. The report text and full response are preserved in the evidence branch as a3b/status-1935.md and a3b/handoff-report-1935.stdout. This does not establish a payload-size explanation. The earlier create was not repeated: exact-title lookup proved the original write committed.

Validation provenance: the original five predecessor regressions have accepted main fail-first records. The nine A3b-added failure/edge cases have native head pass records and independent source/test review; no claim is made that all nine failed on main. Some protect preserved behavior rather than define a behavior changed by A3. Do not invent missing baseline results.

At19:55 the next formal report attempt exited1 at setBlockedReason with an ambiguous write result after connection close/EOF; the CLI stopped before report upload. The previous19:34 body/status ACK remains the last confirmed write. New report text and exact response are preserved as a3b/status-1954.md and a3b/handoff-report-1954.stdout. No repeated create or blind write replay was made. Full suite is still executing named tests, with no aggregate result, at the19:55 progress snapshot.