Handoff: Finish UrlProtocol's bounded event-loop handoff and prove every callback path

Written 2026-10-19, Lane B. RE-VERIFY: this is a write-time snapshot. Fetch origin/main, compare remote branch heads with the table, query each linked PR with gh pr view <full-url> --json state,headRefOid,mergeStateStatus,statusCheckRollup, and read native BuildTest attempt records before building on a claimed result. Native result projections and run lists have lagged actual attempts during this session.

Design decisions in force

The owner stated these requirements on 2026-10-19; they remain in force:

One PR to UrlProtocol. Its effect: code running on a libp2p/Netty event-loop thread never parks waiting on PeerRegistry, or on any lock, future or wait whose duration it does not control. This covers inbound RPC handling (RpcServerMessageHandler.channelRead0 → RelayDiscovery.onRelayAnnouncement → PeerRegistry.addPeer), gossip and peer-exchange ingestion, and every other path from a channel handler into the registry or into service-level code.

Fix the class of problem at one choke point (a dispatcher), not path by path.

Do not change PeerRegistry's internals; if your change and Lane A's touch the same file, keep your edits to the call sites.

Write the failing test first. Record its first-attempt failure on current main, with the attempt count read from the runner's own record, then show it passing. No mocks or mockito, no reflection, and no public API added only for tests.

NEVER raise a timeout, lower an iteration count, weaken or delete an assertion, or skip or disable a test. Any change to an existing test must be listed in the PR, with the reason.

Open the PR as soon as all of the following hold, then stop: every row's test fails first on main (record the failures) and passes on your head on its first attempt in 5 consecutive runs (attempt counts from the runner's record); the existing relay, gossip and discovery tests pass unmodified; one full-suite run from the pushed head has its counts in the PR.

Do NOT merge, enqueue, publish an artifact, deploy, restart anything, or touch production. Stop at the open PR whose checks meet the bar. Do not write "watch until" loops. Every wait needs a hard deadline and must end on every outcome. Before you finish, kill every background process you started.

Branch names start fix/. Fetch/rebase before builds and immediately before creating the PR. Prefer targeted scripts/test.bash --remote --test <name> while iterating; at most one full suite machine-wide. Required kotlin.build (remote) must be green; kotlin.build (kompile-remote-build) is reported red fleet-wide. A failure must be reproduced on main before calling it unrelated. No blind retries. Do not push any earlier PR branch. If superseding a prior PR, say so in the new PR and comment its full URL on the old PR; do not close it.

PR opening must carry the why below, mechanism, invariants, before/after reproduction and full-suite counts. End PR body with 🤖 Generated with [Claude Code](https://claude.com/claude-code). End commits with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. No new PR is open from this lane, so no supersession comments have been posted.

Mission and current state

Original user motivation: "You said 'That list is full (1,000 entries), and each new entry triggers a slow re-sort while a lock is held.' - this sounds like a disaster and sounds suspiciously like something that a code review https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/CODE_REVIEW.md#further-lock-and-deadlock-considerations would complain about. Seems that we should fix it, delegate the work to gpt 6.1-sol, drive it to completion including code review and merge and deploy."

This lane's later, specific limit is one reviewed/tested open PR, with no merge/publish/deploy. Lane A owns registry internals in a separate checkout. Lane B owns the event-loop handoff; its current implementation is incomplete and must not be called landing-ready. The three-hour execution window is the reason for preserving partial work, not an argument for skipping a gate.

The observed production evidence supplied in the brief: TinySite/Filedrop operations timed out at 30 seconds; CN's 16 libp2p workers had 1–9 threads BLOCKED in every one of twelve thread dumps. Stack: RpcServerMessageHandler → RelayDiscovery.onRelayAnnouncement → PeerRegistry.addPeer → publishCompletedAdd. The lock holder ran bootstrapEvictionVictims/bootstrapRanking, with O(n²) comparisons above the 1000-peer bound. Registry fullness was inferred, not measured live. Raw supplied dumps were /tmp/claude-1000/-code/c501fb07-e4c8-4951-b454-fd406f2af343/scratchpad/inv/evidence/tds/; this lane did not inspect or change production. Main source reproductions confirm a registry callback can park a real event loop; they do not prove every production condition.

Required reading was completed: repository README (no root AGENTS.md existed); full CODE_REVIEW, TESTING, PHILOSOPHY from DocumentationRepository; every prior PR listed below. Read them again if resuming without their context:

• https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/CODE_REVIEW.md — especially locks across IO, further lock/deadlock considerations, unbounded inputs/scale tests.
• https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md — deterministic public-API reproduction and hardware-independent verdicts.
• https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md

Relevant PRs / refs

Main at last fetch: https://github.com/CodexCoder21Organization/UrlProtocol/commit/d8d98998bb2ceef17b5cbd97433c17fa12efbecb. Own source branch first test checkpoint was https://github.com/CodexCoder21Organization/UrlProtocol/commit/0d054252e643a912fe92ed70a1eace0c7f22c9f9. The evidence bundle is https://github.com/CodexCoder21Organization/UrlProtocol/tree/fix/lane-b-extra-reproducers/lane-b-evidence and includes full failure stacks, attempt records, static inventory and independent reviews.

Prior PRs, all OPEN at 14:27 UTC (do not push their branches):

• https://github.com/CodexCoder21Organization/UrlProtocol/pull/694 — CONFLICTING; publication off event loops. Read/reused concepts only; registry changes not copied.
• https://github.com/CodexCoder21Organization/UrlProtocol/pull/725 — MERGEABLE; one-way gossip callbacks off network workers. Concepts incorporated in a fresh bounded owner.
• https://github.com/CodexCoder21Organization/UrlProtocol/pull/724 — MERGEABLE; batch registry ranking.
• https://github.com/CodexCoder21Organization/UrlProtocol/pull/701 — MERGEABLE; wake publication waiters.
• https://github.com/CodexCoder21Organization/UrlProtocol/pull/682 — MERGEABLE; bound addition waits/preserve membership.
• https://github.com/CodexCoder21Organization/UrlResolver/pull/1318 — MERGEABLE; secondhand referral check scheduling.

Nothing was merged, enqueued, published, deployed, restarted or uploaded to production by Lane B. This is verified against this invocation's git/tool/action history; no production-state query was needed to establish absence of our own changes. No statement is made about what sibling lanes or other sessions deployed.

What was implemented and what still needs work

1. Static handler inventory is in lane-b-evidence/handler-inventory.md and findings. It follows channelRead/channelActive/inactive/error/handler hooks, promise/completion listeners, read-state locks, service callbacks and registry paths. No userEventTriggered override was found. Line numbers describe original main, so search names on newer heads.
2. NetworkCallbackDispatcher is one serial daemon worker per factory host, with CAS-only immutable bounded intake: 256 pending items plus one active, estimated 8 MiB retention including active payload. Same typed key replaces pending periodic work. Refusals/coalesces/completions/errors/retained bytes have public operational snapshot diagnostics. The worker catches and reports callback failures, then continues. Host stop quiesces normal work, interrupts active work, waits for transport close, then closes/joins actual worker via a close-join thread.
3. NetworkCallbackLifecycle installs a root attribute owner before connection observers. A raw DSL root falls back to a connection-owned worker. dispatchNetworkCallback runs inline off event loops, queues on event loops. The fallback's complete ownership/join lifecycle is not yet proved.
4. RPC one-way announcements, withdrawals, exchange responses, effect responses, direct-contact and authenticated-interaction callbacks now hand off. PeerExchange admission was moved wholly inside accepted work so queue refusal cannot discard previously allocated unsettled admission; close/refusal must still be tested broadly. Close-record construction/reporting now uses the owner rather than the Netty close-future thread.
5. Proven additional defect: both stream readers called caller-supplied Clock while holding readLock needed by channelRead0. Source moves those Clock calls outside the lock, snapshots work under the lock and revalidates terminal state/ownership before copying, moving queued chunks or waiting. Byte cursors/accounting remain locked. Independent review found no production regression; new fixture constructor cleanup gap was fixed afterward. Corrected-head reader runtime verdict is pending.
6. Required remaining scope, not silently deferred: inbound-admission owner-effect callbacks and aggregate/read-state Clock under monitors; outbox observers/delivery-loss effects and completion callbacks; RPC forwarded-response close/cancellation; RelayStreamController close listeners/responseEnded completion; relay publication failureStillOwnsRegistration provider lock after scheduling onto event loop; relay close-observer executor rejection fallback; raw-host/request-scope worker lifecycle; authoritative withdrawals and mandatory terminal cleanup under queue saturation. Do not simply queue diagnostics asynchronously where callers rely on thrown failure identity, fatal handling or connection-close outcomes. Earlier speculative outbox/cleanup moves were reverted for this reason.
7. Known candidate defects awaiting exact failure details: pending replacement preserves its old queue position, which can run relay A2 before an intervening peer announcement A1 updating the same peer. The mixed-kind order test fails on candidate attempt1. Fix by making replacement occupy latest arrival position while keeping bounded capacity/byte accounting. Three plain host factories eagerly create a worker but do not close it if host construction throws; invalid-address public constructor test fails attempt1. Server factory already has a catch path. Close a newly created empty worker on failed construction without changing public failure identity.
8. Review also found early drain return can skip an unreported refusal on shutdown; however the added public shutdown-overflow test PASSED on candidate7cbb, possibly because terminal close records force another drain pass. That test does not establish a red reproducer for the claimed branch. Refine the public sequence or refute the finding before changing it. Fatal Throwable policy also needs explicit decision/test: current worker catches all Throwable, unlike existing rethrowFatalJvmError sites.
9. Actual same-loop fixture gap found in root review: moving a root does not change an already registered MuxChannel child's eventLoop. Latest fixture closes setup child, moves root, opens echo child afterward, and asserts actual echo and announcement child executors are identical. Earlier registry parked-stack failures are valid, but their root-only echo check is not the final same-child-loop proof. Exact latest variant needs red main then final-head runs.

Invariant gates and test evidence

All tests are new; no existing tests were modified. No timeout/iteration was raised/lowered, assertion weakened, or test disabled. Address-order expectation in a new test was corrected after checking the public registry merge contract: latest address is first and older addresses remain.

Native runs (use per-attempt records, not aggregate PENDING projection):

• https://buildtest.kotlin.build/run?id=aa8681fb — original registry main red attempt1, parked callback stack.
• https://buildtest.kotlin.build/run?id=ef651c7b — five original main tests each FAILED, canonical/highest attempt1, one FAILED attempt. Four fail receiver handoff while held callback parks E; registry shows actual parked stack.
• https://buildtest.kotlin.build/run?id=d9f96236 — first candidate eaa38 source, four older scenarios PASS attempt1; PeerOrder FAIL attempt1 because test expected replacement address rather than actual documented merge. Fixture corrected; not called a flake.
• https://buildtest.kotlin.build/run?id=8e0343dd — close-observer main FAIL attempt1; first two reader fixtures failed preparation compilation (child.connection), no runtime attempt and not behavior proof.
• https://buildtest.kotlin.build/run?id=aef8ac12 — corrected reader fixtures against unchanged main production, both FAIL canonical/highest attempt1, exactly one FAILED attempt, droplet610533439. E WAITING in ReentrantLock.lock → NettyStreamController.channelRead0:9240 / RelayStreamController.channelRead0:13789 while reader's injected Clock held. Full errors in clock-baseline-native-final.json and findings. Snapshot provenance checked via instrumented buildscript bytecode rewriting File(src) to main-target/src.
• https://buildtest.kotlin.build/run?id=360a0f2a — candidate7cbb source, mixed-order/factory cleanup FAILED attempt1, byte-budget/shutdown-overflow PASSED attempt1, droplet610542527. CLI2/4 summary saved. WUI native attempts terminal while top projection still PENDING/empty error. Full failures retrieval in progress at handoff draft.
• Phase2 submission failed before obtaining any runId (dial protocol negotiation); ERROR0/0 is not a test failure.

In flight at draft: relay registry baseline (old root-only executor fixture) is https://buildtest.kotlin.build/run?id=c42b6a28, pending; head-reader16-test submission on source82720 (also old root-only registry fixture) has not exposed a runId yet. Their local client processes have hard deadlines (~14:46 and15:01 UTC) and will be stopped/recorded before final. Re-verify final evidence update for actual outcomes. No full-suite was submitted by Lane B; zero five-consecutive-run gates satisfied. Auto-CI on pushed evidence/source branches is not a substitute for the exact-head full-suite count requirement.

Next steps

1. Re-verify current main, all branch SHAs, prior PRs and native run outcomes. Read every evidence file and mandatory docs. Keep PeerRegistry untouched. Review latest fixture actual child executor assertions before interpreting future results.
2. Get exact candidate failure stacks, implement the two demonstrated candidate corrections, run targeted tests and independent review. Do not infer PASS from successful compilation. Resolve every remaining handler inventory row through a shared bounded owner and explicit mandatory-admission policy.
3. Complete public-API hostile tests for shutdown reentry, terminal delivery under saturation, raw host lifecycle and each newly covered callback boundary. They must first fail on main/old source for the actual mechanism; do not tune thresholds to a host.
4. Run all five invariant tests unchanged on final head in five consecutive runs, checking each runner attempt count is1. Run existing relay/gossip/discovery tests unchanged. Resolve actual failures locally rather than iterate through CI.
5. Coordinate one full-suite machine-wide slot. Fetch/rebase then run full suite from exact head, record counts and push. Before PR fetch/rebase and test again if changed. Open the single PR only when all bar conditions hold. Monitor required remote check with bounded build-watchman task. Do not merge/enqueue/publish/deploy.

Reusable operational knowledge

kompile-cli0.0.124 remote execution uses url://buildtest/ only. --attach reads logs/status/results but does not fetch build output. RPC API getArtifact/getArtifactChunk exists after successful output retention; no supported HTTP artifact route/submission transport was found. Published protocol0.0.1425 lacks source/SCM provenance and was deliberately not treated as exact main.

Targeted immutable workspaces contain build.kts/src/tests, passed using scripts/test.bash --remote --workspace <snapshot> --test <functionName>. Select explicit function names; absolute test paths outside the invocation root selected incorrectly and that local client was stopped before submission. In all remote snapshots verify production source origin separately from test fixture origin.

Local duplicate baseline compilation produced no verdict after long preparation; owned processes were stopped when remote proof arrived. No machine-size/timeouts/count changes were made. Do not call an uncompleted local build a test pass/fail.

The shared WUI may show PENDING after a run completes. Retrieve /api/test-results?id=<id>&page=1&limit=500 and inspect canonicalAttemptNumber, highestAttemptNumber, attempts, and each attempt's terminal verdict. /api/test-progress, /api/run-timing, /log/raw?id=<id> provide other read-only evidence. Never delete/requeue just because the projected list is stale.

Handoff CLI coursier launch handoff:cli:0.0.8 ... spent75–90 seconds resolving dependencies; direct published flat-POM classpath with all87 direct deps (78 unique cached jars) avoided that. Local exact classpath file is /tmp/lane-b-reading/handoff-cli-cp.txt but is a cache path, not a portable artifact. Public command is coursier launch handoff:cli:0.0.8 -r https://kotlin.directory -M handoff.cli.MainKt -- <command>. Health first direct attempt failed EOF0of4/ambiguousRPC; later direct health returned {"status":"OK"}. Never blindly repeat a create/update after an ambiguous write; list/search unique title first.

No report-challenge record was filed because that skill auto-merges, while this brief expressly prohibits merges. Tool/service friction is preserved here and in evidence; do not silently invoke the conflicting auto-merge flow. No build outputs, jars, dependency caches, tokens or private keys belong in the checkpoint.