Handoff: relay announcement loss remains 100% reproducible after isolating ACK-admission starvation

Written 2026-07-17 UTC.

RE-VERIFY: Everything below is a write-time snapshot. Before acting, re-run gh pr view 369 --repo CodexCoder21Organization/UrlProtocol --json state,mergeStateStatus,statusCheckRollup,mergedAt and gh pr view 788 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt. For investigation branches, fetch the named remote branch and verify its head SHA. Fresh GitHub/build state always wins over this document.

Mission summary

Continue the second session of the UrlResolver relay announcement-loss investigation to one of two honest outcomes:

1. identify the remaining end-to-end mechanism, fix it at the owning layer, publish/bump upstream as needed, and prove the unchanged four-JVM/two-CPU amplifier passes at least seven times; or
2. after substantial unsuccessful work, stop and leave a complete central handoff.

The fixed reproducer is UrlResolver/tests/stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation.kts: 52 providers, 8 CPU spinners, 128 MiB heap per child, four child JVMs concurrently pinned to CPUs 0-1. It must not be weakened. Baseline is 4/4 failures (100%) on both protocol 0.0.360/snapshot-14-effective and protocol 0.0.367/snapshot-18.

This session reached outcome 2. It isolated and deterministically reproduced a real relay registration ACK-starvation sub-mechanism, but the bounded upstream correction still failed the unchanged consumer amplifier 4/4. It was therefore not published or offered as a fix.

What was found and done (the chain)

1. Re-established the prior evidence. The full ongoing record is committed at UrlResolver investigation-log.md. Snapshot-18 eliminates the Negotiator.ResponderHandler close exception but not loss. Prior executor isolation, bounded PEX, mux preservation, targeted-gossip removal, and acknowledged relay-registration refresh experiments all remained below verification.

2. Traced the relay registration transaction end to end. Temporary generation-tagged diagnostics recorded registration-map apply, ACK admission, controller activation, peer-registry apply, stream cleanup, keepalive overwrite, expiry, and the UrlResolver callback. Two unchanged traced batches failed 4/4 each:

   • batch A missing 11, 7, 12, and 13 services;
   • batch B missing 16, 9, 11, and 11 services.

   No run contained a stale stream cleanup removing a newer registration or a stale keepalive overwrite. Missing providers either never reached relay apply, or reached registeredPeers.put but never reached ACK/peer-record apply.

3. Proved one exact rollback mechanism. A repeat with explicit ack-failed tracing failed with 51/52 learned. The only missing service, creg.provider.service.50, was applied twice; both attempts failed before ACK and peer-record apply. All 16 ACK failures in that run were:

   foundation.url.protocol.RelayException: Relay message was not admitted within the 200ms budget:
   stream/connection was closed or remained unwritable, or the per-connection parked-write budget was exhausted
   

   Verified mechanism sentence: ordinary writes on a provider's multiplexed connection can occupy all 16 shared parked-write permits, so the relay applies a registration but cannot admit its small ACK commit record and correctly rolls the unacknowledged registration back; repeated attempts can leave that provider absent from the relay peer record.

4. Built a deterministic upstream RED guard. The new real-libp2p test testRelayControlFrameIsNotStarvedByOrdinaryParkedWrites parks exactly 16 ordinary writes on one physical connection, submits a registration-ACK-shaped control frame, holds the stall beyond the normal 200 ms admission window, then restores writability. Before the behavioral change it failed exactly at the intended assertion:

   The control frame must remain admitted in its reserved bounded lane...
   Expected <1>, actual <0>.
   

5. Implemented and locally verified the bounded sub-fix. The experiment added RelayMessageSender.sendControlMessage, one separate control permit per root connection, and routed registration ACKs through that lane using the existing bounded delivery wait. Worst-case parked writes rose from 16 to 17 per physical connection; it did not introduce an unbounded queue. The deterministic upstream test passed after the change.

6. Falsified it as the complete announcement-loss fix. Using the locally built, unpublished foundation.url:protocol:0.0.370, the exact unchanged consumer amplifier again failed 4/4:

   | Run | Learned | Missing | |---|---:|---:| | 1 | 42/52 | 10 | | 2 | 41/52 | 11 | | 3 | 38/52 | 14 | | 4 | 40/52 | 12 |

   Local logs are /tmp/creg-control-fix-batch-1/run-{1..4}.log on the investigation host. The result means ACK starvation is real but not sufficient: many requests still never reach relay apply, or the confirmed PEX recovery path still cannot complete. Do not publish 0.0.370 or present this experiment as the announcement-loss fix without new evidence and a passing seven-run batch.

7. Observed a separate withdrawal-integrity defect, not yet causally generalized. One prior traced run fully ACKed and added creg.provider.service.46, then replaced its services with [] after a withdrawal before the assertion. Sender-tagged follow-up traces proved the relay accepts a ServiceWithdrawal.providerPeerId different from the registered stream sender. However, that follow-up had zero pre-assertion withdrawals, so third-party/synthetic withdrawal remains a real protocol-integrity hypothesis rather than the general mechanism behind the controlled 4/4 loss.

8. Recorded a tooling challenge. Kompile selected-test builds reused a stale same-coordinate Maven artifact despite dirty source/build script, hiding sendControlMessage until the repository cache was moved aside. Exact evidence and workaround: stale same-version artifact challenge.

Relevant PRs / refs

| Item | Write-time state (2026-07-17 UTC) | |---|---| | UrlProtocol refresh PR | OPEN, DIRTY; all three checks succeeded, but main advanced with protocol 0.0.368 and the branch now conflicts in build.kts. Head still 7e316748997718e13486406f7e4721c071cd624c. This API was published as 0.0.368/0.0.369 but did not fix the stress reproducer. | | Blocked UrlResolver PR | OPEN, BEHIND; GitHub build and kotlin.build remote checks both failed. Head 0f633dde336d8dd2193f901841e68c5c9bfc1771. It is not the D2 experiment but must remain visible to the next investigator. | | UrlResolver D2 evidence branch | Pushed, no PR. Head e1e34f45; contains the complete investigation-log.md plus temporary resolver callback traces. Checkout: /code/workspace/urlresolver-announce-loss. | | UrlProtocol D2 experiment branch | Pushed, no PR. Head edda455; contains server diagnostics, the fail-first control-lane test, and the bounded but consumer-falsified experiment. Checkout: /code/workspace/urlresolver-announce-loss-urlprotocol. | | foundation.url:protocol:0.0.369 | Published refresh API artifact from the prior session. | | foundation.url:protocol:0.0.370 | Not published. Coordinate exists only on the experiment branch/local cache. | | Protocol main at rebase time | 2acce11 (Release protocol 0.0.368 (transport admission-failure surfacing)). |

There were no uncommitted source changes at handoff creation: both investigation trees were committed and pushed to their dedicated branches. No code PR was opened for the falsified experiment.

Next steps

1. Re-verify current state first with the PR commands in the banner, then fetch both investigation branches and confirm the heads above.

2. Instrument the client-side relay registration request, not only server ACK:

   • one correlation ID from registerWithRelay start;
   • stream creation/negotiation completion;
   • request-frame transport admission result and root connection identity;
   • server receive/apply/ACK;
   • client ACK read or stream close;
   • next retry schedule.

   The important unresolved population is the missing providers whose service-bearing registration never reached server apply. Distinguish request admission rejection, stream negotiation starvation, cancellation, and connection replacement. Do not add retries/timeouts as a fix.

3. Instrument the confirmed PEX request and response independently at both ends:

   • request and response admission result;
   • normal parked-permit occupancy;
   • event-loop versus worker context;
   • physical connection/substream identity;
   • apply-before-ACK marker;
   • exact continuation/retry scheduling delay.

   PEX is the long-lived convergence mechanism, and the ACK-control experiment left the 4/4 consumer failure rate unchanged.

4. Reproduce the pre-assertion third-party withdrawal before acting on it. Enable both protocol and resolver debug, tag the registered stream sender, and stop analysis at [CREG] (exclude teardown). If senderPeerId != providerPeerId correlates with a missing service before the assertion, write a real upstream authorization test and fix RelayService.processIncomingMessage at the owning layer. Keep this as a separate defect unless it explains the main missing set.

5. Any proposed fix must:

   • have a reliably failing end-to-end or upstream reproducer before implementation;
   • preserve the 16-write OOM bound or account explicitly for any bounded increase;
   • leave the consumer test unchanged;
   • pass at least seven identical amplified runs (prefer two four-run batches);
   • then publish upstream, bump UrlResolver, open unmerged PRs, adversarially review, and wait for green CI.

6. Do not merge any code PR without explicit user approval. The handoff-document PR is the only standing exception and was merged as part of this handoff workflow.

Reusable / operational knowledge

• Exact amplifier launch shape: compile the existing test once, then run four child JVMs concurrently, each taskset -c 0,1, -Xmx128m, with a 190-second outer guard matching the existing 180-second test timeout. Do not run four sequentially; shared two-CPU contention is the calibrated reproducer.
• A single process or unconstrained CPU run can pass and is not evidence. Controlled baseline is 4/4 failure on both old and current stacks.
• scripts/test.bash --test <name> still compiles every test script before selecting one; expect several minutes and very large warning output.
• Kompile's repository cache may ignore dirty same-coordinate sources. Preserved local cache snapshots: /tmp/urlprotocol-cache-backup.IAlfa2/repository-cache and /tmp/urlprotocol-cache-pre-green.Fz4h00/repository-cache. Moving the exact repository-specific cache aside (rather than deleting it) forced a source-current build.
• Diagnostic logs from this session:
  • /tmp/creg-trace-batch-2/run-{1..4}.log
  • /tmp/creg-trace-batch-3/run-{1..4}.log
  • /tmp/creg-withdraw-sender.log
  • /tmp/creg-ack-failure.log
  • /tmp/creg-control-fix-batch-1/run-{1..4}.log
• Before any build, fetch and rebase on origin/main. Before updating an existing PR branch, first verify that its PR is still open. The refresh branch currently needs a conflict resolution because main now owns the 0.0.368 release entry.