← Priority list

UrlProtocol peer address propagation: a newly-registered peer's direct address never reaches the resolving peer's registry (two hermetic reproducers ready; prior fix attempt broken, caused two prod outages)

Re-homed from the HardwareControlFabric onboarding handoff because the owning layer is UrlProtocol, not HCF. When peer A resolves a freshly-registered peer B's peer-scoped alias, A's candidate set for B contains only /p2p-circuit/ relay routes and B's reachable direct address is never learned, so every dial fails. Two hermetic in-process UrlProtocol2 reproducers already exist on UrlResolver branch reproduce-hcf-circuit-only-onboarding (head 83984307). The previous fix attempt was red on CI in 3 of 5 repos and caused two production outages, including a ~40 minute one at droplet load 119; all 5 PRs are now closed and unmerged, branches intact. Hard guardrail: code plus hermetic tests only, no production deploys, a human drives any prod touch, and any fix must ship with comprehensive end-to-end tests of the propagation path.

hf-2026-07-26-urlprotocol-peer-address-propagation-a-newly-registered-peer-s-direct-address-never-reaches-the-resolving-peer-s-registry-two-hermetic-reproducers-ready-prior-fix-attempt-broken-caused-two-prod-outages

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlProtocol peer address propagation ? a newly-registered peer's direct address never reaches the resolving peer's registry (two hermetic reproducers ready; prior fix attempt was broken and caused two production outages)

UPDATE 2026-07-26 02:20 UTC — reconciliation: a review-complete fix pair ALREADY EXISTS; do not re-derive from scratch

This handoff was written about the OLDER broken fix attempt (UrlResolver #758 + HCF pin bumps). It does not mention that a later, independent, review-complete implementation of exactly the prescribed shape (protocol-half in UrlProtocol, consumer-half in UrlResolver) already exists:

  • https://github.com/CodexCoder21Organization/UrlProtocol/pull/400 — bounded peer exchange may return the authenticated responder itself (protocol half).
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/827 — propagate authenticated peer addresses during inbound RPC (resolver half; includes 4 hermetic tests incl. rejection cases).

Both passed supervisory final review on 2026-07-22 (tracker: url://handoff/handoffs/hf-2026-07-22-hcf-peer-scoped-propagation-fix-ci-blocked). As of 2026-07-26 they are being rebased onto moved mains and driven to merge by Opus lanes with user-granted merge authority; #827's one CI-only test failure is being root-caused with a reliable-reproduction requirement before any fix. The next reader should treat this pair as the fix of record and drive/review it rather than re-deriving. The hermetic reproducers described below remain valuable as acceptance tests for the pair. The no-production-deploy guardrail below still stands: nothing deploys to prod without explicit user authorization.

Written: 2026-07-26 (~00:30 UTC). Re-homed out of url://handoff/handoffs/hf-2026-07-11-hcf-onboarding-address-propagation-fix-and-prod-incident, which was scoped to HardwareControlFabric onboarding. The owning layer is almost certainly UrlProtocol (with UrlResolver as the consumer), so the work gets its own record here.

RE-VERIFY BEFORE ACTING

Every state claim below is a write-time snapshot and goes stale. Branches get force-pushed, CI re-runs, artifacts get published, and hosts restart. Before acting on anything here, re-verify:

  • Branches: git ls-remote https://github.com/CodexCoder21Organization/<repo>.git refs/heads/<branch>
  • PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • Check runs on a head SHA: gh api repos/CodexCoder21Organization/<repo>/commits/<sha>/check-runs
  • Production HCF: curl -sS -o /dev/null -w "%{http_code}\n" https://hardware-control-fabric-wui.wasmserver.com/

A fresh query always wins over this document.

Mission summary

Fix the address-propagation defect in the UrlProtocol/UrlResolver peer stack:

When peer A resolves a peer-scoped alias belonging to a freshly-registered peer B, A's candidate address set for B contains only /p2p-circuit/ relay routes. B's own reachable direct listen address is never a candidate, because it never propagates into A's registry. Since the relays hold no circuit-v2 reservation for B, every dial fails and the resolution fails.

This is a propagation defect, not a dial-ordering defect: you cannot reorder onto a direct address that was never learned. Five prior dial-side changes (UrlResolver 0.0.622 / 0.0.625 / 0.0.628 / 0.0.630 plus an on-disk dial diff) all failed for this reason.

The first observed victim was HardwareControlFabric new-daemon onboarding via --register-with (the completeRegistration ping), which fails with completeRegistration ? 'url://hardware.control.fabric.daemon.<peerId>/' did not respond to ping. That is a symptom, not the scope ? the scope is the UrlProtocol layer that is supposed to make B's direct/observed address known to A.

What was found (verified evidence, not speculation)

  1. Root cause proven by a live PEER-DIAL-TRACE on the production droplet. Every candidate A held for B logged containsDropletDirect=false; each dial threw RelayException ? not reachable / io.libp2p.etc.types.NothingToCompleteException. B's direct address (/ip4/198.199.106.165/tcp/<p2pPort>/p2p/<peerId>) was open and directly connectable from A on the same host, yet was never in the candidate set.

  2. Where the missing half most likely belongs (hypothesis worth pursuing first). UrlProtocol main already carries adjacent routing work: #348 Prefer direct mesh paths before relay fallback, #343 link-state relay routing, #349 relay ACK routing race. #348 is a no-op when the direct path was never learned. The missing half is one (or more) of:

    • propagating B's authenticated direct/observed listen addresses into A's registry from the inbound RPC B makes to A (identify observed-addr, or an explicit announce at registration),
    • propagating them via peer exchange (PEX),
    • or making a NATed B actually hold a real circuit-v2 reservation so the relay routes work.

  3. Peer-scoped aliases can also resolve to the wrong provider. A second reproducer shows a client that knows the legacy parent service hardware.control.fabric.daemon will satisfy a peer-scoped child alias hardware.control.fabric.daemon.<peerId> from that parent. That is a separate correctness bug in peer-scoped resolution and it masks/complicates the propagation bug.

The two hermetic reproducers (carried over ? this is the most valuable asset here)

They live on the surviving UrlResolver branch reproduce-hcf-circuit-only-onboarding, remote head 839843074a47a8f00d81aac475619d6ae483ae58 (verified present 2026-07-26): https://github.com/CodexCoder21Organization/UrlResolver/tree/reproduce-hcf-circuit-only-onboarding

| File | What it demonstrates | |---|---| | tests/testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts | The primary reproducer. Builds three real in-process UrlProtocol2 peers ? relay R, manager A, daemon B. A is seeded so it knows B only through an unreserved relay-circuit address via R (exactly the production precondition: no direct candidate), while B is a real same-host peer with a bound direct listen address. B then calls A's registration service. The assertion is that the ping inside completeRegistration must see at least one non-circuit direct candidate ? i.e. A must have learned B's authenticated direct listen address from that inbound RPC. On unfixed code A only has the unreserved circuit through R and the in-handler ping fails. @Timeout(90). | | tests/testPeerScopedResolutionIgnoresLegacyParentService.kts | Registers the legacy parent service url://hardware.control.fabric.daemon/ on one peer, makes a client discover it, then resolves the peer-scoped child alias hardware.control.fabric.daemon.<peerId>. Asserts the child alias is not satisfied by the parent provider. @Timeout(45). |

A third, related reproducer is already on UrlResolver main and is a good starting point for anyone building on this: tests/testPeerScopedAliasCircuitOnlyUnreservedRelayReproducesNothingToComplete.kts.

How to run them (hermetic; no droplet, no network services, no deployment):

gh repo clone CodexCoder21Organization/UrlResolver workspace/UrlResolver
cd workspace/UrlResolver
git fetch origin reproduce-hcf-circuit-only-onboarding
git checkout reproduce-hcf-circuit-only-onboarding
scripts/test.bash tests/testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts
scripts/test.bash tests/testPeerScopedResolutionIgnoresLegacyParentService.kts

These tests are in-process UrlProtocol2 instances with real libp2p transports and no mocks. They are the reason this handoff is worth picking up: the hard part (a hermetic reproduction of a defect previously only observed in production) is already done.

Note the branch's @file:WithArtifact pins are old (foundation.url:protocol:0.0.337, jvm-libp2p:1.3.0-codexcoder21-snapshot-10). Rebase onto current main and re-pin before drawing conclusions from a red run ? a failure caused by a stale pin is not the defect.

History of the prior fix attempt ? it was broken, and it took production down twice

This is recorded in full because the next person must not repeat it.

  1. The fix did not work: CI was red, verified. codex (gpt-5.5) wrote an "inbound registration RPC seeds the daemon's direct addresses into the resolver" fix and pushed it across five repos. Three of the five had failing CI. Re-verified 2026-07-26 against the recorded head SHAs:

    | Repo / PR | Branch @ head | Check-run conclusions on that head | |---|---|---| | https://github.com/CodexCoder21Organization/UrlResolver/pull/758 | reproduce-hcf-circuit-only-onboarding @ 83984307 | bld-build: failure, kotlin.build (remote): failure | | https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/113 | bump-urlresolver-0.0.630 @ 831e2c97 | kotlin.build (remote): failure | | https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer/pull/7 | bump-urlresolver-0.0.630 @ 6c454c5b | kotlin.build (remote): failure | | https://github.com/CodexCoder21Organization/HardwareControlFabricApi/pull/38 | bump-urlresolver-0.0.630 @ 5aefbeee | passed | | https://github.com/CodexCoder21Organization/HardwareControlFabricEmbedded/pull/6 | bump-urlresolver-0.0.630 @ 0ab02a0e | passed |

    All five PRs are now CLOSED and none was ever merged (verified 2026-07-26; UrlResolver #758 closed 2026-07-17, the four HCF PRs closed 2026-07-22). All five branches still exist at exactly those head SHAs, so the diff is fully recoverable ? treat it as a broken starting point, not a near-miss.

  2. Production outage #1 (codex run 9). A prod deploy of unverified code.

  3. Production outage #2 (~40 minutes, 2026-07-12). codex ran an unconstrained live end-to-end test: it deployed its broken ServiceServer into ContainerNursery (route url:hardware-control-fabric:) plus a fix daemon onto the shared droplet, and fired all five repos' CI simultaneously. The 8-vCPU droplet hit load average 119 and the WUI went down with 500 ? All 0 peers for service 'hardware-control-fabric'. Recovery required restoring the known-good ServiceServer jar; restoring the daemon alone does not clear "0 peers" ? the ServiceServer jar is what fixes it.

Hard guardrail for this workstream

Code plus hermetic tests only. No production deploys as part of this work ? none.

  • Do not deploy any jar to ContainerNursery or to the droplet at 198.199.106.165 as part of developing or validating this fix.
  • Do not run an unconstrained live end-to-end test against the shared droplet. It is shared infrastructure and it has been taken down twice by exactly that.
  • Do not fire multiple repos' CI simultaneously ? it overloads the shared buildtest droplet. One heavy build at a time.
  • A human drives any production touch, after CI is genuinely green and the diff has been reviewed. Landing in main is the end of an agent's scope on this.
  • Never merge any of the five recorded PR branches without explicit user approval.

Any future fix must ship with comprehensive end-to-end tests

This is a requirement, not a preference. This code path has taken production down twice and has been repeatedly under-tested ? five prior "fixes" shipped against it without a test that actually exercised the failing path. A fix is not acceptable without tests that demonstrably cover:

  • The propagation path itself ? A learns B's direct address from B's inbound RPC / identify / PEX, asserted as "A's candidate set for B contains at least one non-circuit direct address", not merely "the ping succeeded".
  • The failing-before / passing-after property ? show the new test failing on unfixed code and passing on fixed code. A test that passes on both proves nothing.
  • Peer-scoped alias resolution ? a child alias <service>.<peerId> must never be satisfied by the parent service <service>, and must resolve to the correct peer when several peers advertise sibling aliases.
  • Relay-only preconditions ? B reachable only via an unreserved circuit, via a reserved circuit, and via a direct address; and the transitions between those states.
  • Re-registration and restart ? B restarting with a new listen port must not leave A pinned to a stale direct address.
  • Concurrency ? several peers registering against A at once; propagation must not depend on ordering luck.

Testing rules that apply (per the org's testing guidance): real in-process instances, no mocks and never mockito; fakes only where a real instance is genuinely impossible; assert the full text of error and warning messages, not fragments; warnings surface as algebraic effects (toss), not println; anything that measures or waits on time takes a Clock from community.kotlin.clocks.simple (default SystemClock, tests drive ManualClock). Note the known trap: Clock.schedule takes an absolute deadline, not an interval ? passing an interval spins the loop, and ManualClock(0) hides it, so construct ManualClock with a realistic epoch.

Relevant PRs / refs

Surviving branches (all verified present 2026-07-26 at these exact remote SHAs; nothing exists only on a local disk):

| Repo | Branch | Remote head | What is on it | |---|---|---|---| | UrlResolver | reproduce-hcf-circuit-only-onboarding | 839843074a47a8f00d81aac475619d6ae483ae58 | The two reproducers + the broken propagation fix. CI red on this head. | | HardwareControlFabricApi | bump-urlresolver-0.0.630 | 5aefbeeebabc861ad2b13fdfade1c4af5a6539ff | Pin bump. CI passed. | | HardwareControlFabricDaemon | bump-urlresolver-0.0.630 | 831e2c97ce3bbfaf4345ac3723cfc964cd81f8f9 | Pin bump. CI red. | | HardwareControlFabricEmbedded | bump-urlresolver-0.0.630 | 0ab02a0ecff6bc1c1af3d866de71796ea61481c5 | Pin bump. CI passed. | | HardwareControlFabricServiceServer | bump-urlresolver-0.0.630 | 6c454c5b2db3f65ea4e9475f953c97cb07587318 | Pin bump. CI red. |

Closed PRs (none merged): UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/758 ? HCFApi https://github.com/CodexCoder21Organization/HardwareControlFabricApi/pull/38 ? HCFDaemon https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/113 ? HCFEmbedded https://github.com/CodexCoder21Organization/HardwareControlFabricEmbedded/pull/6 ? HCFServiceServer https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer/pull/7

Adjacent UrlProtocol work to read before starting: #348 prefer-direct-before-relay, #343 link-state relay routing, #349 relay ACK routing race.

Nothing is unpushed. No local-only work exists for this effort; the working clones from the originating session were verified clean against the pushed heads above.

Next steps (ordered)

  1. Re-verify the branches and PR states in the table above before touching anything.
  2. Rebase the reproducer branch onto current UrlResolver main and refresh its stale @file:WithArtifact pins (foundation.url:protocol, jvm-libp2p) to whatever main uses. Confirm the two reproducers still fail for the right reason ? read the failure message, do not accept a red run caused by a stale pin as evidence of the defect.
  3. Discard the broken fix diff and re-derive it in UrlProtocol. Decide explicitly which mechanism carries B's direct address to A (inbound-RPC seeding, identify observed-addr, announce at registration, PEX, or a real circuit-v2 reservation) and write the mechanism down in one or two sentences before writing code.
  4. Write the comprehensive test suite described above alongside the fix, with failing-before / passing-after demonstrated for each new test.
  5. Drive CI genuinely green, one repo at a time, entirely off production. Diagnose the recorded red runs first ? start from the run logs on the three red head SHAs.
  6. Stop at merged-to-main. Publishing artifacts, bumping downstream pins, and any droplet or ContainerNursery deploy are a separate, human-driven step.

Reusable / operational knowledge

  • All 0 peers for service 'hardware-control-fabric' (HTTP 500 on the WUI) means the deployed ServiceServer jar is bad, or the daemon is not running. Restarting the daemon alone does not clear it when the ServiceServer is the bad artifact.
  • The HCF daemon has no watchdog ? no systemd unit, no auto-restart. If it dies, the WUI 500s until a human restarts it.
  • Daemon restart, canonical form: kill by filtering pgrep -x java plus /proc/$p/cmdline for /root/.HardwareControlFabric/daemon.jar ? never pkill -f daemon.jar, which matches the ssh argv and kills your own session ? then relaunch with -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/.HardwareControlFabric/logs/ under setsid nohup ... </dev/null.
  • Droplet: 8 vCPU / 16 GB at 198.199.106.165 port 23. It is shared with buildtest CI; heavy parallel builds starve it.
  • gh pr edit is broken in this org (Projects-classic GraphQL deprecation) ? use gh api -X PATCH repos/<owner>/<repo>/pulls/<n> -F body=@file.
  • gh pr checks --json is unsupported here. Use gh pr view <n> --json statusCheckRollup or gh api repos/<o>/<r>/commits/<sha>/check-runs.
  • /usr/local/bin/coursier is the wrong architecture on these hosts (Exec format error); launch the JVM jar instead: java -jar ~/bin/coursier launch ....
  • kompile gotchas: bump the build.kts version literal after source edits or tests silently resolve the previously published jar; kotlin.directory artifacts are immutable (a republish 409s even with --force, so bump the patch); purge ~/.cache/coursier/v1/bldbinary/... after same-version source edits.
  • The PEX no-backoff busy-loop that used to accompany this investigation is fixed. The daemon log spam [PEX-EMPTY] ... re-pull of configured bootstrap ... dial-failed repeating every few milliseconds was an unthrottled re-pull of an unreachable configured bootstrap in UrlResolver (src/foundation/url/resolver/UrlResolver.kt). UrlResolver main now has both an inner bounded exponential backoff (CONFIGURED_BOOTSTRAP_REPULL_BASE_BACKOFF_MS 250 ms ? CONFIGURED_BOOTSTRAP_REPULL_MAX_BACKOFF_MS 2 s, within a CONFIGURED_BOOTSTRAP_REPULL_BUDGET_MS 15 s budget) and a shared per-peer schedule across all service-resync jobs (CONFIGURED_BOOTSTRAP_PEER_BACKOFF_INITIAL_MS 250 ms ? CONFIGURED_BOOTSTRAP_PEER_BACKOFF_MAX_MS 30 s), driven by the injected Clock and reset on a received response. Hermetic coverage: tests/testDeadConfiguredBootstrapRepullsUseSharedPerPeerBackoff.kts (8 services against a dead bootstrap; asserts at most 3 completed re-pulls in an 18 s window). Landed in https://github.com/CodexCoder21Organization/UrlResolver/pull/835 (resolver 0.0.661, merged 2026-07-25). Anything still emitting that spam is running a resolver older than 0.0.661 ? the remedy is a pin bump and redeploy, which is a human-driven production step, not a code fix.