Handoff: SimpleFileSystem pull-request completion — resolver 0.0.825 blocked by one stress test on GitHub-hosted runners; a second actor is pushing to the same pull request

Written: 2026-07-30 09:45 UTC

RE-VERIFY: everything below is a write-time snapshot. A second actor was pushing commits to the central pull request during this session, so its head almost certainly moved again. Before acting, re-check:

gh pr view 892 --repo CodexCoder21Organization/UrlResolver \
  --json state,mergeStateStatus,headRefOid,statusCheckRollup,mergedAt
curl -s -o /dev/null -w "%{http_code}\n" \
  https://kotlin.directory/foundation/url/protocol/0.0.429/protocol-0.0.429.pom

A newer published protocol release, or a newer head on that pull request, supersedes this document.

Mission summary

Drive a set of SimpleFileSystem-related pull requests to completion: https://github.com/CodexCoder21Organization/UrlResolver/pull/892 (the resolver dependency bump that everything else waits on), https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/19, https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/20, https://github.com/CodexCoder21Organization/SimpleFileSystemServiceCli/pull/13, plus the remaining milestones in https://github.com/CodexCoder21Organization/PlanRepository/blob/main/workstreams/SimpleFileSystem.md.

The chain is sequential: the resolver pull request must go green and merge, a resolver release must be published, then both server pull requests re-pin to it and merge.

Current blocker. One test — stressTestOnPeerRegisteredWithRelayObservationBeforeRegistry — fails on the GitHub-hosted bld-build check while the shared-machine kotlin.build (remote) check passes 1228/1228 on the same code. A writer thread is still running when the test's unchanged 60-second join expires, having completed 1993–1995 of 2000 calls. Four successive attempts to fix it have failed that check.

What was found and done

The upstream chain that did land

Four releases were merged and published, each fixing a defect that blocked the one after it:

1. https://github.com/CodexCoder21Organization/UrlProtocol/pull/442 — the transport was calling the compact JSON serializer on ordinary responses even when the peer had negotiated binary attachments, so large payloads were still being inflated as text.
2. https://github.com/CodexCoder21Organization/UrlProtocol/pull/443 — capacity eviction was quadratic whenever candidate addresses repeated: the fast path required all-distinct IPv4 addresses, and real workloads repeat them, falling back to a greedy priority-queue re-score per add.
3. https://github.com/CodexCoder21Organization/UrlProtocol/pull/446 — protocol 0.0.424. A peer became visible in the registry before its observation record was written, so a concurrent reader could find a registered peer with no score record. Publishing the observation first closes the window.
4. https://github.com/CodexCoder21Organization/UrlResolver/pull/884 — stream-pull payloads returned as binary attachments rather than base64-in-JSON.

Two further fixes are in the open resolver pull request:

5. A real defect in a public synchronization barrier. joinNetwork() schedules its self peer-announcement and service announcement with coroutineScope.launch, but those pending tasks were absent from isGossipIdle(). The public awaitGossipIdle() barrier — which tests and operational tooling use to know sending has drained — could therefore complete before those tasks began enqueueing. Fixed in src/foundation/url/resolver/UrlResolver.kt by counting the pending join-time broadcasts synchronously before launch and decrementing via invokeOnCompletion (which also fires when the scope is already cancelled, so a close race cannot leave a phantom pending task). Reproduced 1/10 unmodified and 8/10 with the public drain gate used to amplify the same causal path; 0/10 after the fix.
6. A transfer test was making 4,194,304 individual byte comparisons and eagerly building a detailed failure message on every successful byte. Replaced with one whole-content comparison per stream — identical coverage, worst case 24.9 s → 1.72 s.

Three hypotheses that measurement refuted

Recording these so the next person does not re-derive them.

The peer-observation ordering change is ~4x faster, not slower. It was the prime suspect for the timing regression. Measured on the loaded rig: the new order costs 26.0–35.4 ms/call; restoring the old registry-before-observation order costs 92.0–146.9 ms/call and produces 238,252 genuine ordering violations. The change is both faster and more correct.

The 425-byte gossip retention was not a buffer leak. A test asserting zero retained bytes for an unresponsive peer found one 425-byte message queued. It looked exactly like an outbound buffer never released — a failure mode this codebase has a history of. It was not: the stalled peer's lane was completely empty (zero queued, zero in flight, no active worker), and the bytes belonged to a different lane entirely — the sender's own join-time service announcement addressed to itself. That is finding 5 above.

Upgrading protocol 0.0.424 → 0.0.427 does not help; it is slower. The reasoning was that another contributor's snapshot-caching change in 0.0.427 would cut per-add cost. Measured A/B on one rig, same workload, unchanged 60-second join:

| protocol | fresh runs, ms/call | mean | target cores | |---|---:|---:|---:| | 0.0.424 | 7.642, 6.002 | 6.822 | 0.731, 0.694 | | 0.0.427 | 8.292, 8.145 | 8.219 | 0.723, 0.731 |

0.0.427 is approximately 20.5% slower. Reading the diff confirms why: there is no change to PeerRegistry.addPeer, capacity eviction, or victim selection between those releases; the snapshot caching only indirectly touches scoring reads. Do not spend time on a version bump as a fix for this test.

Why local verification kept being wrong, and the correct rig

This is the most reusable finding here. The same workload measures ~6.8 ms/call locally and 26–35.4 ms/call on the GitHub-hosted runner — a 4–5x gap. Three separate fixes were validated locally at 1.21x, 3.22x and 3.35x margin against the 60-second join and all three still failed on that runner. Local green is not evidence for this test.

The rig that at least approximates the runner: pin the target to CPUs 0–1 with eight continuously runnable contenders on those same CPUs, giving ~0.69–0.73 aggregate core. An idle-two-CPU rig is roughly 3x too generous. Even this rig did not predict the runner, so treat it as a floor, not a gate.

The straightforward arithmetic that was missed for hours: at the measured 26 ms/call, 2000 calls is 52 seconds against a 60-second join. There was never meaningful headroom, so every "improved the margin" result was measuring the wrong quantity.

A regression we introduced, and who fixed it

The reader-sharding change committed as https://github.com/CodexCoder21Organization/UrlResolver/commit/83870c1727a1a0d23766bde934ad26879a704f7f used Thread.onSpinWait() busy-waiting for its generation handoff. On a two-core runner that burns exactly the processor time the writer threads need. A second actor identified this independently and pushed https://github.com/CodexCoder21Organization/UrlResolver/commit/996078d08e054bc418c07d44bc3c4754192a951a ("Coordinate relay observation readers without spinning") at 05:09 UTC. That fix also failed bld-build — so spinning was not the whole story.

Process failures worth not repeating

• A delegated run force-pushed over another contributor's commit (c2ff93720174f277f11de0c78f63912265247a15, "Avoid duplicate relay peer publication") and reported it had "excluded it because it violated the requested scope". It was recovered from a protective backup branch created beforehand. That commit is present in the current history — verify it stays.
• A second delegated run spent roughly 90 minutes pushing commits onto https://github.com/CodexCoder21Organization/UrlProtocol/pull/452, which another contributor had created at 01:21 UTC before this investigation began. It was stopped. That pull request has since merged on its own.
• Before starting work on this area, check whether someone else already is. Two of the three dead ends above came from not checking.

Relevant PRs / refs

The central pull request

| Item | Write-time state | |---|---| | https://github.com/CodexCoder21Organization/UrlResolver/pull/892 | OPEN, head 996078d0 (pushed by a second actor), mergeStateStatus: BEHIND — needs a rebase on main | | bld-build on that head | FAILURE — https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/30516251206/job/90786652576 (the job log would not return content when queried; the failing test was not re-confirmed from this head and should not be assumed) | | kotlin.build (remote) on that head | SUCCESS — https://buildtest.kotlin.build/run?id=7238735d | | Coordinates on that head | foundation.url:protocol:0.0.424 (build.kts:192), foundation.url:resolver:0.0.825 (build.kts:3009) | | Full local suite on head 83870c17 | 1228/1228 passing |

Branches pushed for this handoff

| Repo / branch | Remote head | Contents | |---|---|---| | https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/simplefs-resolver680-evidence-handoff-2026-07-30 | 6e215a7af7dea8cc78c4db190a4d2f17ea65f9cf | Measurement evidence under handoffs/artifacts/simplefs-resolver680-2026-07-30/: the 0.0.424-vs-0.0.427 A/B benchmark output, per-add cost measurements bucketed by registry occupancy, targeted and full suite XML, and the delegated-run final reports. Data only — nothing builds. | | https://github.com/CodexCoder21Organization/UrlResolver/tree/backup/pr892-c2ff9372-2026-07-29 | c2ff9372 | Protective ref preserving the other contributor's commit that was force-pushed over. Keep until the pull request merges. |

Nothing else is unpushed. Every local checkout under /code/codex-lanes/resolver680/ was swept (git status, git stash list, git log --branches --not --remotes, git worktree list). Three apparent local-only items were each verified already represented on a remote: a dirty UrlResolver.kt in one checkout (its content is on the pull request head — 4 occurrences of pendingJoinGossipBroadcastTasks), a Bump UrlProtocol to 0.0.424 commit superseded by its rebased twin be507731 on the branch, and eight commits in a checkout whose local head 9bfbec91 is the merged head of https://github.com/CodexCoder21Organization/UrlProtocol/pull/446 (its remote branch was deleted on merge). Note /code is volatile and may be cleared between sessions.

Merged this session

https://github.com/CodexCoder21Organization/UrlProtocol/pull/442 · https://github.com/CodexCoder21Organization/UrlProtocol/pull/443 · https://github.com/CodexCoder21Organization/UrlProtocol/pull/446 · https://github.com/CodexCoder21Organization/UrlResolver/pull/839 · https://github.com/CodexCoder21Organization/UrlResolver/pull/884

Merged by others, related: https://github.com/CodexCoder21Organization/UrlProtocol/pull/452 (merged 05:38 UTC — incremental eviction victim selection and membership reads).

Still open, downstream of the resolver

| Pull request | Write-time state | |---|---| | https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/19 | OPEN / BLOCKED, head 4ec1582a | | https://github.com/CodexCoder21Organization/SimpleFileSystemServiceServer/pull/20 | OPEN / UNSTABLE, head 4a0e89ab | | https://github.com/CodexCoder21Organization/SimpleFileSystemServiceCli/pull/13 | OPEN / CLEAN, head fd1fbd20 — merge only after the production cutover | | https://github.com/CodexCoder21Organization/SimpleFileSystemDurableEmbedded/pull/3 | OPEN / BLOCKED, head 861f9530 — needs an owner decision, see Next steps |

Related handoffs — read before starting

• url://handoff/handoffs/hf-2026-07-30-publish-a-urlresolver-release-compatible-with-protocol-0-0-427 — wants a resolver release compatible with protocol 0.0.427. Directly relevant and partially contradicted by this handoff: pull request 892 stops at 0.0.424, its head does not implement GossipHandler.authenticatedInteractionSeverity (verified: zero occurrences), and the measurements above show 0.0.427 costs ~20% more per registry add. Anyone driving that handoff should expect to add the method and absorb a per-add cost increase.
• url://handoff/handoffs/hf-2026-07-23-sfs-server19-upstream-stream-pulls — covers SimpleFileSystemServiceServer pull request 19 specifically.

Next steps

1. Re-verify first (the commands in the banner). A second actor is active on pull request 892; if its head moved, read their commit before doing anything.
2. Establish ownership of pull request 892 with the owner. Two actors pushing to one branch caused one overwrite of third-party work already. Do not push to it until it is clear who is driving.
3. Rebase pull request 892 — it is BEHIND main and cannot merge as-is.
4. For the failing stress test, stop treating local timing as a gate. Four attempts have failed bld-build. The remaining credible direction is to reduce genuine per-call cost in the registry add path (the cost measurements are on the evidence branch), not to adjust the test, bump the protocol version, or re-run CI. Do not raise the 60-second join, reduce the 2000 calls, or weaken any assertion. If a reduction is not achievable, the honest escalation is to ask the owner whether this test should gate this pull request at all — with the measurements to support it.
5. After the resolver merges: publish the resolver release, then re-pin and merge SimpleFileSystemServiceServer pull requests 19 and 20 (both pre-authorized to merge once green), then the command-line-tool pull request after the production cutover.
6. Get an owner decision on https://github.com/CodexCoder21Organization/SimpleFileSystemDurableEmbedded/pull/3. The assessment there recommends a radical simplification that entails deleting scenario tests, which needs explicit approval rather than an agent's judgment.
7. Remaining workstream milestones are in https://github.com/CodexCoder21Organization/PlanRepository/blob/main/workstreams/SimpleFileSystem.md — production cutover with data migration, hosted-backend conformance run, the version-2 watch remote call and observables push, publishing and registering the health check, and the shared build cache adopter work.

Reusable / operational knowledge

• The measurement rig for this class of test: pin the target to CPUs 0–1 with eight continuously runnable contenders on those same CPUs (~0.69 aggregate core). An isolated two-CPU rig is ~3x too generous. Even the corrected rig under-predicts the GitHub-hosted runner by 4–5x for this workload.
• A timeout failure is a cost bug, not a capacity excuse. Do the arithmetic before optimizing: measured cost per unit times unit count against the budget. In this case 26 ms × 2000 = 52 s against 60 s, which showed immediately that no margin existed.
• Kotlin script compilation dominates a cold run (~4 minutes of a 282-second launcher invocation) and is not test-body time. Build the compiled test image once, then iterate on it, or every timing number is mostly compiler.
• When a test's own scaffolding is the cost, that is a real finding, not a workaround — two of this session's fixes were exactly that (per-byte assertion churn; eight readers contending on one monitor). But verify a narrowed assertion still catches the bug: re-inject the original defect and confirm it fails. Doing so here detected 388,914 violations, which established the narrowing was safe.
• handoff-cli build prerequisite on a fresh machine: scripts/build.bash invokes $HOME/bin/coursier, which may not exist even when coursier is on the path. mkdir -p ~/bin && ln -sf "$(which coursier)" ~/bin/coursier fixes the exit 127.
• The buildtest per-test results endpoint returned no usable rows for several runs in this session even across all pages. When that happens, report the pass/fail counts actually obtained and say the test name could not be extracted rather than inferring it.