Handoff: Land the UrlResolver extreme-condition bug fixes — clear the two merge-queue flakes, then merge the reviewed PRs and finish the rest

Written: 2026-10-14 13:36 UTC (effort began 2026-10-13).

⚠ RE-VERIFY BEFORE ACTING. Every state claim below is a snapshot from the moment of writing. PRs merge, queues shuffle, CI re-runs, branches get force-pushed. Re-check each PR with gh pr view <url> --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid and each branch with git ls-remote <repo> refs/heads/<branch>. A fresh query always wins over this document.

Mission summary

The user asked, verbatim:

Consider https://github.com/CodexCoder21Organization/UrlResolver which recently has had many bugs related lack of testing behaviors at scale as per https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md#extreme-condition-testing-load-fuzzing-failure-injection-and-chaos - Ultrathink are the tests comprehensive? Your goal is to add end-to-end integration tests which identify bugs (the tests should initially fail), and then fix the bug in the implementation. Go through like 5 rounds of bugs and fixes. You are approved to merge after you are confident in your changes and have an opus subagent do an adversarial review.

Merge bar used, and still in force: the final head is green in CI, including kotlin.build (remote); an independent adversarial review of that exact head found nothing, not even nits; and the orchestrator's own final review is clean. Any head change voids the review.

Expansions during the session:

• Two bugs turned out to live upstream, so they got upstream PRs: UrlProtocol and the Observable library.
• One review uncovered a pre-existing bug fixed by an earlier effort's open PR, https://github.com/CodexCoder21Organization/UrlResolver/pull/1163, which was adopted.
• Merge-queue evictions were traced to two flaky tests already on main, which now need fixing before anything lands reliably.

User decisions recorded during the session:

1. For PR 1163, Option A (run to completion). When a client goes away mid-request, by disconnect or as the losing request of a direct-versus-relay race, the server frees the request thread but lets the service's code finish and discards the result. This matches main, and it reverses 1163's earlier "interrupt the losing request" contract. The server cannot tell those two cases apart.
2. For UrlProtocol PR 697, wire compatibility (orchestrator decision; the user asked about it and did not object). Every value main delivered must keep main's exact bytes. Refusal applies only to values receivers would refuse, or that recurse without bound. A pathologically deep chain built from toString() or reflection may still overflow the stack, as on main; this is documented, because bounding it would need new reflection code.

What was found and done (the chain)

Each bug below was proven by a new end-to-end test that fails on main, then fixed. Reviews were independent Opus subagents; their findings files were in the session scratchpad and are summarised here.

1. Disk cache reload ran out of memory (https://github.com/CodexCoder21Organization/UrlResolver/pull/1252).
   • Bug: DefaultUrlResolverCache read every persisted JAR into memory before evicting anything, so reopening a directory larger than the budget hit an OutOfMemoryError in the constructor.
   • Fix: a single selectRetainedEntries rule now drives expiry, LRU and load across peers, services and bytecode. Only the surviving JARs are read, and each is bounded to its measured size. A per-record failure drops only that record and is reported as PersistedBytecodeEntryDroppedWarningEffect, or once as PersistedBytecodeIndexUnreadableWarningEffect if the index itself is unreadable.
   • Reviews: six rounds. The final round found NO FINDINGS at head 948f915, including a randomized equivalence check against main over 6,000 seeds.
2. Unread uploads stayed open on the service (https://github.com/CodexCoder21Organization/UrlResolver/pull/1255).
   • Bug: a handler that returned without reading its InputStream argument left the upload (up to 2 MiB buffered) registered for the life of the connection, and forever on the relay or no-connection path.
   • Fix: once any response has arrived, the client sends __stream_close for every upload whose final push was not accepted. The service fails that stream with a descriptive "incomplete" IOException. Every release path now does the same instead of closing cleanly. Interrupts are preserved.
   • Reviews: three rounds. The final round found NO FINDINGS at b84e660, including a 4-size echo stress comparison with main showing no regression.
3. A NaN or infinity result never got a reply (https://github.com/CodexCoder21Organization/UrlResolver/pull/1251).
   • Bug: the caller waited out its full timeout.
   • Fix: results are pre-checked using the encoder's own traversal rules and cycle guard. The result's streams and observables are released, and the request fails with INTERNAL_ERROR naming the value and its path, for example rows[0].score. Any serialization failure fails only that request. The response is written through an 8 KiB buffer, byte-identical to before.
   • Reviews: six rounds. The final round found NO FINDINGS at bbfd3839.
4. A connection kept one close listener per subscription ever made (https://github.com/CodexCoder21Organization/UrlResolver/pull/1253).
   • Bug: listeners grew unbounded with subscription churn on a long-lived connection.
   • Fix: the root projection owns one listener per connection, plus an index from each connection to the (handler, token) pairs bound to it. A close does work proportional to that connection's own subscriptions, and every failure is collected (ProjectionGraceStartFailedEffect, ProjectionGraceReapCancelFailedEffect).
   • Reviews: six rounds. The final round found NO code findings at 9c90f6f0e; 121 of 121 testLiveProjection* tests passed remotely. One nit remains in the PR description (see Next steps).
5. A parameter observable started one thread per invalidation (https://github.com/CodexCoder21Organization/UrlResolver/pull/1250).
   • Bug: against a stalled server, 300 changes left 257 threads alive.
   • Progress: five review rounds converged on a design that needs a new Observable library API, below. The current PR head af980d895 has server-side per-observable delivery workers, but still has an open should-fix: a stale read after a reconcile.
   • Plan: consume the new API so request threads only mark values stale, and consumer callbacks run on per-observable workers.
6. Observable library API (https://github.com/CodexCoder21Organization/Observable/pull/105).
   • What it adds: DerivedObservable.invalidate(dispatch) marks the observable and all subscribed dependents stale on the caller's thread and runs no listener there. Notifications go through the supplied dispatcher, with at most one un-run closure per observable. Ordering is kept by per-registration claims, never waits, so there are no deadlocks.
   • Reviews: seven rounds. The final review found NO FINDINGS at 78165fe.
   • Then evicted: the merge queue evicted it because its own new test repeatedStaleMarksBeforeDispatchKeepReadsCurrentAndDeliverEveryNotification failed once ("Mark 17: nothing is delivered by the mark itself. expected:<16> but was:<17>", Actions run 38565272325). This is not yet root-caused: it could be a library defect (a listener running inside the mark) or a race in the test.
7. 64 parked long-polls starved all other requests (https://github.com/CodexCoder21Organization/UrlResolver/pull/1163, an earlier effort's PR, plus test-only https://github.com/CodexCoder21Organization/UrlResolver/pull/1254).
   • Mechanism: EffectAwareServiceHandler blocked a thread from the 64-slot inbound dispatcher while waiting. 1254 proves it fails on main; it passes on top of 1163.
   • Remaining requirements from the review: (a) Option A, run to completion; (b) bounded handler workers with a descriptive rejection, since the fix removed the only implicit bound; (c) nits.
   • Status: lane 11 implemented these and was stopped before verifying. Its work is on a wip branch, not on 1163.
8. The relay listener stalls behind 16 long-running handlers (upstream https://github.com/CodexCoder21Organization/UrlProtocol/pull/699).
   • Bug: a provider behind NAT read relayed requests on one thread that blocked waiting for one of 16 handler permits. It failed 5 of 5 runs.
   • Fix: the reader never blocks. There are at most 64 handlers, the excess gets an immediate retry-safe RelayRequestNotDispatchedException, and slots are released on every path.
   • Reviews: no blockers, 4 nits. Round-2 nit fixes were in progress on a wip branch.
   • Consumer reproducer: https://github.com/CodexCoder21Organization/UrlResolver/tree/ext2/lane8-relayed-parked-handlers-stall.
9. Unencodable responses got no reply on the default libp2p path (upstream https://github.com/CodexCoder21Organization/UrlProtocol/pull/697).
   • Scope: the guard now covers byte encoding, the receiver's validation of each frame, error data, and raw JSON trees.
   • Status: round 7 at 5b2ff83 restored main's bytes. Its sixth review was stopped mid-way. The open question is whether 3 intermittent service-set failures happen equally often on main: testTcpServerRunsEveryRequestItAdmits, stressTestAggregateRefusalThrottleBoundsScaffoldingAndDoesNotStarvePeers and testOwnedForwardingCancellationSharesCleanupFailure.
10. The first RPC open after bootstrap raced connection retirement (https://github.com/CodexCoder21Organization/UrlResolver/pull/1258).
    • Bug: a 1-second gossip-stream timeout retired the healthy shared connection.
    • Review blocker: skipping retirement on timeout means a genuinely dead connection is never replaced; the reviewer's probe failed 2 of 2.
    • Status: round-2 work, retiring only when a liveness check fails, is partial. The PR branch head moved to 69195306e during the round and is UNREVIEWED.
11. The merge-queue evictions are not caused by our PRs. Evidence from about 460 build-host runs:
    • testPruneVerificationRequiresUrlRpcProtocol: 148 failures in 518 attempts. Passing runs take 27–30 s against a 30 s limit.
    • testLargeRpcRequestSurvivesOrdinaryTrafficOnItsOwnConnection: about 27% failures since 2026-10-08.
    • Plus testEagerJoinHostCreationCompletesWithOccupiedCarrier and testLiveProjectionResiliencePollFallbackKillRestartRecovers (rare), and one build-upload failure.
    • https://github.com/CodexCoder21Organization/UrlResolver/pull/1231 (earlier effort) claims to fix the prune test; it is unverified.

Relevant PRs / refs (write-time state)

Partial-work branches (all pushed; remote SHAs verified at write time):

Nothing is deployed or published from this effort. No production state was changed.

Next steps

1. Re-verify all state above first.
2. Fix the two high-rate merge-queue flakes, using the fix-flakey-test skill: root cause, a reliable reproducer, and no timeout raises.
   • (a) testPruneVerificationRequiresUrlRpcProtocol: check whether https://github.com/CodexCoder21Organization/UrlResolver/pull/1231 actually removes the ~27 s of real waiting (an injected Clock should replace real time). Rebase it, verify, review and land it, or replace it.
   • (b) testLargeRpcRequestSurvivesOrdinaryTrafficOnItsOwnConnection: its error says "Harness fault … throttled relay failed with SocketException: Broken pipe". Verify whether the fault is in the test harness or the product.
3. Re-enqueue the three review-clean PRs: 1252, 1255 and 1251. Use the merge-queue GraphQL enqueuePullRequest mutation, or build-watchman --to-merged, once the flakes are fixed. All three have clean reviews of their current heads. If a rebase changes a head, re-review it.
4. Land 1253: rebase it, fix the PR-description nit (cite the 121/121 run on the current head; correct the "fake connections only for listener counts" sentence), re-review, then enqueue.
5. Observable 105:
   • Root-cause the queue flake. First, did invalidate(dispatch) run a listener inside the call? If so, that is a library defect.
   • Fix it, re-review, then merge.
   • Publish a new community.kotlin.observable:core-jvm version with the publish-maven-artifact skill. The README lists core 0.3.24; probe kotlin.directory for the next free version.
   • Then finish UrlResolver 1250 on the new API: request threads only mark values stale, and consumer callbacks run on per-observable workers.
6. 1163 and 1254:
   • Verify wip/lane11-1163-run-to-completion: new tests fail first on 515d4ed21, plus a 70-test set. Every edit to existing tests must be listed with before and after.
   • Push it to 1163's branch, update the README and PR description for run-to-completion, then review.
   • Land 1163, then 1254.
7. 1258: finish round 2 so that a slow child stream keeps the root, but a root that has stopped making progress is retired (for example, a liveness check on child timeout). Adopt the reviewer's wedged-root probe as a permanent test. Re-review the current head 69195306e, because it changed during the round.
8. UrlProtocol 699: verify and push the wip nit fixes, re-review, then merge.
9. UrlProtocol 697: finish the stopped review. Decide whether the 3 intermittent service-set tests fail as often on main. Re-review, then merge.
10. After 697 and 699 merge:
    • Publish UrlProtocol.
    • Bump UrlResolver's pin.
    • Remove the README caveat from 1251 about the libp2p path.
    • Land the relay reproducer ext2/lane8-relayed-parked-handlers-stall as a test.
11. Follow-ups found but not started (each needs its own failing-first PR):
    • On a protocol-1.0 stream, Libp2pRpcStream.sendRequest sends ByteArray parameters as binary attachments the server cannot read, so the stream closes.
    • An echo handler that returns its upload as its result breaks whenever the client is still uploading at return. On main it hangs, about 16–18% of the time in-process even at 1000 B. Cause: RpcStreamManager.close() interrupts push threads.
    • Interrupting a push thread partway through a frame makes PersistentRpcConnection.doSendRequest abandon the shared connection.
    • Every root SUBSCRIBE recombines all subscribers' required pads, which is O(N²).
    • A self-containing Map or List in a result fails inside the stream and observable registration walk.
    • Two coverage gaps: DiscoveryConnectionRecovery.openStream admits a stream that install() later refuses, and readMessageWithTimeoutAsync turns a caller's cancellation into a stream failure.
    • Other intermittent tests seen: testInputStreamReturnLargeDataOverP2pSandboxedConnection, stressTestRelayUnreadBufferBackpressuresFastWriter, testProxyLibp2pBytecodeResponseStartsBeforeBackendAttachmentEnds, testSandboxedProxyRebuildsAfterSharedReconnectFails.

Reusable / operational knowledge

• Finding failed tests. The buildtest results API often returns 0 rows for failed runs. To find failed tests, page through https://buildtest.kotlin.build/api/test-results?id=<runId>&page=N, or read /root/buildtest-data/runs/<runId>/test-events.jsonl on 198.199.106.165, port 23, read-only.
• Flake statistics. Scanning about 460 runs on the host gave the flake rates above. That is the fastest way to tell a PR regression from a pre-existing flake.
• Merge mechanics.
  • Enqueue through the GraphQL enqueuePullRequest mutation; gh pr merge --auto is rejected.
  • build-watchman 0.0.21 --to-merged re-enqueues once, then stops on a second eviction.
  • Run it as a tracked background task; a detached & loses its completion notice.
• Local runs. Link jars/coursier to /home/u/bin/cs in kompile repos. Local scripts/test.bash --local runs work, but the shared /code box hit load average ~30 with six parallel lanes, so run at most one test.bash per lane.
• Review discipline that worked. Brief each fix round with the full list of properties the code must satisfy, not only the latest finding. Require a failing-first test per property. Treat any head change as voiding the prior review.
• Lessons from this effort.
  • Bounding a resource (threads) exposed couplings that unlimited threads had hidden, which forced the Observable upstream change.
  • The server cannot distinguish a client's cancellation from a disconnect, which drove the Option A decision.