← Priority list

Handoff: UrlResolver #671 — request-path binary attachments (sender), blocked on a fleet-wide merge-queue gridlock

Standing user directive: **"ensure [UrlResolver #671] gets merged."** It is the final, sender-half PR of an already-merged dependency chain that makes RPC **request** params carry `ByteArray` values as raw out-of-band libp2p **binary attachments** on streams that negotiate `/aicli-rpc/1.1.0`, transparently downgrading to base64-in-JSON (`/aicli-rpc/1.0.0`) for pre-1.1.0 peers. This removes the ~1.33× base64 blow-up and the SJVM base64-decode cost that caused the original buildtest `uploadChunk` SJVM timeout. The whole dependency chain (BuildTestServerService #163, UrlProtocol

hf-2026-07-05-urlresolver-671-binary-attachments-sender-handoff

Edit handoff

Title Summary Markdown body

# Handoff: UrlResolver #671 — request-path binary attachments (sender), blocked on a fleet-wide merge-queue gridlock

**Written:** 2026-07-12 ~16:52 UTC (effort began 2026-07-05; this supersedes the original 2026-07-05 version of this file).

> **⚠️ RE-VERIFY BEFORE ACTING.** Every state claim below is a write-time snapshot and goes stale
> fast (merge queue shuffles minute-to-minute, CI re-runs, host load swings). A fresh query ALWAYS
> wins over this document. Authoritative re-checks:
> - PR: `gh pr view 671 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,mergeable,mergedAt,headRefOid`
> - Queue: `gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:8){nodes{pullRequest{number} position state}}}}}'`
> - main HEAD: `gh api repos/CodexCoder21Organization/UrlResolver/commits/main --jq '.sha[0:8]+" "+.commit.committer.date'`
> - Host load (drain check): `ssh -p 23 root@198.199.106.165 'uptime'`

---

## 1. Mission

Standing user directive: **"ensure [UrlResolver #671] gets merged."** It is the final, sender-half
PR of an already-merged dependency chain that makes RPC **request** params carry `ByteArray` values
as raw out-of-band libp2p **binary attachments** on streams that negotiate `/aicli-rpc/1.1.0`,
transparently downgrading to base64-in-JSON (`/aicli-rpc/1.0.0`) for pre-1.1.0 peers. This removes
the ~1.33× base64 blow-up and the SJVM base64-decode cost that caused the original buildtest
`uploadChunk` SJVM timeout. The whole dependency chain (BuildTestServerService #163, UrlProtocol
#325/#330, jvm-libp2p #19, NetLabManagerServer #76) is **MERGED**; #671 is the last piece.

## 2. Current state — code DONE, blocked on the merge queue

**The code is complete, rebased, and verified.** The blocker is **not** #671's code — it is a
**fleet-wide merge-queue gridlock** on UrlResolver.

- **[UrlResolver #671](https://github.com/CodexCoder21Organization/UrlResolver/pull/671)** —
  `OPEN`, `mergeable=MERGEABLE`, `mergeStateStatus=CLEAN` vs main. Branch
  `request-path-binary-attachments-sender` @ **`d972c936`**. **Head checks GREEN** (`bld-build` pass,
  `kotlin.build (remote)` pass on run `d5a4936b`). 3/3 new attachment tests pass locally.
- **main HEAD: `5f84db1b`, committer date 2026-07-11 22:36 UTC** — i.e. **nothing has merged into
  main in ~18 hours** as of this writing. The queue is stuck for the whole repo, not just #671.
- **Queue at write time:** `749:p1:AWAITING_CHECKS  632:p2:AWAITING_CHECKS  653:p3:UNMERGEABLE
  671:p4:UNMERGEABLE`. #671 reads `UNMERGEABLE` **only speculatively** — because
  [#653](https://github.com/CodexCoder21Organization/UrlResolver/pull/653) ahead of it edits the same
  `build.kts` version line. Against real `main` it is `MERGEABLE`.

### Why the queue is gridlocked (verified by observation)
1. **`build.kts` version-line contention.** Many in-flight UrlResolver PRs each bump the
   `foundation.url:resolver` coordinate on the *same line* of `build.kts`. Any two are therefore a
   textual conflict, so whichever is behind goes `UNMERGEABLE` the moment the other is ahead in the
   queue. #653, #749 (and #671) all do this.
2. **Flaky load-sensitive stress/netlab tests evict whatever reaches the front.** The shared
   buildtest droplet (198.199.106.165, 2 vCPU) runs many queue validations at once; under load a
   handful of stress/netlab tests fail non-deterministically, failing the queue check and evicting
   the entry. Net effect: nothing reaches MERGED.

### #671's three evictions this session (all infra, none its own code)
1. **Infra collapse / host overload** — queue check reported mass `Test did not complete during
   build execution` while host load was ~90–100 with ~10 concurrent runs.
2. **Wedged buildtest run** — one run froze (shard control-socket idle >40 min while sibling runs
   advanced); I force-deleted it (`buildtest-cli delete <runId>`) after confirming the wedge on-host.
3. **4-of-1090 flaky tests** — queue run `1d23c39e`: **1086 PASSED / 4 FAILED**, all in the known
   load-flaky family: `stressTestConcurrentLateJoinersDiscoverServicePromptly`,
   `stressTestRegistrationAfterWithdrawalRaceUnderContention`,
   `stressTestRelayRegistrationPrefersConfiguredBootstrapRelay`, `testNetlabDynamicFailureRecovery`.
   None touches this PR's changed code.

## 3. What the rebase did (so you don't re-add a version sweep)

Rebased cleanly onto main as a **minimal 5-file diff, ~413 insertions**, NO version sweep:
- `src/foundation/url/resolver/UrlResolver.kt`: (a) `connectRpcController` (the
  `openPersistentRpcConnection` path only) now offers `listOf(PROTOCOL_ID_V11, PROTOCOL_ID)`; the
  other 4 `protocols = listOf(PROTOCOL_ID)` sites are deliberately untouched. (b)
  `PersistentRpcConnection.doSendRequest`: on a negotiated-1.1.0 controller, send
  `request.toJson(attachments)` + `writeMessage(output, json, attachments)`; else legacy
  `writeMessage(output, request.toJson())`. No-ByteArray request → empty attachment list →
  byte-identical compact frame on both versions.
- `build.kts`: resolver coordinate → **`0.0.631`** (source change). **Probe the pom URL before
  republishing** — earlier numbers are taken: `curl -sI https://kotlin.directory/foundation/url/resolver/0.0.631/resolver-0.0.631.pom`.
  Protocol/libp2p pins UNCHANGED (main already pins protocol `0.0.337` + jvm-libp2p `snapshot-10`,
  which carry `AttachmentRpcFrameDecoder` + `PROTOCOL_ID_V11`).
- 3 new test files (`tests/testPersistentRpcConnectionSends11ByteArrayAsAttachment.kts`,
  `tests/testPersistentRpcConnectionWireFallbackAndBackwardCompat.kts`,
  `tests/testE2ePersistentRpcConnectionNegotiates11AndSendsAttachment.kts`). Pins bumped to main
  consensus: **protocol 0.0.337, jvm-libp2p snapshot-10, observable core-jvm 0.3.13,
  community-kotlin-logging 0.0.3, build-kotlin-annotations 0.0.8, libSJVM 0.0.40.**

**No uncommitted work.** Everything is pushed to the PR branch. Workspace clone:
`/code/workspace/UrlResolver` (branch `request-path-binary-attachments-sender` @ `d972c936`, clean).

## 4. Relevant refs

| Ref | What | Write-time state |
|---|---|---|
| [UrlResolver #671](https://github.com/CodexCoder21Organization/UrlResolver/pull/671) | This PR (sender half) | OPEN, MERGEABLE/CLEAN vs main, head green @ `d972c936`, re-enqueued (speculatively UNMERGEABLE behind #653) |
| [UrlResolver #653](https://github.com/CodexCoder21Organization/UrlResolver/pull/653) | Ahead in queue; edits `build.kts` → conflicts with #671 | OPEN |
| [UrlResolver #749](https://github.com/CodexCoder21Organization/UrlResolver/pull/749) | Also edits `build.kts` version line | OPEN, in queue |
| main `5f84db1b` | Last merge 2026-07-11 22:36 UTC (~18h stale) | — |

## 5. Next steps (re-verify state first — §RE-VERIFY banner)

An **open user decision** governs which path to take (asked 2026-07-12, answer pending). The user's
reply was "Did you create a handoff?" — hence this doc. Re-ask which of these they want:

- **(A) Keep grinding the queue** — leave #671 enqueued; on each flaky eviction, re-enqueue **only
  when the executor is drained** (`ssh -p 23 root@198.199.106.165 'uptime'` → 1-min load < ~15 and
  few active runs) to minimize the stress-test flakes. A durable watcher was running this session
  (background task `bl6uib8nu`) that fires on MERGED or eviction; re-arm an equivalent watch. May
  take many cycles given the gridlock.
- **(B) Fix the flaky tests at root (recommended if the gridlock persists)** — root-cause the 4
  load-sensitive tests in §2 so ANY PR (not just #671) can clear the queue. This unblocks the whole
  repo, not just this PR. Delegate to codex gpt-5.6-sol per the global CLAUDE.md.
- **(C) Treat the 18h no-merge as a merge-queue/infra incident** — investigate why nothing merges
  (executor overload, queue config, whether the version-line contention is itself the deadlock) and
  fix at that layer.

To merge once a queue run is green: enqueue via GraphQL (repo auto-merge is OFF; `gh pr merge --auto`
is rejected):
```bash
NODE=$(gh pr view 671 --repo CodexCoder21Organization/UrlResolver --json id -q .id)
gh api graphql -f query='mutation($pr:ID!){ enqueuePullRequest(input:{pullRequestId:$pr}){ mergeQueueEntry{ position state } } }' -f pr="$NODE"
```
Merging #671 is **pre-authorized** by the standing directive — no further approval needed. NEVER use
`--admin`/`--delete-branch` or bypass the queue.

**If main has moved** since `5f84db1b` and a `build.kts`-touching PR merged: rebase the live remote
tip (`cd /code/workspace/UrlResolver && git fetch origin && git rebase origin/main`), re-pick a free
resolver version number (probe the pom URL), keep the diff minimal (no version sweep), force-push
`--force-with-lease`, let head checks re-run green, then enqueue.

## 6. Reusable / operational knowledge

- **Local isolated test run** (the workspace has a cross-`.kts` `WithArtifactVersionMismatch` storm
  that aborts a full `scripts/test.bash`): purge caches
  (`rm -rf ~/.aibuildcaches/_code_workspace_UrlResolver/{buildRuleResultIndex,artifactBinaryCache,persistent-scratchspace,scratchspace,buildRuleLocalArtifactDirIndex}`
  and `~/.cache/coursier/v1/bldbinary/repository/foundation/url`), move all but the 3 new tests out of
  `tests/`, run `scripts/test.bash --test <each>`, then restore. Expected `ALL TESTS PASSED (3/3)`.
- **buildtest frontend flaps HTTP 503 under host load** — this is degraded *observability*, not a
  dead run. Ground-truth on host: `ssh -p 23 root@198.199.106.165 'stat -c "%y %s" /root/buildtest-data/runs/<id>/{build.log,test-events.jsonl}'`.
  A run is only WEDGED if build.log mtime AND content are frozen >20 min while pending.
- **Diagnose a queue-check failure from the host** (frontend `/api/test-results` often returns empty
  under load): `ssh -p 23 root@198.199.106.165 'python3 -c "import json,collections;d=json.load(open(\"/root/buildtest-data/runs/<id>/test-results.json\"));print(collections.Counter(t[\"status\"] for t in d))"'`.
- **Enqueue only on a drained executor** — the stress/netlab flakes are load-triggered; enqueueing
  when 198.199.106.165 is at high load nearly guarantees an eviction.
- **git identity** in a fresh clone must be set before rebase-continue (the container has none):
  `git config user.name/user.email`.
- Version numbers `0.0.596`–`0.0.630` on kotlin.directory were already taken during this contention;
  always probe the pom URL before publishing a new resolver version.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver #671 — request-path binary attachments (sender), blocked on a fleet-wide merge-queue gridlock

Written: 2026-07-12 ~16:52 UTC (effort began 2026-07-05; this supersedes the original 2026-07-05 version of this file).

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below is a write-time snapshot and goes stale fast (merge queue shuffles minute-to-minute, CI re-runs, host load swings). A fresh query ALWAYS wins over this document. Authoritative re-checks:

PR: gh pr view 671 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,mergeable,mergedAt,headRefOid

Queue: gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:8){nodes{pullRequest{number} position state}}}}}'

main HEAD: gh api repos/CodexCoder21Organization/UrlResolver/commits/main --jq '.sha[0:8]+" "+.commit.committer.date'

Host load (drain check): ssh -p 23 root@198.199.106.165 'uptime'

1. Mission

Standing user directive: "ensure [UrlResolver #671] gets merged." It is the final, sender-half PR of an already-merged dependency chain that makes RPC request params carry ByteArray values as raw out-of-band libp2p binary attachments on streams that negotiate /aicli-rpc/1.1.0, transparently downgrading to base64-in-JSON (/aicli-rpc/1.0.0) for pre-1.1.0 peers. This removes the ~1.33× base64 blow-up and the SJVM base64-decode cost that caused the original buildtest uploadChunk SJVM timeout. The whole dependency chain (BuildTestServerService #163, UrlProtocol #325/#330, jvm-libp2p #19, NetLabManagerServer #76) is MERGED; #671 is the last piece.

2. Current state — code DONE, blocked on the merge queue

The code is complete, rebased, and verified. The blocker is not #671's code — it is a fleet-wide merge-queue gridlock on UrlResolver.

UrlResolver #671 — OPEN, mergeable=MERGEABLE, mergeStateStatus=CLEAN vs main. Branch request-path-binary-attachments-sender @ d972c936. Head checks GREEN (bld-build pass, kotlin.build (remote) pass on run d5a4936b). 3/3 new attachment tests pass locally.
main HEAD: 5f84db1b, committer date 2026-07-11 22:36 UTC — i.e. nothing has merged into main in ~18 hours as of this writing. The queue is stuck for the whole repo, not just #671.
Queue at write time: 749:p1:AWAITING_CHECKS 632:p2:AWAITING_CHECKS 653:p3:UNMERGEABLE 671:p4:UNMERGEABLE. #671 reads UNMERGEABLE only speculatively — because #653 ahead of it edits the same build.kts version line. Against real main it is MERGEABLE.

Why the queue is gridlocked (verified by observation)

build.kts version-line contention. Many in-flight UrlResolver PRs each bump the foundation.url:resolver coordinate on the same line of build.kts. Any two are therefore a textual conflict, so whichever is behind goes UNMERGEABLE the moment the other is ahead in the queue. #653, #749 (and #671) all do this.
Flaky load-sensitive stress/netlab tests evict whatever reaches the front. The shared buildtest droplet (198.199.106.165, 2 vCPU) runs many queue validations at once; under load a handful of stress/netlab tests fail non-deterministically, failing the queue check and evicting the entry. Net effect: nothing reaches MERGED.

#671's three evictions this session (all infra, none its own code)

Infra collapse / host overload — queue check reported mass Test did not complete during build execution while host load was ~90–100 with ~10 concurrent runs.
Wedged buildtest run — one run froze (shard control-socket idle >40 min while sibling runs advanced); I force-deleted it (buildtest-cli delete <runId>) after confirming the wedge on-host.
4-of-1090 flaky tests — queue run 1d23c39e: 1086 PASSED / 4 FAILED, all in the known load-flaky family: stressTestConcurrentLateJoinersDiscoverServicePromptly, stressTestRegistrationAfterWithdrawalRaceUnderContention, stressTestRelayRegistrationPrefersConfiguredBootstrapRelay, testNetlabDynamicFailureRecovery. None touches this PR's changed code.

3. What the rebase did (so you don't re-add a version sweep)

Rebased cleanly onto main as a minimal 5-file diff, ~413 insertions, NO version sweep:

src/foundation/url/resolver/UrlResolver.kt: (a) connectRpcController (the openPersistentRpcConnection path only) now offers listOf(PROTOCOL_ID_V11, PROTOCOL_ID); the other 4 protocols = listOf(PROTOCOL_ID) sites are deliberately untouched. (b) PersistentRpcConnection.doSendRequest: on a negotiated-1.1.0 controller, send request.toJson(attachments) + writeMessage(output, json, attachments); else legacy writeMessage(output, request.toJson()). No-ByteArray request → empty attachment list → byte-identical compact frame on both versions.
build.kts: resolver coordinate → 0.0.631 (source change). Probe the pom URL before republishing — earlier numbers are taken: curl -sI https://kotlin.directory/foundation/url/resolver/0.0.631/resolver-0.0.631.pom. Protocol/libp2p pins UNCHANGED (main already pins protocol 0.0.337 + jvm-libp2p snapshot-10, which carry AttachmentRpcFrameDecoder + PROTOCOL_ID_V11).
3 new test files (tests/testPersistentRpcConnectionSends11ByteArrayAsAttachment.kts, tests/testPersistentRpcConnectionWireFallbackAndBackwardCompat.kts, tests/testE2ePersistentRpcConnectionNegotiates11AndSendsAttachment.kts). Pins bumped to main consensus: protocol 0.0.337, jvm-libp2p snapshot-10, observable core-jvm 0.3.13, community-kotlin-logging 0.0.3, build-kotlin-annotations 0.0.8, libSJVM 0.0.40.

No uncommitted work. Everything is pushed to the PR branch. Workspace clone: /code/workspace/UrlResolver (branch request-path-binary-attachments-sender @ d972c936, clean).

4. Relevant refs

| Ref | What | Write-time state | |---|---|---| | UrlResolver #671 | This PR (sender half) | OPEN, MERGEABLE/CLEAN vs main, head green @ d972c936, re-enqueued (speculatively UNMERGEABLE behind #653) | | UrlResolver #653 | Ahead in queue; edits build.kts → conflicts with #671 | OPEN | | UrlResolver #749 | Also edits build.kts version line | OPEN, in queue | | main 5f84db1b | Last merge 2026-07-11 22:36 UTC (~18h stale) | — |

5. Next steps (re-verify state first — §RE-VERIFY banner)

An open user decision governs which path to take (asked 2026-07-12, answer pending). The user's reply was "Did you create a handoff?" — hence this doc. Re-ask which of these they want:

(A) Keep grinding the queue — leave #671 enqueued; on each flaky eviction, re-enqueue only when the executor is drained (ssh -p 23 root@198.199.106.165 'uptime' → 1-min load < ~15 and few active runs) to minimize the stress-test flakes. A durable watcher was running this session (background task bl6uib8nu) that fires on MERGED or eviction; re-arm an equivalent watch. May take many cycles given the gridlock.
(B) Fix the flaky tests at root (recommended if the gridlock persists) — root-cause the 4 load-sensitive tests in §2 so ANY PR (not just #671) can clear the queue. This unblocks the whole repo, not just this PR. Delegate to codex gpt-5.6-sol per the global CLAUDE.md.
(C) Treat the 18h no-merge as a merge-queue/infra incident — investigate why nothing merges (executor overload, queue config, whether the version-line contention is itself the deadlock) and fix at that layer.

To merge once a queue run is green: enqueue via GraphQL (repo auto-merge is OFF; gh pr merge --auto is rejected):

NODE=$(gh pr view 671 --repo CodexCoder21Organization/UrlResolver --json id -q .id)
gh api graphql -f query='mutation($pr:ID!){ enqueuePullRequest(input:{pullRequestId:$pr}){ mergeQueueEntry{ position state } } }' -f pr="$NODE"

Merging #671 is pre-authorized by the standing directive — no further approval needed. NEVER use --admin/--delete-branch or bypass the queue.

If main has moved since 5f84db1b and a build.kts-touching PR merged: rebase the live remote tip (cd /code/workspace/UrlResolver && git fetch origin && git rebase origin/main), re-pick a free resolver version number (probe the pom URL), keep the diff minimal (no version sweep), force-push --force-with-lease, let head checks re-run green, then enqueue.

6. Reusable / operational knowledge

Local isolated test run (the workspace has a cross-.kts WithArtifactVersionMismatch storm that aborts a full scripts/test.bash): purge caches (rm -rf ~/.aibuildcaches/_code_workspace_UrlResolver/{buildRuleResultIndex,artifactBinaryCache,persistent-scratchspace,scratchspace,buildRuleLocalArtifactDirIndex} and ~/.cache/coursier/v1/bldbinary/repository/foundation/url), move all but the 3 new tests out of tests/, run scripts/test.bash --test <each>, then restore. Expected ALL TESTS PASSED (3/3).
buildtest frontend flaps HTTP 503 under host load — this is degraded observability, not a dead run. Ground-truth on host: ssh -p 23 root@198.199.106.165 'stat -c "%y %s" /root/buildtest-data/runs/<id>/{build.log,test-events.jsonl}'. A run is only WEDGED if build.log mtime AND content are frozen >20 min while pending.
Diagnose a queue-check failure from the host (frontend /api/test-results often returns empty under load): ssh -p 23 root@198.199.106.165 'python3 -c "import json,collections;d=json.load(open(\"/root/buildtest-data/runs/<id>/test-results.json\"));print(collections.Counter(t[\"status\"] for t in d))"'.
Enqueue only on a drained executor — the stress/netlab flakes are load-triggered; enqueueing when 198.199.106.165 is at high load nearly guarantees an eviction.
git identity in a fresh clone must be set before rebase-continue (the container has none): git config user.name/user.email.
Version numbers 0.0.596–0.0.630 on kotlin.directory were already taken during this contention; always probe the pom URL before publishing a new resolver version.