← Priority list

Handoff-drive campaign: 22 PRs landed; transport chain staged but the merge queue will not admit the keystone

Campaign over the open-handoff backlog. 22 pull requests merged, 6 closed with evidence, 8 challenges filed. The remaining work is one dependency chain: a transport fix that stops completed responses being discarded is green on its branch but has been evicted from the merge queue THREE times on the identical commit, with the combined build failing 1, then 11, then 15 tests across unrelated subsystems while GitHub Actions passed every time �� the evidence points at a degraded build host, not the change. Its two dependents, the protocol publication, and a verified review fix are all staged and unpushed behind it. Separately, 23 owner pull requests are green behind a workspace test-harness fix that is partially applied.

hf-2026-07-29-handoff-drive-session-2026-07-29-6-lane-per-handoff-campaign-11-prs-landed-resolver-netlab-cluster-gated-on-the-netlab-concurrency-decision-q8

Edit handoff

Title Summary Markdown body

# Handoff-drive campaign — 22 PRs landed; transport chain staged but the merge queue will not admit the keystone

**RE-VERIFY:** every fact below is a snapshot taken 2026-07-30 ~23:25 UTC. Re-check each pull request with
`gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt`
and re-probe artifact coordinates before relying on them — coordinates were claimed by parallel work several times during this session.

## Mission

Drive the open-handoff backlog campaign to completion. The earlier decision "Q8" was re-framed by the owner: the NetLab manager is not overloaded by worker count; there is a real defect, and it was found (see below).

## Landed this session (22 merges)

Most recent four, all verified merged:
- https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/198 — merged 19:34:55Z
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/379 — merged 19:23:14Z
- https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/63 — merged 18:52:28Z (NetLab worker on the common transport basis)
- https://github.com/CodexCoder21Organization/github-repo-config-manager/pull/41 — merged 18:49:48Z (merge-queue check-response timeout 60 → 240 minutes)

Also: 6 pull requests closed with written evidence, 2 issues filed, 8 challenges recorded, 1 production fix, 4+ handoffs completed.

## The blocking knot

Everything below is ready; the chain cannot advance because its keystone will not merge.

**Keystone — https://github.com/CodexCoder21Organization/UrlProtocol/pull/457** (head `4510849637941ae23cee8d41107bc8d8c21cf281`)
- Fixes a real defect: the transport copied every completed response into one shared 4 MiB per-connection buffer before sending. Four concurrent 1 MiB attachment responses total exactly 4 MiB raw, but each carries an envelope and length fields, so the aggregate overflows. The transport admitted and executed all four non-replayable handlers, then substituted an `OUTBOX_FULL` error for a result already computed — completed work discarded.
- Fix: route attachment frames above 1 MiB through the existing bounded, connection-serialized chunk writer instead of duplicating them in the awaited-response buffer. The 4 MiB cap, consumer concurrency, payload size, iteration counts, heap limits and time limits are all unchanged.
- A fail-first reproducer fails in ~2.8 s on the first run with four real callers on one root connection, and passes against the fixed source.
- **All three branch checks are green.** State OPEN/CLEAN.
- **Evicted from the merge queue three times, on the identical commit:**
  - 21:44 → 21:57 — combined build 1260 passed, **1** failed (`testPeerRegistryPartiallyIndexedPopulationUsesExactEvictionFallback`, per-test 30s timeout)
  - 22:44 → 22:59 — combined build 89 passed, **11** failed
  - 00:09 → ~00:25 (2026-07-31) — combined build 85 passed, **15** failed
- On **all three** evictions the queue branch's GitHub Actions runs (`Bld - Build`, `Bld - All Tests`) **succeeded**; the failing gate was the remote `kotlin.build` check each time.
- **The failure count is growing and the diagnosis is the runner, not this change.** The 15 failures span six unrelated subsystems (gossip callback delivery, listener invocation, loopback dialling, registry concurrency, registry scaling, exact-peer round trips), **eight repeat exactly** from the 11-failure run, and nearly all are deadline expiries: per-test 30s timeouts, 2s dial timeouts **to a loopback peer**, and one 800-peer/20-round case exceeding an 8s budget at 8006ms. A change to how large response frames are written cannot cause a 2-second loopback dial to time out.
- Repository merge-queue check-response timeout reads 240 minutes, so timeout is **not** the cause here.
- **Do not attempt a fourth queue entry until the build host is confirmed healthy** (restarted/reprovisioned, or investigated for starvation, swapping, or zombie processes). Each attempt has failed harder than the last.

**Dependent 1 — https://github.com/CodexCoder21Organization/UrlResolver/pull/901** (head `02fafa6c66e4afa5f14fcd7fd55fa7eba31fcb6f`, based on main)
- Reconciles the published resolver source lineage, implements the gossip-handler callbacks the current protocol requires, adds a guard so an observation record with zero successes and zero failures is not read as evidence of a failed connection, and adds a host-list fallback.
- Needs exactly **8** version-string replacements once the protocol release is published. Staged locally, deliberately unpushed.

**Dependent 2 — https://github.com/CodexCoder21Organization/UrlResolver/pull/900** (head `f8fb3c01a451cd69678f386e7cc7aab84c193406`)
- Object-array / sandbox-collection handling; stacked on dependent 1's branch. Needs **18** replacements.
- Its GitHub Actions suite does **not** dispatch while its base is a feature branch — it can only be fully verified after dependent 1 merges and it is retargeted to main.
- **An unapplied review fix exists**: a cumulative accounting gap, fixed and verified 2/2 with two real sandbox tests, checkpointed at remote commit `9f014311003829246afb109e8f7fa38e7fdd1b31`. Adversarial review traced preflight traversal and conversion across Map, List, Collection, Array, JSONObject, JSONArray and ByteArray and found no further gap. **Not yet applied to the pull request branch.**

**Publication target:** the next free protocol coordinate. `0.0.430`–`0.0.433` all returned HTTP 404 at 23:09 — re-probe before publishing.

**Required order:** merge the keystone → publish its release → dependent 1 → retarget and merge dependent 2.

**Released by this chain when it lands:** six NetLab-gated pull requests plus https://github.com/CodexCoder21Organization/UrlResolver/pull/885 and https://github.com/CodexCoder21Organization/W3WalletTests/pull/30.

## Also in flight

**https://github.com/CodexCoder21Organization/CodexOrganizationWorkspace/pull/182** with 23 owner pull requests, **all 23 verified green**.
- Root cause of the shard failures: helper processes spawned during tests were configured for sustained work (full optimization, multiple collection threads, four at once) rather than the seconds-long tasks they perform, so on a two-core machine compilation overhead crowded out the tests, producing timeouts with a shifting failure set.
- Applying the upstream settings lifted completions from 16-of-27 to 23-of-27; a further reduction of compiler threads per helper is the remaining step. Verified in CI that the settings genuinely reach the child processes.
- No test, threshold, iteration count, or machine size was changed, and none should be.
- Merge order when green: the 23 owners first, workspace pull request last.

## Reusable knowledge earned here

- **Merging a configuration generator does not change any configuration.** The queue-timeout fix corrected a tool's default; the live repositories only changed when the tool was run against them and each value was read back from GitHub. Treat "merged" and "applied" as separate milestones.
- **An emergency edit outlives the emergency.** One repository still carried a hand-set 180-minute value from an incident eight days earlier, disagreeing with what the tooling generates.
- **A build cache keyed on path and build-file hash will silently serve stale classes when only source changed** — a verification run reproduced a pre-fix error exactly until the cache was moved aside and the build re-run from empty.
- **Stacked pull requests do not receive the full check suite** while their base is a feature branch.
- **Compare which test failed, not just that something failed.** Recent failures in the transport repository hit a different single test each time; that is a different signal from the same test failing repeatedly.

## Next steps

1. Determine why the remote test check fails on queue-branch builds while the same commit's GitHub Actions builds pass, and why two consecutive runs produced 1-of-1261 and 11-of-100. Read the queue branch's own run records; do not re-queue speculatively.
2. Merge the keystone once a mergeable route exists; publish the next free protocol coordinate.
3. Apply the 8 replacements to dependent 1, rebase on latest main, push with `--force-with-lease`, drive green, merge.
4. Apply the checkpointed review fix plus 18 replacements to dependent 2, retarget it to main, rebase, drive green, merge.
5. Finish the workspace shard comparison; merge the 23 owners, then the workspace pull request.
6. Close out this campaign.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff-drive campaign — 22 PRs landed; transport chain staged but the merge queue will not admit the keystone

RE-VERIFY: every fact below is a snapshot taken 2026-07-30 ~23:25 UTC. Re-check each pull request with gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt and re-probe artifact coordinates before relying on them — coordinates were claimed by parallel work several times during this session.

Mission

Landed this session (22 merges)

Most recent four, all verified merged:

https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/198 — merged 19:34:55Z
https://github.com/CodexCoder21Organization/UrlProtocol/pull/379 — merged 19:23:14Z
https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/63 — merged 18:52:28Z (NetLab worker on the common transport basis)
https://github.com/CodexCoder21Organization/github-repo-config-manager/pull/41 — merged 18:49:48Z (merge-queue check-response timeout 60 → 240 minutes)

Also: 6 pull requests closed with written evidence, 2 issues filed, 8 challenges recorded, 1 production fix, 4+ handoffs completed.

The blocking knot

Everything below is ready; the chain cannot advance because its keystone will not merge.

Keystone — https://github.com/CodexCoder21Organization/UrlProtocol/pull/457 (head 4510849637941ae23cee8d41107bc8d8c21cf281)

Fixes a real defect: the transport copied every completed response into one shared 4 MiB per-connection buffer before sending. Four concurrent 1 MiB attachment responses total exactly 4 MiB raw, but each carries an envelope and length fields, so the aggregate overflows. The transport admitted and executed all four non-replayable handlers, then substituted an OUTBOX_FULL error for a result already computed — completed work discarded.
Fix: route attachment frames above 1 MiB through the existing bounded, connection-serialized chunk writer instead of duplicating them in the awaited-response buffer. The 4 MiB cap, consumer concurrency, payload size, iteration counts, heap limits and time limits are all unchanged.
A fail-first reproducer fails in ~2.8 s on the first run with four real callers on one root connection, and passes against the fixed source.
All three branch checks are green. State OPEN/CLEAN.
Evicted from the merge queue three times, on the identical commit:
- 21:44 → 21:57 — combined build 1260 passed, 1 failed (testPeerRegistryPartiallyIndexedPopulationUsesExactEvictionFallback, per-test 30s timeout)
- 22:44 → 22:59 — combined build 89 passed, 11 failed
- 00:09 → ~00:25 (2026-07-31) — combined build 85 passed, 15 failed
On all three evictions the queue branch's GitHub Actions runs (Bld - Build, Bld - All Tests) succeeded; the failing gate was the remote kotlin.build check each time.
The failure count is growing and the diagnosis is the runner, not this change. The 15 failures span six unrelated subsystems (gossip callback delivery, listener invocation, loopback dialling, registry concurrency, registry scaling, exact-peer round trips), eight repeat exactly from the 11-failure run, and nearly all are deadline expiries: per-test 30s timeouts, 2s dial timeouts to a loopback peer, and one 800-peer/20-round case exceeding an 8s budget at 8006ms. A change to how large response frames are written cannot cause a 2-second loopback dial to time out.
Repository merge-queue check-response timeout reads 240 minutes, so timeout is not the cause here.
Do not attempt a fourth queue entry until the build host is confirmed healthy (restarted/reprovisioned, or investigated for starvation, swapping, or zombie processes). Each attempt has failed harder than the last.

Dependent 1 — https://github.com/CodexCoder21Organization/UrlResolver/pull/901 (head 02fafa6c66e4afa5f14fcd7fd55fa7eba31fcb6f, based on main)

Reconciles the published resolver source lineage, implements the gossip-handler callbacks the current protocol requires, adds a guard so an observation record with zero successes and zero failures is not read as evidence of a failed connection, and adds a host-list fallback.
Needs exactly 8 version-string replacements once the protocol release is published. Staged locally, deliberately unpushed.

Dependent 2 — https://github.com/CodexCoder21Organization/UrlResolver/pull/900 (head f8fb3c01a451cd69678f386e7cc7aab84c193406)

Object-array / sandbox-collection handling; stacked on dependent 1's branch. Needs 18 replacements.
Its GitHub Actions suite does not dispatch while its base is a feature branch — it can only be fully verified after dependent 1 merges and it is retargeted to main.
An unapplied review fix exists: a cumulative accounting gap, fixed and verified 2/2 with two real sandbox tests, checkpointed at remote commit 9f014311003829246afb109e8f7fa38e7fdd1b31. Adversarial review traced preflight traversal and conversion across Map, List, Collection, Array, JSONObject, JSONArray and ByteArray and found no further gap. Not yet applied to the pull request branch.

Publication target: the next free protocol coordinate. 0.0.430–0.0.433 all returned HTTP 404 at 23:09 — re-probe before publishing.

Required order: merge the keystone → publish its release → dependent 1 → retarget and merge dependent 2.

Released by this chain when it lands: six NetLab-gated pull requests plus https://github.com/CodexCoder21Organization/UrlResolver/pull/885 and https://github.com/CodexCoder21Organization/W3WalletTests/pull/30.

Also in flight

https://github.com/CodexCoder21Organization/CodexOrganizationWorkspace/pull/182 with 23 owner pull requests, all 23 verified green.

Root cause of the shard failures: helper processes spawned during tests were configured for sustained work (full optimization, multiple collection threads, four at once) rather than the seconds-long tasks they perform, so on a two-core machine compilation overhead crowded out the tests, producing timeouts with a shifting failure set.
Applying the upstream settings lifted completions from 16-of-27 to 23-of-27; a further reduction of compiler threads per helper is the remaining step. Verified in CI that the settings genuinely reach the child processes.
No test, threshold, iteration count, or machine size was changed, and none should be.
Merge order when green: the 23 owners first, workspace pull request last.

Reusable knowledge earned here

Merging a configuration generator does not change any configuration. The queue-timeout fix corrected a tool's default; the live repositories only changed when the tool was run against them and each value was read back from GitHub. Treat "merged" and "applied" as separate milestones.
An emergency edit outlives the emergency. One repository still carried a hand-set 180-minute value from an incident eight days earlier, disagreeing with what the tooling generates.
A build cache keyed on path and build-file hash will silently serve stale classes when only source changed — a verification run reproduced a pre-fix error exactly until the cache was moved aside and the build re-run from empty.
Stacked pull requests do not receive the full check suite while their base is a feature branch.
Compare which test failed, not just that something failed. Recent failures in the transport repository hit a different single test each time; that is a different signal from the same test failing repeatedly.

Next steps

Determine why the remote test check fails on queue-branch builds while the same commit's GitHub Actions builds pass, and why two consecutive runs produced 1-of-1261 and 11-of-100. Read the queue branch's own run records; do not re-queue speculatively.
Merge the keystone once a mergeable route exists; publish the next free protocol coordinate.
Apply the 8 replacements to dependent 1, rebase on latest main, push with --force-with-lease, drive green, merge.
Apply the checkpointed review fix plus 18 replacements to dependent 2, retarget it to main, rebase, drive green, merge.
Finish the workspace shard comparison; merge the 23 owners, then the workspace pull request.
Close out this campaign.