← Priority list

Buildtest pending-wedge follow-on: all analysis done; UrlResolver 856 proven green but blocked by rotating merge-queue flakes �� needs netlab-chain redeploy decision or flaky-test campaign or more queue attempts

All five follow-on items are resolved as engineering work: UrlResolver 856 (six reconnect regression tests, zero production changes) is rebased onto current main, adapted to the deliberately-changed dispatch ordering of the stabilization batch, hardened against shard-load observation races (latch waits at unchanged bounds), and proven green in every execution �� isolated, under concurrent load, full bld-build suite, and every buildtest run. The post-merge review of UrlProtocol 430 found one residual High interleaving gap (issue 424 retitled, evidence posted); the two wip/pr430 branches were verified NOT superseded and retained; and netlab non-recovery was proven at request-ID granularity (whole provisioning chain still on protocol 0.0.400; fix published as 0.0.429 but undeployed). The one open item: the PR was evicted from the merge queue four times, every time by a different rotating pre-existing flake (never its own tests); re-enqueue budget deliberately exhausted. Unblock by approving the four-service netlab-chain redeploy on >=0.0.429, running a flaky-test campaign over the rotating queue killers (start with testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork), or authorizing further queue attempts.

hf-2026-07-30-buildtest-pending-wedge-follow-on-upstream-response-drop-fix-merged-only-the-regression-test-pr-urlresolver-856-remains

Edit handoff

Title Summary

All five follow-on items are resolved as engineering work: UrlResolver 856 (six reconnect regression tests, zero production changes) is rebased onto current main, adapted to the deliberately-changed dispatch ordering of the stabilization batch, hardened against shard-load observation races (latch waits at unchanged bounds), and proven green in every execution ��� isolated, under concurrent load, full bld-build suite, and every buildtest run. The post-merge review of UrlProtocol 430 found one residual High interleaving gap (issue 424 retitled, evidence posted); the two wip/pr430 branches were verified NOT superseded and retained; and netlab non-recovery was proven at request-ID granularity (whole provisioning chain still on protocol 0.0.400; fix published as 0.0.429 but undeployed). The one open item: the PR was evicted from the merge queue four times, every time by a different rotating pre-existing flake (never its own tests); re-enqueue budget deliberately exhausted. Unblock by approving the four-service netlab-chain redeploy on >=0.0.429, running a flaky-test campaign over the rotating queue killers (start with testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork), or authorizing further queue attempts.

Markdown body

# Buildtest pending-wedge follow-on: ALL analysis items done; UrlResolver 856 is proven green but cannot survive the flaky merge queue — needs the netlab-chain redeploy decision (or a flaky-test campaign / more queue attempts)

**⚠️ RE-VERIFY — everything below is a write-time snapshot (2026-07-30 ~20:50 UTC). Authoritative re-checks:**
- `gh pr view 856 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt`
- `gh issue view 424 --repo CodexCoder21Organization/UrlProtocol` (all evidence comments live there)
- netlab service health: the deployed chain still runs protocol `0.0.400` classes — verify by extracting the running JAR, not by service liveness.

## Mission

Finish the follow-on of the 2026-07-19 buildtest pending-wedge effort. Four of the five next-steps are **complete**; the remaining one — land https://github.com/CodexCoder21Organization/UrlResolver/pull/856 — is finished as engineering work but blocked by merge-queue instability whose durable fixes need a decision.

## What was found and done this session (2026-07-30, continuing the prior chain)

1. **https://github.com/CodexCoder21Organization/UrlResolver/pull/856 rebased and adapted to current main.** After rebase, one test failed because main's stabilization batch (https://github.com/CodexCoder21Organization/UrlResolver/pull/887) deliberately reordered dispatch: a request queued behind the connection write lock now registers with the *current* transport generation after acquiring the lock (a fresh dispatch, not a replay). Verified with file:line evidence that the new ordering preserves the safety contract (no stale-map registration, no replay after bytes written, no lost requests); the tests were adapted to assert the observable contract instead of the old internal ordering, and their decorators extended to intercept `write(Int)` single-byte writes.
2. **A shard-load test-observability race was reproduced and fixed.** `testStaleWriteFailureDoesNotClobberReconnectedTransport` failed on a saturated CI shard (5-second spin on `isClosed` outrun by scheduling). Reproduced locally under genuine five-way compiler contention (1/10 failure), confirmed the production close path is deterministic, and replaced all flag-polling waits across five test files with latch-based observations at the SAME time bounds (no timeout enlarged, no assertion weakened). 5/5 concurrent-load passes per modified test. Final head: `37b72647ed61824468101b71fb135766345dbef9` (remote-verified; PR description documents both adaptations).
3. **The six regression tests are proven everywhere:** isolated local runs, concurrent-load local runs, `bld-build` full suite (SUCCESS on this head), and every buildtest execution — at least six full-suite runs total with zero failures among the six.
4. **Post-merge review of https://github.com/CodexCoder21Organization/UrlProtocol/pull/430 delivered (prior handoff step 3).** Verdict: original defect genuinely fixed (small replies, relay paths, teardown; no refcount leak); ONE residual High gap — the outbox drain writes without the per-stream frame lock and can interleave into a concurrent >4 MiB chunked reply on the same substream. Issue retitled and evidence posted: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5130419334 (keep open; next step there is a fail-first reproducer).
5. **The two `wip/pr430-*` snapshot branches are NOT superseded** (prior step 4): they hold a root-event-loop ordering safeguard, a stricter mid-frame backpressure test trigger, and an incomplete-Netty-write accounting test with no equivalents on main. Branches RETAINED; inventory posted: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5130587424.
6. **Netlab recovery confirmed NOT achieved, with request-ID-level production proof** (prior step 5): webcron durably persisted the same timer request three times while the netlab-manager saw every attempt fail (pre-fix reply loss live in production); the whole provisioning chain (netlab-manager, netlab-worker, droplet service, webcron) still runs protocol `0.0.400` classes — a JAR re-uploaded 15:44 UTC today included. Restarts cannot fix it. Full forensics: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5133908018. The fix is published (protocol `0.0.429`, verified by artifact inspection to contain the `AwaitedResponseOutbox` classes).
7. **Four merge-queue attempts on the proven-green head, four evictions, each by rotating pre-existing flakes** (never by this PR's tests): cycle 2 `testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork` (30 s timeout); cycle 3 `stressTestConcurrentRegistrationsReliablyPropagateToRelay` + `testPeerAnnouncementForwardingUsesWithdrawalFilteredServices`; cycle 4 `testNetlabResolutionUnderLatencyAndPacketLoss` (netlab acquisition) + the in-memory gossip test again. Validation runs scored 1233–1234 of 1235 each cycle. Re-enqueue budget deliberately exhausted at four.

## Relevant PRs / refs / branches (all verified at write time)

- https://github.com/CodexCoder21Organization/UrlResolver/pull/856 — OPEN, head `37b72647ed61824468101b71fb135766345dbef9` (= remote), branch `tests/persistent-rpc-reconnect-regressions`. `bld-build` ✅ on this head; `kotlin.build (remote)` red only from queue-cycle flakes. **No production source changes; six tests only.**
- https://github.com/CodexCoder21Organization/UrlProtocol/issues/424 — OPEN (deliberately), retitled to the residual interleaving gap; carries all three evidence comments.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/430 — MERGED (context; fix published as protocol `0.0.429`).
- Retained branches: https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/pr430-local-snapshot-handoff-2026-07-30 and https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/pr430-validation-c7-handoff-2026-07-30 — do NOT delete; unported items listed in the second issue comment.
- Nothing exists only locally: the working branch and both wip branches are on remote GitHub; all forensic evidence is in the issue comments; no stashes or unpushed commits remain in the session checkouts.

## Next steps

1. **Decision (the actual blocker): pick one or more of —**
   a. **Approve the coordinated redeploy** of netlab-manager, netlab-worker, the DigitalOcean droplet service, and the webcron URL service rebuilt on protocol `>= 0.0.429` (recommended in the third issue comment; removes the netlab failure class from every CI run, per the request-ID-level proof).
   b. **Run a flaky-test campaign** (fix-flakey-test skill) over the rotating queue killers, starting with `testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork` (killed two of four cycles; 30 s timeout in an in-memory test — no netlab involved), then the relay stress and gossip-forwarding tests.
   c. **Authorize more merge-queue attempts** for https://github.com/CodexCoder21Organization/UrlResolver/pull/856 (each cycle is ~40–60 min; odds improve as fleet load falls overnight; enqueue via GraphQL `enqueuePullRequest` — `gh pr merge --auto` is rejected in this repo — and shepherd with `build-watchman 0.0.13 --to-merged`).
2. After the PR lands: re-verify all five original next-steps one final time and complete this handoff via `handoff-cli`.
3. Independent follow-ups already tracked elsewhere: the interleaving-gap reproducer and the wip-branch ports (both on https://github.com/CodexCoder21Organization/UrlProtocol/issues/424).

## Reusable / operational knowledge (adds to the prior handoff's list, which remains valid)

- **The queue's rotating-flake eviction pattern:** a proven-green PR evicted each cycle by a *different* pre-existing test is a suite-hygiene signal, not a PR problem. Record each cycle's failure list; the rotation itself is the diagnosis.
- **Verify deployed code by artifact contents, not by restart/upload timestamps** — a netlab JAR uploaded today still contained the years-old protocol class.
- **Test waits must observe published signals, not implementation flags** — the shard-load race was fixed by converting 5-second flag-poll spins to 5-second latch waits (same bounds, deterministic observation); reproduce such races with genuine concurrent load (parallel compilers/tests), never artificial spinners.
- **build-watchman 0.0.13 `--to-merged`** attaches cleanly to an existing queue entry after a shepherd restart, auto-re-enqueues exactly once on silent eviction, and its exit is an actionable event, not evidence of watching.
- **The buildtest test-results API paginates (first 100 only)** — "frozen at 100/100" with a growing `build.log` is the pagination artifact; read `/root/buildtest-data/runs/<id>/test-events.jsonl` on the host for ground truth.
- **`gh pr edit --body-file` fails on this box's gh** (deprecated `projectCards` GraphQL); use the pull-request REST endpoint and read back.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Buildtest pending-wedge follow-on: ALL analysis items done; UrlResolver 856 is proven green but cannot survive the flaky merge queue — needs the netlab-chain redeploy decision (or a flaky-test campaign / more queue attempts)

⚠️ RE-VERIFY — everything below is a write-time snapshot (2026-07-30 ~20:50 UTC). Authoritative re-checks:

gh pr view 856 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
gh issue view 424 --repo CodexCoder21Organization/UrlProtocol (all evidence comments live there)
netlab service health: the deployed chain still runs protocol 0.0.400 classes — verify by extracting the running JAR, not by service liveness.

Mission

Finish the follow-on of the 2026-07-19 buildtest pending-wedge effort. Four of the five next-steps are complete; the remaining one — land https://github.com/CodexCoder21Organization/UrlResolver/pull/856 — is finished as engineering work but blocked by merge-queue instability whose durable fixes need a decision.

What was found and done this session (2026-07-30, continuing the prior chain)

https://github.com/CodexCoder21Organization/UrlResolver/pull/856 rebased and adapted to current main. After rebase, one test failed because main's stabilization batch (https://github.com/CodexCoder21Organization/UrlResolver/pull/887) deliberately reordered dispatch: a request queued behind the connection write lock now registers with the current transport generation after acquiring the lock (a fresh dispatch, not a replay). Verified with file:line evidence that the new ordering preserves the safety contract (no stale-map registration, no replay after bytes written, no lost requests); the tests were adapted to assert the observable contract instead of the old internal ordering, and their decorators extended to intercept write(Int) single-byte writes.
A shard-load test-observability race was reproduced and fixed. testStaleWriteFailureDoesNotClobberReconnectedTransport failed on a saturated CI shard (5-second spin on isClosed outrun by scheduling). Reproduced locally under genuine five-way compiler contention (1/10 failure), confirmed the production close path is deterministic, and replaced all flag-polling waits across five test files with latch-based observations at the SAME time bounds (no timeout enlarged, no assertion weakened). 5/5 concurrent-load passes per modified test. Final head: 37b72647ed61824468101b71fb135766345dbef9 (remote-verified; PR description documents both adaptations).
The six regression tests are proven everywhere: isolated local runs, concurrent-load local runs, bld-build full suite (SUCCESS on this head), and every buildtest execution — at least six full-suite runs total with zero failures among the six.
Post-merge review of https://github.com/CodexCoder21Organization/UrlProtocol/pull/430 delivered (prior handoff step 3). Verdict: original defect genuinely fixed (small replies, relay paths, teardown; no refcount leak); ONE residual High gap — the outbox drain writes without the per-stream frame lock and can interleave into a concurrent >4 MiB chunked reply on the same substream. Issue retitled and evidence posted: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5130419334 (keep open; next step there is a fail-first reproducer).
The two wip/pr430-* snapshot branches are NOT superseded (prior step 4): they hold a root-event-loop ordering safeguard, a stricter mid-frame backpressure test trigger, and an incomplete-Netty-write accounting test with no equivalents on main. Branches RETAINED; inventory posted: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5130587424.
Netlab recovery confirmed NOT achieved, with request-ID-level production proof (prior step 5): webcron durably persisted the same timer request three times while the netlab-manager saw every attempt fail (pre-fix reply loss live in production); the whole provisioning chain (netlab-manager, netlab-worker, droplet service, webcron) still runs protocol 0.0.400 classes — a JAR re-uploaded 15:44 UTC today included. Restarts cannot fix it. Full forensics: https://github.com/CodexCoder21Organization/UrlProtocol/issues/424#issuecomment-5133908018. The fix is published (protocol 0.0.429, verified by artifact inspection to contain the AwaitedResponseOutbox classes).
Four merge-queue attempts on the proven-green head, four evictions, each by rotating pre-existing flakes (never by this PR's tests): cycle 2 testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork (30 s timeout); cycle 3 stressTestConcurrentRegistrationsReliablyPropagateToRelay + testPeerAnnouncementForwardingUsesWithdrawalFilteredServices; cycle 4 testNetlabResolutionUnderLatencyAndPacketLoss (netlab acquisition) + the in-memory gossip test again. Validation runs scored 1233–1234 of 1235 each cycle. Re-enqueue budget deliberately exhausted at four.

Relevant PRs / refs / branches (all verified at write time)

https://github.com/CodexCoder21Organization/UrlResolver/pull/856 — OPEN, head 37b72647ed61824468101b71fb135766345dbef9 (= remote), branch tests/persistent-rpc-reconnect-regressions. bld-build ✅ on this head; kotlin.build (remote) red only from queue-cycle flakes. No production source changes; six tests only.
https://github.com/CodexCoder21Organization/UrlProtocol/issues/424 — OPEN (deliberately), retitled to the residual interleaving gap; carries all three evidence comments.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/430 — MERGED (context; fix published as protocol 0.0.429).
Retained branches: https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/pr430-local-snapshot-handoff-2026-07-30 and https://github.com/CodexCoder21Organization/UrlProtocol/tree/wip/pr430-validation-c7-handoff-2026-07-30 — do NOT delete; unported items listed in the second issue comment.
Nothing exists only locally: the working branch and both wip branches are on remote GitHub; all forensic evidence is in the issue comments; no stashes or unpushed commits remain in the session checkouts.

Next steps

Decision (the actual blocker): pick one or more of — a. Approve the coordinated redeploy of netlab-manager, netlab-worker, the DigitalOcean droplet service, and the webcron URL service rebuilt on protocol >= 0.0.429 (recommended in the third issue comment; removes the netlab failure class from every CI run, per the request-ID-level proof). b. Run a flaky-test campaign (fix-flakey-test skill) over the rotating queue killers, starting with testUrlProtocol2GossipAndRpcUseInjectedInMemoryNetwork (killed two of four cycles; 30 s timeout in an in-memory test — no netlab involved), then the relay stress and gossip-forwarding tests. c. Authorize more merge-queue attempts for https://github.com/CodexCoder21Organization/UrlResolver/pull/856 (each cycle is ~40–60 min; odds improve as fleet load falls overnight; enqueue via GraphQL enqueuePullRequest — gh pr merge --auto is rejected in this repo — and shepherd with build-watchman 0.0.13 --to-merged).
After the PR lands: re-verify all five original next-steps one final time and complete this handoff via handoff-cli.
Independent follow-ups already tracked elsewhere: the interleaving-gap reproducer and the wip-branch ports (both on https://github.com/CodexCoder21Organization/UrlProtocol/issues/424).

Reusable / operational knowledge (adds to the prior handoff's list, which remains valid)

The queue's rotating-flake eviction pattern: a proven-green PR evicted each cycle by a different pre-existing test is a suite-hygiene signal, not a PR problem. Record each cycle's failure list; the rotation itself is the diagnosis.
Verify deployed code by artifact contents, not by restart/upload timestamps — a netlab JAR uploaded today still contained the years-old protocol class.
Test waits must observe published signals, not implementation flags — the shard-load race was fixed by converting 5-second flag-poll spins to 5-second latch waits (same bounds, deterministic observation); reproduce such races with genuine concurrent load (parallel compilers/tests), never artificial spinners.
build-watchman 0.0.13 --to-merged attaches cleanly to an existing queue entry after a shepherd restart, auto-re-enqueues exactly once on silent eviction, and its exit is an actionable event, not evidence of watching.
The buildtest test-results API paginates (first 100 only) — "frozen at 100/100" with a growing build.log is the pagination artifact; read /root/buildtest-data/runs/<id>/test-events.jsonl on the host for ground truth.
gh pr edit --body-file fails on this box's gh (deprecated projectCards GraphQL); use the pull-request REST endpoint and read back.