HHandoff
← Priority list

Handoff: Observables tail — flaky-test fix suite (#749 MERGED; #748 → #751 stuck on fleet-wide CI flakiness, decision pending), #680 cascade-owned, then close out the 2026-07-09 observables handoff

Standing directives inherited from the 2026-07-09 handoff: root-cause fixes only (never band-aids / never a timeout-or-retry mitigation of an unexplained contradiction); never blind-retry CI — diagnose every eviction across all three watch layers; codex `gpt-5.6-sol` for heavy lifting; and **merge-when-genuinely-green is pre-approved** (so merging these PRs is *my* remaining step, not a user blocker). The 2026-07-09 handoff closed 16 PRs and all four infra issues; its only open items were the **#707 → #680 merge tail** and a DODSS pacer re-land. This session discharged #707, root-caused and fixed a *residual* resurrection race the earlier fixes did not close ([issue #746](https://github.com/CodexCoder21Organization/UrlResolver/issues/746) → [PR #749](https://github.com/CodexCoder21Organization/UrlResolver/pull/749)), and is shepherding a three-PR flaky-test fix suite — **[#749](https://github.com/CodexCoder21Organization/UrlResolver/pull/749) → [#748](https://github.com/CodexCoder21Organization/UrlResolver/pull/748) → [#751](https://github.com/CodexCoder21Organization/UrlResolver/pull/751), landed serially** — plus #680, to merge so the handoff can be closed out.

hf-2026-07-12-observables-tail-749-substream-fix-680-closeout

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: Observables tail — flaky-test fix suite (#749 MERGED; #748 → #751 stuck on fleet-wide CI flakiness, decision pending), #680 cascade-owned, then close out the 2026-07-09 observables handoff

Written: 2026-07-12 ~16:55 UTC. Updated: 2026-07-12 ~21:55 UTC — #749 is now MERGED; #748 rebased onto the new main (version renumber 0.0.626 → 0.0.629) and pushed, but is now stuck on a fleet-wide CI-flakiness gauntlet and has gone DIRTY again after #653 merged; a strategic decision (grind vs. pivot to the CN-relay infra root cause) is PENDING user input. (Prior update ~17:10 folded in the full serial suite + the merge-queue kick-churn root cause + the auto-re-enqueue loop.) Thread began 2026-07-09. Directly continues 2026-07-09-observables-close-16-merges-fleet-ci-fixed-707-680-tail.md — that handoff's #707/#680 merge tail is the only part still open, and this snapshot supersedes it for that tail. When the work below is fully resolved, both this handoff and the 2026-07-09 one should be deleted.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below is a write-time snapshot. PRs merge, the UrlResolver merge queue reshuffles minute-to-minute (multiple parallel sessions enqueue into it), CI re-runs, branches get force-pushed, and prod services get redeployed by other sessions (coordination file: /root/cn-watchdog.audit on 198.199.106.165:23, key ~/.ssh/id_ed25519_new). Authoritative checks:

  • PRs: gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • UrlResolver queue: gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:10){nodes{position pullRequest{number} state}}}}}'
  • DO rate-budget gate (must be healthy before any enqueue): on the host, tail -80 /root/ContainerNursery/data/container-logs/url_digitalocean-droplets_.stdout.log | grep -cE "budget.*exhausted|429" — a non-zero count means hold enqueues.

A fresh query always wins over this document.

Mission summary

Standing directives inherited from the 2026-07-09 handoff: root-cause fixes only (never band-aids / never a timeout-or-retry mitigation of an unexplained contradiction); never blind-retry CI — diagnose every eviction across all three watch layers; codex gpt-5.6-sol for heavy lifting; and merge-when-genuinely-green is pre-approved (so merging these PRs is my remaining step, not a user blocker). The 2026-07-09 handoff closed 16 PRs and all four infra issues; its only open items were the #707 → #680 merge tail and a DODSS pacer re-land. This session discharged #707, root-caused and fixed a residual resurrection race the earlier fixes did not close (issue #746PR #749), and is shepherding a three-PR flaky-test fix suite — #749#748#751, landed serially — plus #680, to merge so the handoff can be closed out.

The three-PR suite (how the fixes fit together):

  • #749 — sends each gossip frame on its own libp2p substream so batched service withdrawals stop being silently dropped (the resurrection race; full mechanism below).
  • #748 — keys dial dedup on refreshed advertised addresses instead of bare peer ID, so a retry to a peer that has moved actually proceeds instead of being deduped away against a stale in-flight dial.
  • #751 — pins both behaviors with a deterministic seam + regression tests ("retry stable peers after advertised address refresh").

They are independent branches off main (none is stacked on another — verified baseRefName=main for all three), but they touch overlapping relay/announcement code, so they are landed one at a time, rebasing the next onto the new main to avoid semantic conflicts. Serial order is intentional, not a dependency chain.

What was found and done (the chain)

  1. UrlResolver #707 (idempotent projection command dispatch) is MERGED (2026-07-11T19:20:26Z), after 4 CI attempts each of whose evictions was diagnosed (not blind-retried). The A/B gate the 2026-07-09 addendum required — proving the residual resurrection flake was in main and not introduced by #707 — was satisfied by production evidence: buildtest run eab27546 on main + a test-only branch reproduced the RESURRECTION bucket with #707 absent, exonerating #707 and motivating item 2. That evidence was filed as issue #746.

  2. Residual withdrawal-resurrection race ROOT-CAUSED with captured traces, and fixed in PR #749. This is the substantive new work of the session.

    • Mechanism (verified via in-memory ring-buffer traces, not speculation): the gossip transport protocol closes each libp2p substream after a single message (UrlProtocol RpcServerMessageHandler.channelRead0closeStreamQuietly(), pin f3993e3/v0.0.269). UrlResolver's queue drainer, however, batched multiple queued frames onto one substream. The receiver therefore silently discarded every frame after the first — and when frame 2 was a one-shot service withdrawal, the withdrawal was dropped while the sender recorded it FLUSHED/SENT. The withdrawn service then "resurrected" on the next announcement. Stdout logging heisenbugged the race away (3/3 debug-enabled runs passed), which is why tracing is a dump-on-failure ring buffer.
    • Fix: the drainer now sends each queued message on its own substream (sendGossipBatchToPeerAsync calls sendGossipToPeerAsync per message, awaiting each writeAndFlushWithFuture before opening the next), and threads a per-message GossipSendOutcome (SENT / DEFERRED_PENDING_DIAL / FAILED) back through an onOutcome(index, outcome) callback. Only the unsent suffix is retried (no whole-prefix replay → no stream-churn amplification), and a still-pending deduplicated dial is represented as DEFERRED_PENDING_DIAL (retryable) rather than being miscounted as a successful delivery. Transport failure cause is preserved into drainer diagnostics (commit 550344a2, added after a test caught a dropped ": <cause>" suffix).
    • Two adversarial review cycles were addressed (see REVIEW-FINDINGS.md / REVIEW-FINDINGS-V2.md in the working clone): (BLOCKER-1) a "direct" send routed back through the relay branch → fixed with true direct-stream routing; (BLOCKER-2) a pending dedup-dial returned success → fixed with the DEFERRED_PENDING_DIAL outcome + suffix-only retry.
    • Verification was load-audited (a per-run /proc/loadavg sampler; only quiet runs with peak load < 14 counted, in both directions): 6 amplified + 8 guard = 14/14 counted passes, script and results at final-verify3.sh/final-verify3.results in the scratchpad. The amplified reproducer tests/stressTestFreshJoinWithdrawalDeliveryUnderConcurrentReannounce.kts (48 fresh-join trials/run) is ~75% red on unfixed main, 0% red with the fix.
  3. PR #749 is MERGED (2026-07-12T17:35:10Z, merge commit 95e53898, now on main). It got through by catching a rare zero-flake CI window. The auto-re-enqueue loop (autoq_749.sh) did its job: #749 was an innocent bystander repeatedly evicted whenever a sibling in its speculative merge group failed the group check (it stayed CLEAN each time), and the loop re-added it within ~2 min of each CLEAN-state kick (3 kicks recovered) until a clean group window landed it. Lesson banked: a passive watcher that keys "actionable" on mergeStateStatus != CLEAN misses these silent kicks (that cost #749 ~6h idle in one window earlier); the auto-re-enqueue loop is the fix, and it is in Reusable knowledge below.

  4. #748 was rebased onto the post-#749 main and pushed, then hit the fleet-wide CI-flakiness gauntlet, then went DIRTY again — it is NOT landed. Detail:

    • After #749 merged, #748 went DIRTY/CONFLICTING. Rebased fix-peer-scoped-relay-onboarding onto the new main in a fresh clone. The only real conflict was a version-coordinate collision in build.kts — #748 claimed 0.0.626 (now reserved by #680), #749 took 0.0.627, #751 claims 0.0.628, so #748 was renumbered to 0.0.629 with a changelog note. UrlResolver.kt auto-merged cleanly — #748's changes (sendRpcRequest relay-fallback + circuit-address-synthesis rewrite) sit in different regions than #749's gossip-drainer changes; git diff origin/main...HEAD showed only #748's coherent changes. Force-pushed the clean 3-commit stack (was head eba2e51f57197f66), which includes #748's own 2 regression tests (testRelayReannounceOnlyAdvertisesConfirmedRelay.kts, testSendRpcRequestFallsBackToConfirmedRelay.kts).
    • Branch CI on 57197f66: bld-build SUCCESS both times, kotlin.build (remote) FAILURE twice — and the two failure sets were disjoint (run 1: testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup, stressTestConcurrentLateJoinersDiscoverServicePromptly; run 2: testDeeplyNestedMapReturnIsRejectedNotStackOverflow, testRelayClientServiceDiscoveryThroughGossip, testFindPeersForServiceQuiescencePolling, testPeerExchangeFilterSharesPeerAfterSpuriousFailure, stressTestFreshJoinWithdrawalDeliveryUnderConcurrentReannounce, testActiveServiceQueryFindsServiceOnBootstrapNode). This is diagnosed as fleet-wide flakiness, NOT a #748 regression — see item 5.
    • As of 21:50 UTC #748 is DIRTY/CONFLICTING AGAIN: #653 ("Fail loud on nested unmarshalable values", commit b3760739, touches build.kts) merged at 20:36 and re-conflicts #748's build.kts. #748 needs ANOTHER rebase onto current main (b3760739). This is a moving target — every build.kts-touching PR that lands re-DIRTIES #748.
    • #751 (fix/stable-peer-address-refresh, head ac88f7b8, unchanged): DIRTY, both bld-build and kotlin.build (remote) FAILURE (fleet-wide flake, same as #748). Last in the serial plan; needs rebase onto main + a genuinely-green run (expect the same flake gauntlet). Will also need a version renumber (it claims 0.0.628; recheck the ledger after #748 lands).
  5. The suite is now bottlenecked on FLEET-WIDE CI FLAKINESS, and a strategic decision is PENDING the user. Evidence it is fleet-wide infra flake, not a suite regression: (a) #748's two branch runs produced disjoint random failure sets, ~2–6 tests out of 1090 each; (b) those same tests fail on unrelated open PRs (testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup on #761/#737; stressTestConcurrentLateJoinersDiscoverServicePromptly on #662; a scan of ~15 open PRs shows every one failing a different shifting set of timing/stress/netlab tests); (c) #748's own new tests never appear in any failure set; (d) the local bld-build tier passes — only the remote buildtest tier (on the CPU-starved DO fleet) flakes; (e) it is NOT the DO-429 "0 passed/0 failed" signature — real runs, just flaky timing tests. Root cause is the documented CN-relay CPU starvation (kotlin.directory / buildtest co-hosted on the relay box at load ~100 → ~50-min test runs → timing assertions flake), i.e. the same infra issue behind issue #754. Consequence — the merge queue is a "flake gauntlet": both the branch check AND the merge-group check re-run the full 1090-test suite on the starved fleet, so #748/#751 will be evicted repeatedly until each catches a rare zero-flake window (how #749 slipped through). Decision put to the user (interrupted before answering): (A) grind — keep re-running + re-enqueuing until each catches a clean window (pure mitigation; many hours; each re-run adds load to the already-starved fleet); vs (B) pivot to root cause — land the CN-relay deep fix (jvm-libp2p PR#23 yamux O(n²) → publish snapshot → re-pin fleet; mostly written but unlanded, needs user merge approval), which fixes flakiness for the whole fleet and aligns with the root-cause-over-band-aid directive; vs (C) both in parallel. See [[fixA-cn-relay-yamux-oom-mostly-already-in-pr23-2026-07-12]] / [[cn-tls-slowness-relay-cpu-starvation-2026-07-12]] in auto-memory for the infra fix's current state. No grind is running right now — all watchers exited; nothing is auto-re-running #748/#751 pending the decision.

  6. PR #680 (Phase-C resilience acceptance fixture) shepherding was CEDED to the observables-cascade session, which owns its current head. This session had rebased and proven it green twice earlier; ceding avoided two sessions force-pushing the same branch. At write time it is mergeStateStatus=BLOCKED. #680 must merge for the 2026-07-09 handoff to be closable.

  7. Complementary receiver-side fix PR #757 ("Close residual announcement registry resurrection race", another session) attacks the same issue #746 from the receiver side. #749 (sender-side substream fix) and #757 (receiver-side registry fix) are complementary; #746 should be closed only once BOTH land. #757 and #749 conflict in UrlResolver.kt, so whichever merges first makes the other DIRTY — see Next steps for the rebase drill. At write time #757 is OPEN, CLEAN.

  8. Cleanups verified done: the netlab-manager bytecode-serve OOM handoff this session was tasked to finish was driven to completion and deleted (its work fully resolved). No leftover boottest droplets; workspace fat jars gone; item-7 SSH-window flake discharged via BuildTestEmbedded#282; the DODSS pacer re-land was already merged by a prior session.

  9. Fleet-wide storm flake characterized (not this session's to fix): issue #754testGossipBroadcastStormWithStalledPeersBoundsQueuedMemory flaking ~8/14 recent runs across many PRs. The initial "#738 flush-await regression" hypothesis was refuted by full-history analysis (the failure ran ~53% before #738 merged); a correction was posted to the issue. It is longstanding and CI-only; documented for whoever picks it up, no further obligation claimed here. Note: this is a plausible cause of #751's current red — check before assuming #751 has its own defect.

Relevant PRs / refs (write-time state — RE-VERIFY)

| PR / ref | Repo | State @ write | Link | |---|---|---|---| | #707 idempotent command dispatch | UrlResolver | MERGED 07-11 19:20Z | https://github.com/CodexCoder21Organization/UrlResolver/pull/707 | | #749 substream-per-message race fix | UrlResolver | MERGED 07-12 17:35Z, commit 95e53898 | https://github.com/CodexCoder21Organization/UrlResolver/pull/749 | | #748 peer-scoped relay address announcements | UrlResolver | OPEN, DIRTY/CONFLICTING (needs re-rebase onto b3760739); rebased once to 0.0.629 head 57197f66; branch CI bld-build✅ / kotlin.build (remote)❌ (fleet flake) — 2nd in serial plan | https://github.com/CodexCoder21Organization/UrlResolver/pull/748 | | #751 retry stable peers after addr refresh | UrlResolver | OPEN, DIRTY, both checks FAILURE (fleet flake), head ac88f7b8 — 3rd/last, needs rebase + green | https://github.com/CodexCoder21Organization/UrlResolver/pull/751 | | #680 Phase-C resilience fixture | UrlResolver | OPEN, BLOCKED — cascade-session owns (re-verify) | https://github.com/CodexCoder21Organization/UrlResolver/pull/680 | | #757 receiver-side registry race fix | UrlResolver | OPEN (was CLEAN) — other session; conflicts with #749 in UrlResolver.kt; #749 merged first so #757 is now likely DIRTY — re-verify | https://github.com/CodexCoder21Organization/UrlResolver/pull/757 | | issue #746 resurrection race | UrlResolver | OPEN — #749 (sender side) landed; close only when #757 (receiver side) also lands | https://github.com/CodexCoder21Organization/UrlResolver/issues/746 | | issue #754 storm flake | UrlResolver | OPEN — longstanding; part of the same fleet-wide flakiness now blocking #748/#751 | https://github.com/CodexCoder21Organization/UrlResolver/issues/754 | | branch fix-peer-scoped-relay-onboarding | UrlResolver | pushed, head 57197f66 = #748 (rebased once, 0.0.629); STALE vs current main — re-rebase | (PR #748 branch) | | branch fix/stable-peer-address-refresh | UrlResolver | pushed, head ac88f7b8 = #751 head (not yet rebased) | (PR #751 branch) | | main commits landed after #749 | UrlResolver | c3b8c4ba (#632 netlab R4), 7071029854 (#752 relay-restart #742), b3760739 (#653 fail-loud) — the last re-DIRTIED #748 | — |

Merged earlier this session (foundational, verified MERGED): UrlResolver #750 (07-11 19:24Z, removed dead eager encoding of an always-empty peer collection on the Netty event loop — ended broad test stalls), #667 (07-11 19:26Z), #672 (07-11 19:24Z); PlanRepository #816, #991 (netlab-OOM handoff deletion), #743, #1009.

Monitors running at write time: NONE — all background watchers have exited (the #749 auto-re-enqueue loop finished when #749 merged; the #748 land-watchers brkzx9aol/b3x2cteco exited on the two flake reds). Nothing is currently auto-driving #748/#751 — that is intentional, pending the grind-vs-pivot decision (item 5). Reusable watcher scripts live in this session's scratchpad: autoq_749.sh (auto-re-enqueue), land_748_v2.sh (wait-CI-green → enqueue → shepherd, capturing failing-test summaries on red).

Working clone (this session): ~/workspace/UrlResolver-748 on branch fix-peer-scoped-relay-onboarding @ 57197f66 (the once-rebased 0.0.629 #748). It is STALE vs current main (b3760739) — re-rebase before reusing (and purge the aibuildcaches per Reusable knowledge). The ~/workspace/UrlResolver-race-fix / -ab-707 clones referenced in prior revisions were on a different host/session.

Next steps (ordered — re-verify current state FIRST)

  1. Re-verify everything with the banner commands before acting — especially whether #748/#751 are still DIRTY, what merged into main since b3760739, and the current queue. #749 is MERGED (done); the moving target is #748/#751 vs. an advancing main.
  2. RESOLVE THE PENDING DECISION FIRST (item 5 above) — do not just start grinding. The user was asked grind (A) vs pivot to the CN-relay infra root cause (B) vs both (C) and was interrupted before answering. The root-cause-over-band-aid directive and the fact that re-runs worsen the shared starvation both point at (B) or (C). Whoever picks this up should get that steer (or, if acting autonomously under the pre-approval, prefer B/C over an open-ended grind that hammers the starved fleet).
  3. Path A / C — land #748 through the flake gauntlet: (a) re-rebase fix-peer-scoped-relay-onboarding onto current main in ~/workspace/UrlResolver-748 (git fetch origin && git rebase origin/main); the conflict will again be the build.kts version coordinate — pick the next free number after re-reading the changelog ledger (currently #748=0.0.629, #751 claims 0.0.628; bump if #653/others took 0.0.629). Purge caches (rm -rf ~/.aibuildcaches/_code_workspace_UrlResolver-748 ~/.cache/coursier/v1/bldbinary/repository/foundation.url) before any local build. (b) git push --force-with-lease. (c) Because branch CI and the merge-group both re-run the flaky 1090-test suite, use land_748_v2.sh (in scratchpad) which waits for branch green then enqueues then auto-re-enqueues on innocent kicks. On a red, verify it is a flake before re-running (compare the failing set to the prior run + to other open PRs; a repeat of the same test deterministically = investigate as a real regression via codex gpt-5.6-sol, do NOT keep blind-re-running). (d) On #748 MERGED, repeat the whole drill for #751 (rebase → renumber → green → enqueue).
  4. Path B — fix the fleet-wide flakiness at the source (recommended, aligns with the standing directives): land the CN-relay CPU-starvation deep fix — jvm-libp2p PR#23 (yamux ByteBufQueue.take O(n²), ArrayDeque, 42.4s→0.166s; OPEN/green/mergeable per auto-memory) → publish snapshot → cascade re-pin (UrlProtocol/UrlResolver/CN), and stop relaying JAR bootstrap over the relay. This needs user merge approvals and is a bigger cascade, but it unblocks #748/#751 (and every other PR) by making CI fast again. See auto-memory [[fixA-cn-relay-yamux-oom-mostly-already-in-pr23-2026-07-12]], [[cn-tls-slowness-relay-cpu-starvation-2026-07-12]], [[buildtest-slow-concurrency-coursier-resolve-on-cn-relay-starvation-2026-07-12]]. Do NOT ship a per-test timeout/retry band-aid — that mitigates the symptom and strips the diagnostic signal (see the ProductionHealth case study in the global CLAUDE.md).
  5. Housekeeping once #748/#751 land: update issue #746 (sender-side #749 landed; still open until receiver-side #757 lands — re-verify #757's state, it likely went DIRTY when #749 merged first).
  6. Monitor #680 to MERGED (cascade-session-owned — coordinate via /root/cn-watchdog.audit, don't force-push their branch). Re-verify it is still that session's; if abandoned, adopt it.
  7. When #748, #751, and #680 are all MERGED (and #746 closable) → close out the handoffs. Re-verify each is genuinely merged (gh pr view <n> --json state,mergedAt), then via the create-handoff skill's completion path delete both this file and 2026-07-09-observables-close-16-merges-fleet-ci-fixed-707-680-tail.md (git rm on a branch, open the deletion PR, enqueue + merge under the standing handoff pre-approval), and give a final report.

Reusable / operational knowledge

  • Merge-queue "innocent bystander" evictions & the auto-re-enqueue loop: in a congested UrlResolver queue, a CLEAN PR is repeatedly dropped whenever a sibling in its speculative merge group fails the group check — the victim stays CLEAN, just silently leaves the queue. A passive watcher that keys "actionable" on mergeStateStatus != CLEAN will miss this and let the PR sit un-enqueued for hours. Fix: a loop that treats "was enqueued, now OPEN + CLEAN + not-in-queue" as an auto-re-enqueue trigger and only escalates on non-CLEAN/MERGED. Reference script autoq_749.sh (polls gh pr view <n> --json state,mergeStateStatus + the mergeQueue GraphQL every ~120s; re-enqueues via enqueuePullRequest; caps re-enqueue attempts by counting them). Do NOT auto-re-enqueue a non-CLEAN PR — that would mask a real red or a needed rebase.
  • aibuildcaches stale-module trap (bit this session 3×): ~/.aibuildcaches/<mangled-workspace-path> memoizes buildMaven() across checkouts and versions — a "succeeded in 53ms" build after a source edit is a stale hit serving the wrong jar (symptoms: unresolved symbols, Too many arguments for sendRequest). After ANY src change, purge both ~/.aibuildcaches/_code_workspace_<clone> and ~/.cache/coursier/v1/bldbinary/repository/foundation.url before local tests. It also silently contaminated codex's verification history once.
  • Load-audited verification is mandatory for this race: a raw "12/12 PASS" against a ~4% flake has only ~61% power. Sample /proc/loadavg per run; discard runs whose peak load ≥ 14 (starvation produces WITHDRAWAL_NEVER_PROCESSED artifacts unrelated to the race); count only quiet runs, and treat a RESURRECTION-bucket failure on a quiet run as a real defect (abort the fix). Script: final-verify3.sh.
  • Never log the race to stdout to "watch" it — it is a scheduler-sensitive heisenbug (3/3 debug-enabled runs passed). Use the opt-in in-memory ring buffer (enableGossipTrace(serviceIdentifier) / getGossipTraceEvents(), dump-on-failure only).
  • DO zero-test 429 signature: buildtest "0 passed, 0 failed, N total" in ~3 min = the shared DO droplet rate budget drained. Confirm on the host with the gate command in the banner; hold enqueues until the exhausted-count is 0. Do NOT delete the run or blind-retry.
  • UrlResolver merge queue: gh pr merge --auto is disabled repo-wide; enqueue via GraphQL enqueuePullRequest (PR node id from gh pr view <n> --json id). Speculative UNMERGEABLE entries behind a conflicting head self-resolve when the head merges/evicts — a harmless idle placeholder, not a failure.
  • Serial landing of independent-but-overlapping branches: #748/#749/#751 all base off main (not stacked), but touch overlapping relay/announcement code. Land one, rebase the next on the new main, re-verify green, enqueue. Don't try to enqueue all three at once — the second and third are BEHIND and one (#751) is currently red.
  • Root-cause discipline that paid off here: the "server recorded FLUSHED, receiver saw the frame vanish" contradiction was the bug — chasing it (rather than mitigating the symptom with a retry/timeout) is what surfaced the one-message-per-substream protocol invariant.
  • The merge queue is a "flake gauntlet" while the fleet is CPU-starved (2026-07-12): both the branch check AND the merge-group check re-run the full ~1090-test buildtest suite on the shared DO fleet. Under the CN-relay CPU starvation (kotlin.directory co-hosted, load ~100), test runs take ~50 min and 2–6 random timing/stress/netlab tests flake per run. So a correct, green-on-bld-build PR still gets evicted repeatedly until it catches a rare zero-flake window (how #749 landed). How to tell flake from regression WITHOUT blind-retrying: (1) compare failing sets across runs — disjoint sets ⇒ flake, a repeated test ⇒ investigate; (2) grep other open PRs' kotlin.build (remote) check summaries for the same test names (they flake fleet-wide); (3) confirm the PR's own new tests never fail; (4) confirm the local bld-build tier passes. If all four hold, it is infra flake — but the honest fix is Path B (fix the starvation), not grinding, because re-runs add load to the very fleet that is starved.
  • kotlin.directory version-probe is UNRELIABLE for "is this version free": curl of …/resolver/0.0.<v>/resolver-0.0.<v>.pom returns an identical catch-all 200 (same byte count) for every version, published or not. Use the build.kts changelog ledger + open-PR branch coordinates as the source of truth for reserved version numbers instead. (Prior revisions' "401 = free" probe no longer discriminates.)
  • #748/#751 are a moving rebase target: every PR that touches build.kts and merges to main re-DIRTIES them on the version-coordinate line (e.g. #653 b3760739 re-conflicted #748 hours after its rebase). Expect to re-rebase + renumber each time; don't treat a single successful rebase as durable.