← Priority list

Handoff: UrlResolver main-branch flake sweep — all fixes done & pushed, merge train blocked on repository-wide remote-CI instability

Original request: *"Looks like there are some flakes on the main branch? We need to use our flakey test skill to resolve: https://github.com/CodexCoder21Organization/UrlResolver/runs/86599526915"*, expanded via `/goal` to: *"ensure all flakes are fixed (including proposing PRs for fixing the underlying infrastructure if that's the problem). Ensure your PR to resolve the flakes is consistently green, and then merge them, ensure they get merged."* **Merging is pre-authorized** for this flake-fix train.

hf-2026-07-12-urlresolver-flake-sweep-merge-train-blocked-remote-ci-instability

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver main-branch flake sweep — all fixes done & pushed, merge train blocked on repository-wide remote-CI instability

Written 2026-07-14 ~17:05 UTC.

RE-VERIFY BANNER: Everything below is a write-time snapshot. PRs merge, CI re-runs, branches get force-pushed, buildtest droplets auto-terminate, and the remote-CI weather changes hour to hour. Before acting on any line here, re-check live:

  • Any PR: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,headRefOid,mergedAt
  • A buildtest run's ground truth (if you have Fabric/SSH access): on the prod host ssh -p 23 root@198.199.106.165 under /root/buildtest-data/runs/<runId>/ (run.json, test-events.jsonl, test-results.json, build.log).
  • A fresh query always wins over this document.

Mission summary

Original request: "Looks like there are some flakes on the main branch? We need to use our flakey test skill to resolve: https://github.com/CodexCoder21Organization/UrlResolver/runs/86599526915", expanded via /goal to: "ensure all flakes are fixed (including proposing PRs for fixing the underlying infrastructure if that's the problem). Ensure your PR to resolve the flakes is consistently green, and then merge them, ensure they get merged." Merging is pre-authorized for this flake-fix train.

Scope grew over the session from the two original flakes to the full set of live flakes on UrlResolver main, plus two upstream/infrastructure defects the flakes exposed. Every code fix is now written, locally verified under a reproducing profile, pushed, and in a PR. The train is BLOCKED at the final gate — not by our code, but by repository-wide remote kotlin.build CI instability: over the last several hours, every UrlResolver full-suite run has failed with a different set of unrelated flakes (wedged/frozen runs, 8-minute droplet-provisioning stalls, 0 ms Coursier classpath-resolution failures, frozen test-event counters). The targeted fix in each PR passes; the surrounding suite does not stay green long enough on one run to merge.

This is the same remote-CI-infrastructure problem described in the parallel handoff 2026-07-14-buildtest-ci-flakes-and-runner-parallelism-gaps.md (a different session working buildtest sharding/crash-recovery). Our train cannot merge until that infrastructure recovers or a green window appears.

Current verified state (2026-07-14 ~17:00 UTC)

Merged this session (6 — verified MERGED)

  • https://github.com/CodexCoder21Organization/UrlProtocol/pull/359 — MERGED 2026-07-13T00:55Z (original withdrawal-chain flake, protocol side)
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/757 — MERGED 2026-07-13T02:38Z (announcement registry resurrection race)
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/766 — MERGED 2026-07-13T07:43Z (relay preference / retry preferred candidates)
  • https://github.com/CodexCoder21Organization/UrlResolver/pull/747 — MERGED 2026-07-13T11:57Z (withdrawal durability under shared gossip dial)
  • https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/177 — MERGED 2026-07-13T03:38Z (CI-dispatch infra fix)
  • https://github.com/CodexCoder21Organization/sandboxjvm/pull/93 — MERGED 2026-07-14T09:39Z (upstream SJVM interpreter speedups; see deep-map chain below)

Open, code-complete, blocked on remote CI (all pushed)

| PR | Branch / head | What it fixes | Own diff | Blocker | |---|---|---|---|---| | https://github.com/CodexCoder21Organization/UrlResolver/pull/774 | fix/deep-map-rejection-vs-sjvm-timeout-race @ 230ec17d | deep-map depth-rejection vs SJVM 5 s timeout race | GREEN (target passes 10/10 local + every remote attempt) | unrelated repo-wide suite flakes | | https://github.com/CodexCoder21Organization/UrlResolver/pull/775 | test/fixed-runner-failure-reporting @ 805cc353 | bump UrlResolver to kompile-cli 0.0.83 (structural failure reporting) | GREEN on target | depends on kompile-cli #127 + remote CI | | https://github.com/CodexCoder21Organization/kompile-cli/pull/127 | (CLI 0.0.83) | upstream structural child→parent failure-report decoding | bld-build-fat-jars PASS | remote kotlin.build pending/flaky | | https://github.com/CodexCoder21Organization/UrlResolver/pull/770 | fix/gossip-storm-retained-heap @ 2acea350 | gossip-storm retained-heap bound + progress-anchored queue-pressure guard (rounds 1–4) | GREEN on diff | unrelated main-side flakes | | https://github.com/CodexCoder21Organization/UrlResolver/pull/771 | fix/fresh-join-withdrawal-setup-miss @ 01c18f96 | fresh-join registration delivery durable under contention | GREEN on diff | unrelated main-side flakes | | https://github.com/CodexCoder21Organization/UrlResolver/pull/772 | fix/peer-exchange-discovery-load-margin @ 5a5012a2 | peer-exchange discovery stress guard load-robust | GREEN on diff | unrelated main-side flakes | | https://github.com/CodexCoder21Organization/UrlResolver/pull/773 | fix/same-tier-relay-winner-under-load @ a33df0a5 | same-tier relay guard anchored to healthy handshake | GREEN on diff | unrelated main-side flakes | | https://github.com/CodexCoder21Organization/UrlResolver/pull/765 | test/issue-357-rereg-equal-timestamp @ c03c41b1 | immediate re-registration after withdrawal | held deliberately (merges LAST) | merge order + remote CI |

No live automation remains — all codex lanes and all build-watchman watchers have wound down/been stopped as of ~17:00 UTC. Nothing is actively driving these PRs right now. Nothing is unsaved: every branch above is pushed and clean (git status clean, local head == pushed head).

What was found and done (the chain)

  1. Original two flakes fixed and merged — withdrawal-loss chain (https://github.com/CodexCoder21Organization/UrlProtocol/pull/359 + https://github.com/CodexCoder21Organization/UrlResolver/pull/757 + https://github.com/CodexCoder21Organization/UrlResolver/pull/747) and relay-preference (https://github.com/CodexCoder21Organization/UrlResolver/pull/766).

  2. Full flake matrix enumerated from ~20 recent main runs. Test-hardening fixes (770/771/772/773) all use the same durable pattern: replace wall-clock waits with causal anchors (observe the system's real progress events instead of "assume X within N seconds"), so slow-but-correct runs pass under CPU starvation while genuine hangs still fail. 770's queue-pressure guard took four hardening rounds (rate-window → combined-bound restatement → progress-anchored 20 s wait) each validated under two-core-pinning + spinners.

  3. Deep-map race — the CI-wide blocker — root-caused and fixed upstream. testDeeplyNestedMapReturnIsRejectedNotStackOverflow failed ~75% on CI because on slow runners the required full depth-limit rejection lost a race against the sandbox's unchanged 5 s SJVM invocation deadline. Reproduced 10/10 by pinning every BootstrapRunner thread to one CPU with 48 equal-priority spinners; interrupted stacks proved the 5 s was spent in interpreted Avian HashMap construction inside the sandbox (host-side depth walk never started). Fixed at the owning layer in https://github.com/CodexCoder21Organization/sandboxjvm/pull/93 (loaded-class cache bypasses the loader mutex; native HashMap(int)/put), published SJVM 0.0.47, and — because 0.0.47 alone still lacked margin — also corrected the test fixture to build the 1,001-level map in bounded setup calls so the timed assert only measures marshaling. All 1,063 UrlResolver test-script SJVM pins aligned to 0.0.47 (a partial bump emitted 3,186 mismatch warnings). Full narrow detail in the sub-handoff 2026-07-14-urlresolver-deep-map-rejection-ci-blocked.md.

  4. Test-runner serialization defect — infrastructure fix. Three independent signatures (a RelayException frame the parent couldn't decode; a SerializableFailure → Throwable ClassCastException found by host forensics; the org-wide pattern of real failures masked as bare 30 s timeouts) converged on the kompile test-runner: child-JVM failure reports carrying exception classes the parent can't load crashed the decoder. Fixed to decode failure reports structurally (class name + message + stack + cause chain as data, never loading test-project exception classes into the parent). Published through the runner chain (bootstrap 0.0.13, JVM runner 0.0.21, execution-environment 0.0.30, manager 0.0.118, kompile-cli 0.0.83 = https://github.com/CodexCoder21Organization/kompile-cli/pull/127) and consumed in https://github.com/CodexCoder21Organization/UrlResolver/pull/775. Proven end-to-end: the previously-trapped relay-failover stress test now surfaces real failures / passes cleanly (82.9 s) through the fixed CLI instead of wedging into a 30 s timeout.

  5. Active-query TTL flake — investigation closed STUCK, no guessed fix. testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup would not reproduce: 40/40 passes across escalating starvation profiles (two-core, then one core + 8/48/192 spinners). Per the reproduce-first rule, no fix was written; the investigation is preserved in sub-handoff 2026-07-14-urlresolver-active-query-ttl-flake-reproduction-blocked.md (PlanRepository PR #1380). Parked pending https://github.com/CodexCoder21Organization/UrlResolver/pull/771's merge (its registration-durability fix targets the most plausible mechanism); re-attempt with a different amplifier if it recurs on fresh main afterward.

  6. The final blocker is remote-CI infrastructure, not our code. Deep-map lane logged five consecutive 774 full-suite attempts (993c8d94, 23f6d1ca, 43426389, 9603a2b0, c76e3ea2), each 1090–1097 / 1102 with a substantially different failing set each time, plus 0 ms Coursier non-executions and a counter frozen at 0/100 for 20 min on a run that nonetheless produced all 1102 outcomes. Runner-serialization lane hit genuine wedges (955fac54, 963b1dfc watchdog-failed after correlated 8-minute droplet-provisioning stalls; 6fd18eb7, 631d0a91 frozen >20 min). Challenges filed: https://github.com/CodexCoder21Organization/PlanRepository/pull/1392 and PlanRepository challenges/2026-07-14-1647-buildtest-simultaneously-watchdog-failed-fresh-cli-and.md.

Relevant PRs / refs

  • Sub-handoffs (narrow, already merged into handoffs/): deep-map (PlanRepo #1398), active-query (PlanRepo #1380).
  • Parallel infra effort (different session, same remote-CI root cause): buildtest CI flakes + runner parallelism.
  • Challenge records: https://github.com/CodexCoder21Organization/PlanRepository/pull/1392 + challenges/2026-07-14-1647-…, challenges/2026-07-14-0745-kompile-test-runner-crashes-while-decoding.md, challenges/2026-07-14-0643-kompile-local-test-dependency-resolution-hung-indefinitely.md, challenges/2026-07-14-0522-exporting-an-isolated-home-does-not-isolate-coursier.md.

Next steps (for whoever picks this up)

  1. Wait for a green remote-CI window, then merge in dependency order. The code is done; this is a scheduling problem. When kotlin.build is healthy (verify via the buildtest-ci-flakes handoff's host checks or simply by watching a couple of clean full-suite runs):
    • First: https://github.com/CodexCoder21Organization/kompile-cli/pull/127 (CLI 0.0.83) → then https://github.com/CodexCoder21Organization/UrlResolver/pull/775 (consumes it).
    • Then: https://github.com/CodexCoder21Organization/UrlResolver/pull/774 (deep-map — this is the one that unblocks every other UrlResolver PR's CI; it rides merged sandboxjvm 0.0.47).
    • Then rebase & rerun https://github.com/CodexCoder21Organization/UrlResolver/pull/770, https://github.com/CodexCoder21Organization/UrlResolver/pull/771, https://github.com/CodexCoder21Organization/UrlResolver/pull/772, https://github.com/CodexCoder21Organization/UrlResolver/pull/773 onto the new main (774 + 775 remove their two biggest unrelated-flake sources: the deep-map failure and the timeout-masking).
    • Last: https://github.com/CodexCoder21Organization/UrlResolver/pull/765. Re-arm one build-watchman --to-merged per PR (coursier launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr <N> --to-merged, backgrounded, one per PR).
  2. Do NOT re-roll blindly. The standing rule this session used: diagnose every red run (infra-signature vs. real); a third occurrence of any specific test failure is fix-lane work, not a re-roll. Every 774/770/771/772/773 red so far has been classified as unrelated main-side or infra flakes — do not weaken/skip/delete any test to "make it pass".
  3. Post-merge: run a fresh main-run observation pass; if testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup, testPersistentRpcReconnectDoesNotBlockConcurrentCaller, testRequestTimeoutDoesNotResendAndDoubleInvoke, stressTestEagerJoinDoesNotRaceWithExplicitJoinNetwork, or testReconnectRecoversViaAlternativeProvider still flake, they are the next fix lanes (the runner-serialization fix in 775 may make several of them finally surface their real failure instead of a 30 s timeout).
  4. When the whole train is merged and main is clean, delete this handoff and both sub-handoffs.

Reusable / operational knowledge

  • Reproduce-first amplifiers that worked here: two-core taskset pinning + one equal-priority spinner per core, escalating to one core + N spinners (48 got deep-map to 10/10; active-query stayed 0/40 even at 192 → correctly declared STUCK rather than guessing).
  • Isolated HOME is not enough — Coursier caches under the Java user.home, not $HOME. Use isolated HOME and COURSIER_CACHE and -Duser.home (challenge 2026-07-14-0522). Shared ~/.aibuildcaches gets cleared by other sessions mid-run; never rm -rf it.
  • Remote-CI wedge vs. slow: a WEDGE requires zero test-event progress and frozen build.log mtime and frozen build.log content for >20 min while pending; a frozen visible counter alone is not a wedge (one run showed 0/100 for 20 min yet produced all 1102 outcomes). Only the single delete/rerequest owner may delete a run, and only with host stat/tail confirmation — this environment lacks Fabric mTLS certs, so host verification was often unavailable and runs were left for buildtest to terminate.
  • Causal-anchoring pattern (the durable flake-guard fix used 5× this session): replace any fixed wall-clock window with observation of the system's own progress events — deliveries advancing ⇒ keep waiting; no progress for the bounded window ⇒ fail. Never raise a timeout; never reduce stress iterations.
  • Fix upstream, never mitigate downstream: both infra defects (SJVM interpreter speed, test-runner failure-report decoding) were fixed at their owning repos and published, then consumed by version bump — not worked around in UrlResolver.