Handoff: Resolving the open handoffs (Priority-2 campaign) — proven process, verified classification, and the untriaged tail
Written: 2026-07-09 ~04:50 UTC. This is a meta-handoff: its mission is PRIORITIES.md Priority 2 — work through handoffs/, verify what each claims, close out what is resolved, and drive what is still live. It hands over both the process (proven over the 2026-07-07→09 flake campaign) and a write-time classification of the directory.
⚠️ RE-VERIFY BEFORE ACTING
Every classification below is a write-time snapshot. Handoffs are point-in-time by nature; several parallel sessions write and resolve them continuously. Before acting on ANY entry: re-run the authoritative check its handoff names (gh pr view … --json state,mergedAt, service probes, host SSH). A fresh query always wins over this document — and over the handoff it classifies.
✅ RESOLUTION UPDATE (2026-07-09 ~15:15 UTC): all three C-items are resolved
Full evidence in the campaign handoff's resolution update. Summary:
- UrlResolver #727: MERGED 09:04:36Z; resolver 0.0.602 published + verified. (3 queue evictions were all DO droplet-quota 429 storms; post-hourly-reset solo re-enqueue was the tactic that landed it.)
- BuildTestWui #114: merged + deployed — but the dump storm had a second root cause (per-render full-table
getExecutionDurationStats() SJVM marshaling), fixed by #115 (merged 14:57:43Z + deployed 15:03; concurrent-render path verified fixed). Residual: a SINGLE full-table marshal (102MB, table now 8-16k+ entries) still OOMs GET /api/duration-stats per browser page view — needs the upstream windowed abstract-method API (owned by a parallel session, bounded-collections-api branches) before the WUI is fully stable.
- PSI-warmup branch
fix-cancelled-build-flake-psi-warmup (f410fcdc): DO NOT SHIP — 0/10 reproduction on main under the commit's own recipe; the cold-JVM-PSI premise doesn't hold under scripts/test.bash (in-process test running); UPLOADING dominates the contended window; plus a real SHARED_PARSER_WARMUP_THREAD init race in the fix. Branch preserved; next reproducer must run via BuildTestRunner on 2-vCPU droplet-class hardware.
- Backlog item "buildtest-server-service 0.0.230 redeploy": already live (verified via
rateLimitRetryMaxWaitMs symbol in the deployed jar, Jul 8 21:43).
- New upstream defect for the ledger:
uploadFileChunk is not retry-idempotent across LazyReconnectingMethodDispatcher reconnects (Expected chunk index 1 ... but received 0 — cost one queue round; likely explains historical goldenScreenshots failures 61433d7f/c956fc49).
Mission summary
User instruction (this session): "let's create a handoff for resolving these handoffs." Context: the 2026-07-07→09 flake-elimination campaign (campaign handoff) consumed many of the flake-class handoffs; the directory (59 files at write time) still contains a mix of resolved, superseded, and genuinely-live efforts, plus ~10 new handoffs from parallel sessions this triage has NOT assessed. The resolver's job: verify → resolve → close out, honestly.
The proven process (use it — it worked 11 times in 2 days)
- Mine, then verify. Never start work from a handoff's claims. For each candidate: re-check the PRs it names, probe the services, read the code region on current main. Roughly a third of recorded claims were stale or wrong across this campaign (examples: three "top candidates" already fixed; a "not yet written" fix that was merged; two refuted root-cause hypotheses in kompile-core #209 round 3).
- Doctrine precedence. Claims conflicting with TESTING.md are presumptively wrong: "CPU overloaded" is never a root cause; "transient" = un-root-caused flake; OOM = leak/unbounded materialization; a timeout/retry/fallback offered in place of a mechanism is a band-aid.
- Lane pattern, max 3 concurrent. One repo per lane, fresh clone per lane (
/code/workspace/<lane>/<repo>), self-contained brief passed via "$(cat brief.md)" (inline quoting once wedged a lane on stdin for 3.5h). Reproduce-before-fix gate (≥50% or deterministic forced-path; STOP-with-evidence is a valid outcome). Full-suite runs serialized machine-wide via /tmp/claude-test-slot.lock (mkdir-loop acquire / rmdir release even on failure).
- Implementation fleet: codex gpt-5.5 (canonical invocation, reasoning=high) when quota allows — exhausted until ~Jul 13 21:52 at write time;
codex exec resume equally blocked — otherwise Opus subagents (user-authorized 2026-07-09).
- Ship it fully: PR → all checks green → GraphQL
enqueuePullRequest (warm https://github-webhooks.wasmserver.com/ 3× first) → diagnose every eviction from https://buildtest.kotlin.build/api/test-results?id=<runId> (re-enqueue only provably-unrelated infra; batch-poisoning is real — a SOLO re-enqueue is the decisive experiment) → publish (probe https://kotlin.directory/<path>/<v>/<artifact>-<v>.pom first: 200=taken, 401=free) → deploy → verify the production symptom actually stopped (uptime/dump probes, not vibes).
- Close the loop on the handoff itself: when an effort is resolved, update or supersede its handoff (small PlanRepository PR — handoff PRs are pre-approved and merge immediately), close related issues with evidence, and file challenge reports for ecosystem friction.
Write-time classification of handoffs/
A. Resolved by the 2026-07-07→09 campaign (verified — spot-check the named PRs before closing out)
B. Superseded / historical (verify then annotate, no work expected)
C. Still live — campaign in-flight (highest priority for the resolver)
- UrlResolver #727 — rebased head
a2c351d2, reproducer passed; needs checks-green → enqueue → merge → publish resolver 0.0.602.
- BuildTestWui #114 — WUI OOM pin fix; was 128/129 in CI; needs green → enqueue → merge → rebuild + deploy to
buildtest.kotlin.build:443 (droplet dump-storm continues until then).
e2eCancelledBuildDestroysDropletPromptly — Opus lane stopped by user; its fix is preserved UNREVIEWED and UNVERIFIED on branch fix-cancelled-build-flake-psi-warmup (head f410fcdc, "Warm buildscript PSI parser at startup") — pick up: verify the reproducer statistics both directions, review, PR.
D. Not yet triaged (parallel sessions' handoffs — this campaign never assessed them)
2026-07-07-backlog-triage-cascade-decision-and-merge-rollout.md, 2026-07-07-containernursery-challenge-fixes-noop-put-singleton-lock-cli-flags.md, 2026-07-07-gossip-storm-cn-thread-starvation-incident.md, 2026-07-07-gossip-storm-fleet-redeploy-blocked.md, 2026-07-07-hardware-control-fabric-netlab-e2e-blocked.md, 2026-07-07-kompile-remote-build-optimizer-impl-and-deploy.md, 2026-07-07-performancetest-workstream-graduation-and-infra-forensics.md, 2026-07-07-urlresolver-race-leak-bundle-ci-blocked.md, 2026-07-07-urlresolver-remote-command-observable-706-blocked.md, 2026-07-07-w3wallet-workstream-complete-urlresolver-708-buildtest-cutoff.md, 2026-07-08-urlresolver-workstream-merge-blocked-buildtest-server-oom.md, plus all pre-2026-07-05 entries. Treat every claim in these as unverified. Note: …706-blocked.md intersects the observable-core DerivedObservable contract memory (addChangeListener = deltas only; do NOT mis-fix as an upstream bug).
E. Known-remaining flake backlog (from the campaign, all needs-verification)
See the campaign handoff's backlog list: buildtest 0.0.230 redeploy; kotlin-build-ci jar-shrink; testNetlabP2PIntegration; testOpenPersistentRpcConnectionWaitsForGossipWhenNoCandidatesFound; e2eShardedBuildAcrossRealDroplets (re-verify post-0.0.337); testLanDiscoveryLoopbackRpcStream; CN entropy starvation; CN tmpdir sweep #479; kotlin-build-ci EventLog interrupt + child OOM; GitHub-429 CI bootstrap; fleet pin/deploy sweep (resolver 0.0.602+, protocol 0.0.337).
Next steps (ordered)
- Land the three C-items (727, 114+deploy, PSI-warmup branch review) — they are half-done and decay fastest.
- Sweep classification A/B: for each, run the named re-checks; where confirmed done, add a one-line supersession note to the handoff (single batched PlanRepository PR is fine).
- Triage D one at a time (newest first): verify its claims, then either execute its next steps (lane pattern above) or reclassify.
- Work E as flake lanes (Priority 1 beats Priority 2 whenever a "transient" shows up — reclassify upward per PRIORITIES.md).
- Keep the ledger honest: every resolution gets its handoff annotated + issues closed with evidence.
Operational state at write time
- Subagents: ALL STOPPED by user instruction (2026-07-09 ~04:45 UTC). Codex lanes dead (quota until ~Jul 13 21:52; resumes blocked too). Opus lanes O-A/O-B stopped — O-A's work preserved on the branch above; O-B's PR #114 has everything pushed.
- Nothing unsaved: all lane clones verified clean; O-A's dangling commit pushed at handoff time.
- Session-local automation (status cron, CI waiters) dies with the authoring session — re-arm your own watches.