Handoff: Flake-elimination campaign ("fix ALL of them, max concurrency 3") — 11 PRs resolved, 5 production deploys, 3 lanes in flight, codex quota exhausted until Jul 13

Written: 2026-07-09 ~04:30 UTC. Campaign ran 2026-07-07 → 2026-07-09 (follows on from 2026-07-06-flake-elimination-buildtest-oom-relay-bytearray-and-700-netlab-block.md).

✅ RESOLUTION UPDATE (2026-07-09 ~15:15 UTC — the resolver session)

All three in-flight items and the top backlog item are resolved; verified evidence below. Re-verify anything you act on.

1. UrlResolver #727 MERGED 2026-07-09T09:04:36Z (merge commit 88ad90b6); resolver 0.0.602 published to kotlin.directory and verified (pom/jar 200). Queue history: 5 enqueues / 3 evictions, ALL provably DO droplet-quota infra (hourly API budget 5000/hr exhausted by concurrent queue entries: 10/10 shards 429 POST /v2/droplets, 0 tests executed) — the PR's own tests passed 1048/1048 every time droplets existed. Working tactic: dequeue, wait for the hourly budget reset (:28:52 past the hour at the time), re-enqueue solo.
2. BuildTestWui #114 MERGED 05:31:20Z + deployed 05:53 — but the dump storm did NOT stop: a second, independent root cause OOMed the WUI from 06:24 (fresh-dump MAT evidence: per-render getExecutionDurationStats() marshaled through TypeMarshaling.hostListToJvmList at ~33MB/in-flight call; 3-4 concurrent renders kill the 128m heap; the SSE error handler's 10s reload sustains the loop). Fixed at the WUI layer by BuildTestWui #115 MERGED 14:57:43Z (bounded pagination + windowed duration-stats + single-flight TTL DTO cache; 7 reproducers fail→pass; reproducers authored by a parallel session, fix coordinated via shared memory) + deployed 15:03. Concurrent run-page renders verified fixed (4x parallel 200 in ~3.2s). RESIDUAL (analyzed): GET /api/duration-stats still OOMs — post-fix dump (/root/wui-postfix-oom.hprof, MAT report in /code/workspace/wui-oom-lane/reports/postfix/) proves ONE uncontended full-table materialization retains 102MB/79.5% of the 128m heap (JvmValue[16384] ⇒ the stats table is 8-16k+ entries; a single copy no longer fits at any concurrency, so no WUI-side bounding can fix it). Only real fix: windowed/paged abstract method on a BuildTestApi extension interface (SJVM proxies only dispatch abstract methods) — a parallel session owns that upstream lane on bounded-collections-api branches and has been asked to prioritize it; the WUI then pages into its DTO cache. Interim exposure is bounded: only browser run-page views trigger it now (per-load JS fetch); watchman/API polling paths are fixed.
3. O-A / e2eCancelledBuildDestroysDropletPromptly PSI-warmup branch (f410fcdc): DO NOT SHIP on current evidence. Verification lane got 0/10 reproduction on main under the commit's own amplification recipe (worst submit→BUILDING 12.6s vs the 15s gate); the "cold JVM PSI parse" premise doesn't hold under scripts/test.bash (kompile-cli runs tests in-process, having already PSI-parsed hundreds of scripts), and measured timings show UPLOADING, not preparation, dominating. Review also found a real race: SHARED_PARSER_WARMUP_THREAD is assigned after the CAS winner starts the thread, so a concurrent constructor's awaitStartupRecovery() can miss the join. Branch preserved untouched. Next-step reproducer recipe: run via BuildTestRunner on a 2-vCPU droplet-class environment (where the 384/385 flake actually occurred), and fix the race before any PR.
4. buildtest-server-service 0.0.230 redeploy: already done by a parallel session (verified: deployed jar mtime Jul 8 21:43, 20 min after the pin-bump PR merged, and the jar contains rateLimitRetryMaxWaitMs from BuildTestEmbedded #265) — quota backpressure/lease reaping/429 retry are live in production.
5. New upstream bug discovered (cost #115 a queue round): uploadFileChunk chunked upload is not idempotent across LazyReconnectingMethodDispatcher retries — queue-run goldenScreenshots failed with Expected chunk index 1 ... but received 0 after a transient RPC failure; the same SHA passed 90 min earlier; likely explains prior goldenScreenshots failures (runs 61433d7f, c956fc49). Fix belongs in the UrlResolver retry layer / ScreenshotTest chunk-state handler.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below is a write-time snapshot (verified 2026-07-09 ~04:27 UTC) and will be stale. Authoritative re-checks:

• PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• Queue membership: gh api graphql -f query='query{repository(owner:"CodexCoder21Organization",name:"<repo>"){pullRequest(number:<n>){isInMergeQueue}}}'
• Services: curl -m 25 -s -o /dev/null -w "%{http_code} %{time_total}s" https://buildtest.kotlin.build/api/runs?limit=1 (same for kotlin.directory/api/list-artifacts, kotlin.build/health)
• Host: ssh -p 23 -i ~/.ssh/id_ed25519_20260706_035316 root@198.199.106.165 (dumps at /root/ContainerNursery/heapdumps/, runs at /root/buildtest-data/runs/)
• Published artifacts: probe https://kotlin.directory/<group-path>/<v>/<artifact>-<v>.pom (200 = taken; 401 = NOT FOUND, not auth — verified both ways)

---

Mission (verbatim intent)

User: "Your job is to fix all of them, with a maximum concurrency of 3" — "them" = all known flakes per PRIORITIES.md Priority 1, discovered via handoff mining + live evidence. Standing rules given by the user: verify every recorded diagnosis before fixing (handoffs/issues may be obsolete or wrong); doctrine precedence — claims conflicting with TESTING.md are presumptively wrong ("CPU overloaded" is never a root cause); deploy/merge/publish pre-authorized ("You are approved to deploy, merge, and publish as needed to make everything go green. Ensure it all gets merged."). Implementation by codex gpt-5.5 lanes (canonical invocation, briefs via "$(cat brief.md)"); after codex quota exhaustion (2026-07-08 ~22:15 UTC, resets Jul 13 ~21:52 per its error message) the user authorized Opus subagents for the backlog.

What was found and done (the chain)

Resolved campaign PRs (all verified merged/closed at write time)

| PR | What it fixed | Evidence chain | |---|---|---| | BuildTestEmbedded #254 | DO quota-422 backpressure + lease-based abandoned-run reaping + delete-time droplet cleanup (issue #253) | 4 hermetic reproducers; 403/403 | | BuildTestEmbedded #265 | DO 429 rate-limit bounded retry/backoff (+Retry-After); 429 previously escaped createBuildDroplet and was terminal at shard level (attempts=1/3 signature) | FakeDropletService repro 2/2 fail→pass; composes with #254 (9/9) | | ContainerNursery #507 | The big one: bounded child-log ingestion (131072-byte/line cap, head+tail + marker). CN supervisor (-Xmx128m) was OOMing ~hourly ingesting 5.59MB uploadChunk log lines (multiple copies per line via StringBuilder→String→JSONObject→JSONL) — this caused the platform-wide 503 windows | hprof histogram (87MB byte[] of log lines); truncation repro fail→pass; 311/311 + queue pass | | WebCronApiServer #48 + #49 | webcron OOM: execution-history budget ignored CronExecution.error text (10k × unbounded error strings) + hand-built multi-MB Base64 __bytecode_request JSON → streaming bind; resolver 0.0.600/protocol 0.0.337 pins | budget repro 3/3 fail→pass; 83/83 | | UrlResolver #724 | Issue #717 (closed): PersistentRpcConnection.reconnect() held reconnectLock across the reconnect factory's sleep/dial loop; deletion-worker + whole url:// API blocked behind it (production outage 2026-07-07 ~21:40, thread-dump proven). Factory now runs outside the lock; racing callers bounded-join/fail-fast; close() interrupts the owner | repro 2/2 fail on main → 3/3 pass; 13/13 family; 1034/1034 remote | | kompile-core #209 | Transient artifact-fetch failures fail builds (org-wide flake class): all fetch sites routed through fetchBoundedBySocketTimeouts, bounded 3-attempt retry (transient-only), total time budget per resolution operation, child JVMs inherit budget. 3 rounds: round-2 fixed retry-latency amplification; round-3 proved the queue timeouts were expensive nested-Kotlin test fixtures (converted to synthetic Maven artifacts / tiny Java jars) — NOT retry-induced cache corruption (refuted with evidence) | 8/8 + 9/9 retry family; queue merged after fixture slimming | | build-kotlin-jvm #61 | Upstream layer of the fetch-budget work (CoursierFetchGuard); published kompile:build-kotlin-jvm:0.0.30 | merged via queue | | UrlProtocol #349 | Relay registration ACK-before-visibility race (root cause of testCrossRelayRoutingImmediateDistanceVectorBroadcast 30s timeout): handleRegister() ACKed before registeredPeers/routing commit → ACK-gated sender, registration visible before ACK. Same defect class as PR 345's PeerExchange fix | standalone harness 100/100 fail → 0/100; deterministic in-repo repro; 952/952 local; merged after my manual rebase (see below) | | BuildTestEmbedded #263 | CLOSED superseded — parallel sessions shipped 0.0.230 (via #267) containing #254+#265 | — |

Also: yesterday's 8 PRs (see prior handoff) all remain merged.

Production deploys performed (all verified at deploy time)

1. buildtest-server-service 0.0.135 then 0.0.136 → url:buildtest: (14:04 and 18:20 UTC 07-08). 0.0.136 contains the execution-duration projection OOM fix (#246). The ~35-60-min OOM crash-loop is verified dead (115+ min continuous uptime probe). NOTE: the service does NOT yet run 0.0.230 (with #254/#265 quota/lease/429 fixes) — see next steps.
2. ContainerNursery rebuilt from main with #507 → deployed 14:35 UTC 07-08 via ssh-restart-containernursery CLI. Supervisor OOM verified dead (zero supervisor dumps since; another session redeployed an even newer CN at ~15:53 — also fine).
3. webcron-url-server with #48+#49 → url:webcron: 17:48 UTC 07-08 (SHA c2dd007a).
4. netlab-manager with NetLabManagerServer #93+#94 → url:netlab-hosted: 14:09 UTC 07-08 (SHA e7c5e321).
5. kotlin-build-ci 0.0.59 (contains the #162 self-healing reconciler, merged 07-06 but never deployed) → scp fallback (jar is 234MB > CN's 200MB ADMIN_UPLOAD_REQUEST_LIMIT_BYTES): scp -P 23 -i ~/.ssh/id_ed25519_20260706_035316 <jar> root@198.199.106.165:/root/ContainerNursery-uploads/jars/kotlin-build-ci.jar, then cold-start via curl https://kotlin.build/health. Verified: health shows build: kotlinbuild.ci:kotlin-build-ci:0.0.59. This closed the "buildtest run green but GitHub check never finalizes" gap that cost multiple lanes hours.

Published artifacts

buildtest-embedded 0.0.217 (+ parallel sessions took 218-230); buildtest-server-service 0.0.136; foundation.url:protocol 0.0.337; foundation.url:resolver 0.0.601 (from #727's PRE-rebase branch — see in-flight); kompile:build-kotlin-jvm 0.0.30.

In flight at write time (RE-VERIFY FIRST)

1. UrlResolver #727 — SJVM ObjRef leak fix (SandboxedConnectionHandle.closeSandboxedConnection() local lifecycle handle for close-owning proxies; from heap dump java_pid218301.hprof: 307,922 ObjRefImpl retained in a CN child service). Codex lane 3E died (exit 144) after enqueueing; the entry was evicted (train collateral) and the branch conflicted with main. I rebased it manually: coordinate conflict resolved to 0.0.602 (0.0.601 is published from the pre-rebase branch; both changelog comments kept), perf-test pins → published 0.0.601, testCloseOwningSandboxedProxyHasLocalLifecycleHandle PASSED clean-cache on the rebased head a2c351d2, pushed. A background waiter (this session only) was armed to auto-enqueue on green — re-verify checks and enqueue manually if the session is gone: warm https://github-webhooks.wasmserver.com/ 3×, then GraphQL enqueuePullRequest.
2. BuildTestWui #114 — buildtest-wui OOM = stale protocol pin (resolver 0.0.600's POM needs protocol 0.0.337); pin aligned + fat-jar leak-fix regression guard. Opus lane O-B had run 562a62e4 at 128/129 (only goldenScreenshots left, ~31 min test) with build-watchman --to-merged armed (auto-enqueue authorized). After merge: rebuild the fat jar and deploy to CN route buildtest.kotlin.build:443 (CN CLI upload-jar), then verify 200. Urgency: the screenshottest droplet's OOM storm refills /root/…/heapdumps at ~2GB/30min until deployed (a sweeper exists but disk-full killed a goldenScreenshots run once already).
3. Opus lane O-A — e2eCancelledBuildDestroysDropletPromptly (BuildTestEmbedded) 15s BUILDING-phase timing flake (evicted queue run 9001dfdb, 384/385, 2026-07-08). Was mid-reproduction; no PR yet. If the lane is gone, restart per the campaign brief pattern (verify-first; never widen the 15s; fix the phase-transition mechanism).

Remaining backlog (verify each before working — some may be stale)

• kotlin-build-ci jar-shrink PR — the clean fix for the 200MB cap (86MB embedded build-kotlin-jvm-builtin-rules.jar dominates; org convention says deps should not be embedded as fat jars). scp deploy already done, so this is hygiene not urgency.
• buildtest-server-service redeploy with 0.0.230 — the deployed 0.0.136 predates #254/#265 (quota backpressure/lease reaping/429 retry are merged+published but NOT live). Bump BuildTestServerService pin → release → deploy, or verify a parallel session already did (check deployed jar mtime + a run's behavior on quota).
• testNetlabP2PIntegration READY-acquisition re-verify (post netlab-manager redeploy + quota fixes).
• testOpenPersistentRpcConnectionWaitsForGossipWhenNoCandidatesFound (UrlResolver, 30s timeout, seen 07-08 run 2d1090dc).
• e2eShardedBuildAcrossRealDroplets PEX-handshake flake — re-verify after fleet adopts protocol 0.0.337 (may be fixed by #345).
• testLanDiscoveryLoopbackRpcStream (UrlProtocol) — single-flake cost #349 a full run 07-09.
• CN URL-facade entropy starvation (2026-06-30 handoff; needs blocking-/dev/random kernel to reproduce).
• CN tmpdir sweep #479 (stale PR — re-verify relevance).
• kotlin-build-ci testE2E_watchdogDetectsTimedOutBuilds EventLog interrupt flake (upstream community.kotlin.eventlog).
• kotlin-build-ci child OOM (dump java_pid2521161, 05:10 07-08) — not yet root-caused.
• GitHub-429 Coursier bootstrap failures in merge-queue jobs (challenge) — fix = authenticated/cached coursier download in CI bootstrap.
• Fleet deploys: resolver 0.0.602+ (after #727) and protocol 0.0.337 to remaining consumers (buildtest-server, netlab-manager, CN, kotlin-build-ci pins).

Reusable / operational knowledge (hard-won this campaign)

• codex quota: exhausted 07-08 ~22:15 UTC; error says retry Jul 13th ~21:52. codex exec resume is ALSO blocked (verified). In-flight sessions keep working; only new/resumed API turns fail. Long prompts MUST go via "$(cat brief.md)" — one inline-quoted launch wedged forever at "Reading additional input from stdin..." (caught after 3.5h; check wc -c of a lane's stderr early).
• Version-coordinate collisions: parallel sessions publish from branches continuously. Probe https://kotlin.directory/<path>/<v>/<artifact>-<v>.pom (200=taken, 401=free) or api/list-artifacts before publishing; bump main via release PR when taken (pattern: #251, #347).
• Merge-queue batch poisoning: an entry ahead can fail your queue run (extra tests in the batch = another PR riding along). Diagnose by test-count deltas; a SOLO re-enqueue is the decisive experiment. Evictions with NO failed run = lost queue-ref check-suite dispatch → on enqueue, poll the queue ref's check-suites for ~3 min and rerequest on runs:0 (worked first try for #254 attempt 3).
• DO droplet quota saturation ≠ zombies: verify each run (check concluded + event stream frozen >10 min) before deleting — 07-08's 97/100 pool was legitimately-slow concurrent runs. See memory buildtest-droplet-quota-zombie-runs.
• watchman phase-budget false alarms: ground-truth /root/buildtest-data/runs/<id>/build.log + test-events.jsonl mtimes before force-failing; compile phases freeze output 20+ min normally; "Resuming sharded build after service restart" + freeze WAS the wedge signature before 0.0.135, but post-0.0.136 runs genuinely survive restarts.
• Host memory pressure: co-tenant sessions can eat 15/16GB and OOM-kill local test JVMs (exit 137/Killed). Gate heavy runs on free -m available >3000; a Killed JVM is not a test result.
• Full-suite mutex: /tmp/claude-test-slot.lock (mkdir-loop acquire, rmdir release even on failure) serialized all lanes' full-suite runs machine-wide — no collisions all campaign.
• Challenge reports merged this campaign: kompile-cli ≤0.0.77 silent stalls, buildtest remote run disappeared, buildtest green but check stuck, merge-queue bootstrap 429, plus kompile-preflight and issue-717-verification-blocked records.

Next steps (ordered)

1. Re-verify + land #727 (rebased head a2c351d2): checks green → enqueue → after merge publish resolver 0.0.602 from main.
2. Re-verify + land #114 → rebuild + deploy buildtest-wui → confirm the screenshottest droplet dump storm stops.
3. Finish/relaunch the O-A timing-flake lane (e2eCancelledBuildDestroysDropletPromptly).
4. Redeploy buildtest-server-service with 0.0.230+ (quota/lease/429 fixes live).
5. Work the backlog above, 3 lanes at a time (Opus until codex returns Jul 13).

Nothing unsaved: all lane clones under /code/workspace/{wave*,opus*} verified clean; all branches pushed. Local artifacts (jars, logs) are disposable.