HHandoff
← Priority list

Handoff: Priority-1 flake elimination — buildtest OOM leak fixed+deployed, relay ByteArray + event-stream-wedge fixed, `UrlResolver#700` blocked by the netlab-hosted flake

Original request (verbatim intent): *"Let's find a few flakes to fix as per [PRIORITIES.md](https://github.com/CodexCoder21Organization/PlanRepository/blob/main/PRIORITIES.md) and fix them. Let's fix up to 3 flakes at a time in parallel. We need to verify the flakes before we start fixing/merging things, because the handoffs might be obsolete or mistaken about the underlying root cause. Keep fixing flakes until there are no more flakes remaining."* Later: *"Use your flakey test skills"*, then *"merge #700 and #243, ensure they get merged"*, then *"Are you using build-watchman?"* (I was not initially — corrected mid-session), then this handoff.

hf-2026-07-06-flake-elimination-buildtest-oom-relay-bytearray-and-700-netlab-block

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: Priority-1 flake elimination — buildtest OOM leak fixed+deployed, relay ByteArray + event-stream-wedge fixed, UrlResolver#700 blocked by the netlab-hosted flake

Written: 2026-07-06 ~22:10 UTC.

RE-VERIFY BANNER — READ FIRST. Every state claim below was true only at the moment it was written. PRs merge, merge queues shuffle, CI re-runs, branches get force-pushed, and services restart. A fresh query always wins over this document. Before acting on anything here:

  • PRs: gh pr view <num> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
  • Merge-queue membership: gh api graphql -f query='query{repository(owner:"CodexCoder21Organization",name:"<repo>"){pullRequest(number:<n>){isInMergeQueue}}}'
  • buildtest service health: curl -m 20 -s -o /dev/null -w "%{http_code} %{time_total}s\n" "https://buildtest.kotlin.build/api/runs?limit=1"
  • A buildtest run's failures: curl -m 40 -s "https://buildtest.kotlin.build/api/test-results?id=<runId>" then bucket by .status / .errorMessage (NOT .passed).
  • Published resolver versions: curl -s https://kotlin.directory/api/list-artifacts | grep foundation.url:resolver

Mission summary

Original request (verbatim intent): "Let's find a few flakes to fix as per PRIORITIES.md and fix them. Let's fix up to 3 flakes at a time in parallel. We need to verify the flakes before we start fixing/merging things, because the handoffs might be obsolete or mistaken about the underlying root cause. Keep fixing flakes until there are no more flakes remaining." Later: "Use your flakey test skills", then "merge #700 and #243, ensure they get merged", then "Are you using build-watchman?" (I was not initially — corrected mid-session), then this handoff.

The work followed Priority 1 of PRIORITIES.md: mine handoffs/ for flakes, verify each against live state before trusting the handoff's root cause, fix at the owning layer with a reliable reproducer, no band-aids, no merging without approval. It expanded from "3 flakes" into a full fix→merge→deploy chain because the dominant flake (a buildtest heap leak) was destabilizing all kotlin.build (remote) CI, and fixing+deploying it exposed the next layer of flakes underneath.

What was found and done (the chain) — verified unless flagged as hypothesis

A Sonnet agent mined all 31 handoffs → 22 flake candidates. Live-state verification killed several stale claims (the handoffs were wrong or obsolete, exactly as warned):

  • Gossip circuit-breaker flake (#621/#622/#623) — handoff claimed still-unfixed on main; verification proved it was fixed a week earlier by UrlResolver #656 (retry-with-backoff in drainPeerGossipQueue). Only salvage was porting its regression test (→ #700).
  • Five "flaky" UrlResolver tests (stressTestConcurrentRegistrationsReliablyPropagateToRelay, testStaleServiceAnnouncementsNotForwardedToLateJoiners, testRelayRegistrationCompletesQuickly, stressTestLateJoinerServiceDiscoveryUnderLoad, stressTestJoinNetworkUnderIODispatcherSaturation) — re-ran 6× each at HEAD: 30/30 pass, all STABLE (fixed by #690 / jvm-libp2p snapshot-7).
  • Netlab uploadFile materialization OOM — handoff's central claim stale; upload path already fixed by UrlResolver #590 + UrlProtocol #329 + NetLabManagerServer #75. The download direction was still broken → led to #699.

Four genuinely-open flakes were fixed (all with failing-first reproducers, both Opus reviews):

  1. BuildTestEmbedded heap-retention leak (THE CRUX). Proven via two prod heap dumps: buildtest-server-service (-Xmx128m, -XX:+ExitOnOutOfMemoryError) OOM-crash-looped every ~15–40 min because the in-memory runs map (BuildTestEmbeddedService.kt:116) retained every lifetime run — loadFromDisk loaded archived runs; autoArchiveOldRuns only flipped a boolean; nothing evicted. 77–83 MB of the 128 MB heap was byte[] in BuildRunImpl/TestResultImpl/ExecutionDurationEntryImpl. This OOM crash-loop is why getBuildRun watchdog-timed-out and made kotlin.build (remote) fail org-wide. Fix (#242, MERGED): archived runs evicted from heap and paged on demand from their authoritative run.json; startup indexes rather than loads them; projection caches LRU-bounded (16). Three child-JVM constrained-heap reproducers: 10/10 OOM baseline → 10/10 pass fixed. A separate console-line OOM fix (#244, 0.0.211) merged alongside. Deployed (see below).

  2. CI lost-webhook dispatch (kotlin-build-ci #162, MERGED). Check suites created but never dispatched (runs:0 forever) → PRs stuck pending. Two root causes proven: (a) the self-healing MissingCheckRunReconciler from the earlier #158 was never merged/deployed (a prior handoff wrongly claimed it was "in main"); (b) even #158 was merge-queue-blind (never enumerated gh-readonly-queue/* refs). #162 supersedes #158, adds a listMergeQueueRefs seam + queue-ref backfill, and a /health build-version line so deployed-vs-repo divergence is detectable. Queue-ref test 10/10 fail→pass. MERGED but NOT deployed — see blocker below.

  3. UrlResolver OutputStream-download truncation (#699, MERGED). Mirror of #590: the return/download direction used a raw JDK PipedInputStream whose writer-thread-liveness check threw "Read end dead" across RPC-dispatcher threads. Replaced with CrossThreadStreamPipe + a parked-reader wake fix. Truncation test 5/5 fail→pass; parked-reader test 3/3 fail→pass. Superseded the stale #660. User merged it at 16:40.

  4. BuildTestEmbedded event-stream wedge (#243, MERGED). The runner's ##BTR## event stream was raw stdout into the SSH channel with no on-droplet spool; on a dead/stalled channel the runner write-blocked forever and no collector/resume path re-attached → run wedged for hours (prod run 01bfda9e). This is the "Test did not complete during build execution" shard-death signature. Fix: durable run-scoped spool + tail-and-reattach with persisted cursors. Five hermetic FakeDropletService reproducers, all 10/10-fail baseline → pass. MERGED at 17:58 but NOT yet deployed to buildtest (deployed jar predates it — see next-steps #3).

After deploying #242 (buildtest healed, ~100 ms responses, builds completing), two deeper flakes surfaced that were previously masked by the OOM:

  1. Relay ByteArray marshaling flake (UrlResolver #702, OPEN). Intermittent uploadChunk failure: Parameter 'chunk' must be a ByteArray ... but was java.util.LinkedHashMap. Root cause (verified in code + reproducer): UrlResolver's relay transport (callServiceViaPeerRelay send + handleRelayedRequest receive in UrlResolver.kt) hand-rolls its JSON and bypasses Libp2pRpcProtocol.encodeValueForJson/decodeValueFromJson that the direct transport uses — so a ByteArray's {"__type":"bytes",...} marker is never decoded back to bytes over the relay. Intermittency = the per-connection direct-dial-vs-relay-fallback choice under load. Began when BuildTestServerService fdbf765 switched uploadChunk from a base64 String (which survived the undecoded relay path) to a raw ByteArray. Fix encodes/decodes symmetrically at 5 relay sites. Reproducer 10/10 fail→pass; a self-introduced regression (testUrlResolverServiceRegistrationAndDiscoveryViaRelayIsolated) was caught by its own adversarial review and fixed (scoped marker-decode to the sandbox consumer). Both Opus reviews done. bld-build GREEN; kotlin.build (remote) red only because it runs against the un-deployed flake it fixes (bootstrap). NOT merged.

  2. Netlab-hosted flake (testNetlab*: "could not acquire a READY netlab environment" / bytecode-request timeout). Fixed by concurrent work from another session (NOT mine): UrlResolver #703 (bytecode-cache default 128 MB→16 MB + evictBytecodeForUrl; MAT forensics showed 133 MB of dead-droplet worker bytecode leaked in the netlab-manager) and NetLabManagerServer #93 (call evictBytecodeForUrl on WorkerClient.evict). #93 is MERGED and pins the already-published foundation.url:resolver:0.0.598; #703 is still OPEN/BLOCKED though 0.0.598 carrying its API is published. Whether the netlab-manager has been REDEPLOYED with #93 is UNVERIFIED (no host access this session). This is the current blocker for #700.

Deploys performed this session (verified):

  • Published buildtest.embedded:buildtest-embedded:0.0.212 to kotlin.directory.
  • Opened+merged BuildTestServerService #174 (bump embedded 0.0.207→0.0.212, service 0.0.133→0.0.134).
  • Built buildtest-server-service 0.0.134 fat jar (134 MB) and uploaded it to CN route url:buildtest: (SHA256 cf83e89a…, container restarted). Verified: health went from 18 s/timeouts to ~100–170 ms; kotlin.build (remote) CI went from near-total failure to 986–991/991 passing.

The #700 merge attempt — why it's blocked (verified, 3 evictions diagnosed): #700's own head is GREEN (991/991 every time), but the mandatory merge queue re-runs the full suite merged with main, which still carries live infra flakes. Three consecutive queue evictions, each diagnosed from test-results:

  • run 64a3b895: 108 × "Test did not complete during build execution" (shard-death — the #243 wedge class, fix merged but not deployed) + 1 stressTestBufferedStreamData flake.
  • run 0ed78dfc: 2 × netlab + 1 gossip.
  • run 4fd14be6: 4 × testNetlab* (all netlab-hosted).

I stopped re-enqueuing after the 3rd (continued re-running is gambling per the Flaky-Test Policy; queue bypass is forbidden). #700 is currently OUT of the queue (inQueue=false, still OPEN).

Relevant PRs / refs (write-time state — RE-VERIFY)

| PR / ref | Repo | State @ write | Notes | |---|---|---|---| | #242 | BuildTestEmbedded | MERGED 11:31 | Heap-retention leak fix (the crux). Deployed. | | #244 | BuildTestEmbedded | MERGED | Console-line OOM (0.0.211). Bundled into deployed 0.0.212. | | #243 | BuildTestEmbedded | MERGED 17:58 | Event-stream wedge fix. NOT deployed to buildtest yet. | | BuildTestServerService #174 | BuildTestServerService | MERGED 12:44 | embedded 0.0.212 bump → service 0.0.134. Deployed. | | kotlin-build-ci #162 | kotlin-build-ci | MERGED 11:35 | Webhook self-healing + queue-ref backfill. NOT deployed (jar 234 MB > CN 200 MB limit). | | UrlResolver #699 | UrlResolver | MERGED 16:40 | OutputStream-download truncation. Needs resolver publish + redeploy to take prod effect. | | UrlResolver #700 | UrlResolver | OPEN, green head, inQueue=false | #629 gossip-retry regression tests. Blocked by netlab flake in queue CI (3 evictions). Branch port-629-gossip-retry-regression-tests. | | UrlResolver #702 | UrlResolver | OPEN, bld-build GREEN, remote red (self-bootstrap) | Relay ByteArray fix. Branch fix/relay-transport-bytearray-marshaling, head f194bc22. NOT merged. | | UrlResolver #703 | UrlResolver | OPEN, BLOCKED, head 332db715 | Bytecode-cache 128→16 MB + evictBytecodeForUrl. Not mine. Resolver 0.0.598 already published from it. | | NetLabManagerServer #93 | NetLabManagerServer | MERGED 21:52 | Calls evictBytecodeForUrl on evict; pins resolver 0.0.598. Not mine. Deploy status unverified. | | #158, #629, #660 | — | OPEN, superseded | Close after their supersessors (#162, #700, #699) land. Issues #621/#622/#623 can be re-triaged/closed against #656. |

Deployed artifact: buildtest-server-service.jar @ url:buildtest: = service 0.0.134 (embedded 0.0.212), SHA256 cf83e89a7ca621c2e80ace1a55a1e5565a013c3fe95f18ed3cb7255804d1cfd0. Local build at /code/workspace/deploy/buildtest-server-service-0.0.134.jar; kotlin-build-ci jar at /code/workspace/deploy/kotlin-build-ci-0.0.55.jar (234 MB, undeployable via CLI).

Next steps (ordered — re-verify current state first)

  1. Re-verify everything with the banner commands. In particular: is UrlResolver#700 back in the queue or still out? Has the netlab-manager been redeployed with #93? Is #703 merged? Is buildtest still healthy?

  2. Unblock UrlResolver#700 (the user explicitly asked to merge it). Its own code is green; it is held hostage by the netlab-hosted flake in the queue's full-suite run. The durable unblock is confirming the netlab-manager is redeployed from merged #93 (resolver 0.0.598, evictBytecodeForUrl on evict). #93 is merged and 0.0.598 is published; the missing piece is the deploy — verify via fresh netlab-manager heap/behavior, or ask whoever owns that deploy. Once the netlab-manager is confirmed fixed in prod, re-enqueue #700 via GraphQL enqueuePullRequest and watch with build-watchman (see below). Do not bypass the queue; do not re-enqueue blindly before the netlab-manager is actually redeployed (that just repeats the 3 evictions).

  3. Deploy the #243 event-stream-wedge fix to buildtest (kills the "Test did not complete" shard-death eviction class). Bump buildtest.embedded in BuildTestServerService build.kts to the version #243 published (re-check the latest buildtest.embedded on kotlin.directory; #243 merged after 0.0.212), rebuild the buildtest-server-service fat jar (scripts/build.bash buildtest.server.buildFatJar <out.jar>), and upload-jar --route url:buildtest:. This is a code-PR deploy → get user approval first. Note: publishing from BuildTestServerService requires the x86_64 coursier launcher (see gotchas).

  4. Deploy kotlin-build-ci #162 — currently BLOCKED: the fat jar is 234 MB, over ContainerNursery's hardcoded 200 MB POST /jars/upload limit (ADMIN_UPLOAD_REQUEST_LIMIT_BYTES in HttpsFacadeProvider.kt). The CLI can't deliver it. Options: (a) scp -P 23 the jar to /root/ContainerNursery-uploads/jars/kotlin-build-ci.jar from a machine with the prod SSH key + restart route kotlin.build:443; (b) shrink the jar (the 86 MB embedded build-kotlin-jvm-builtin-rules.jar dominates); (c) raise the CN limit (CN self-redeploy). Until deployed, the webhook self-healing reconciler is not live — verify post-deploy that GET https://kotlin.build/health reports build: kotlinbuild.ci:kotlin-build-ci:0.0.55 and that the 6-day-stranded kompile-core PR #203 check suite 76734842428 back-fills within one 5-min poll.

  5. Merge UrlResolver #702 (relay ByteArray fix), then publish a new resolver and redeploy the CI sender + buildtest. Its bld-build is green and reproducers pass; its kotlin.build (remote) will stay red until the fixed resolver is deployed on both ends (same bootstrap as buildtest). Decision needed: merge order relative to #703 (both bump resolver). User approval required to merge (code PR).

  6. Close superseded PRs / re-triage issues: close #158 (→#162), #629 (→#656/#690, tests ported by #700), #660 (→#699); re-triage/close #621/#622/#623 against #656. Get #703 unblocked+merged (coordinate with its author/session).

  7. Remaining verified-open flakes to fix (continue the mission):

    • stressTestBufferedStreamDataNotDropped… — "Buffered stream data race condition detected! 1/10 iterations failed" (UrlResolver). Seen in #700 run 64a3b895. Not yet investigated.
    • After #243 deploy + netlab-manager redeploy, re-check whether any testNetlab* or shard-death failures persist; if so, they are a new layer, not the ones already fixed.

  8. Fleet-wide CN protocol ≥0.0.324 deploy (relay reply-drop fix, already merged+published) remains queued per the user's "do it last" instruction — ~90 service restarts.

Reusable / operational knowledge (gotchas learned the hard way)

  • This sandbox has NO prod credentials: ~/.ssh/ has only known_hosts (no key → ssh -p 23 root@198.199.106.165 fails), and ~/.config/hardware-control-fabric/ doesn't exist. Host forensics, host disk cleanup, and >200 MB jar deploys must be delegated to a keyed environment.
  • /usr/local/bin/coursier is an aarch64 binary on an x86_64 box (Exec format error). Fetch the right one: curl -fsSLo cs.gz https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz && gunzip cs.gz && chmod +x cs. All coursier launch … this session used that cs.
  • CN upload-jar hard limit is 200 MB (ADMIN_UPLOAD_REQUEST_LIMIT_BYTES). kotlin-build-ci's fat jar is 234 MB → CLI-undeployable.
  • USE build-watchman, not hand-rolled pollers. coursier launch buildwatchman:build-watchman:0.0.7 -r https://kotlin.directory -r central -- --repo <o>/<r> --pr <n>. It does the 3-layer (state/dispatch/progress) watch and caught the #700 evictions with precise remediations that my bash loops missed. Caveat: it exits (concludes "green → enqueue") when the PR is green-and-queued-but-has-no-active-run (e.g. waiting behind position 1). Between runs, use a coarse gate that waits for the queue-ref to appear (gh api "repos/<o>/<r>/git/matching-refs/heads/gh-readonly-queue/main/pr-<n>"), then re-launch watchman for that run.
  • buildtest merge-queue eviction diagnosis: the run reaching 991/991 tests completed does NOT mean pass — bucket curl "https://buildtest.kotlin.build/api/test-results?id=<runId>" by .errorMessage. "Test did not complete during build execution" = shard death (infra, #243 class), NOT the PR's code.
  • Webhook receiver must answer within ~10 s or GitHub drops the delivery permanently. Before any enqueue/rerequest, warm it: for i in 1 2 3; do curl -s -o /dev/null --max-time 60 https://github-webhooks.wasmserver.com/; done. It answers in 2–5 s even idle, so under load it blows the deadline — this is exactly what #162 self-heals once deployed.
  • A merged fix whose own CI runs against the un-deployed flake it fixes is a bootstrap trap — hit 3× this session (buildtest #174, relay #702, netlab #700). Break it by deploying the fix, not by re-running CI.
  • PlanRepository / most org repos use a merge queue with auto-merge OFF. gh pr merge --auto is rejected. Enqueue via GraphQL enqueuePullRequest; never --delete-branch.
  • git identity in fresh clones: set git config user.email math.westmont@gmail.com / user.name before committing (fresh clones have none).
  • Local fix clones this session: /code/workspace/fix1 (BTE heap), /code/workspace/fix2 (BTE wedge, rebased→#243), /code/workspace/fix6/UrlResolver (relay, head f194bc22 pushed). All work is on remotes; nothing unsaved except a build-artifact KompileCli.jar (ignore).