Handoff: Observables workstream close-out — 16 PRs merged, all four infra issues fixed+deployed, droplet-service leak fixed at root; #707/#680 merge tail and pacer re-land remain

Written: 2026-07-09 ~22:10 UTC. Continues (and largely closes) 2026-07-09-observables-workstream-713-730-cascade-and-four-infra-issues.md.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below is a write-time snapshot. PRs merge, the UrlResolver merge queue shuffles, CI re-runs, and prod services get redeployed by parallel sessions (coordination file: /root/cn-watchdog.audit on 198.199.106.165:23, key ~/.ssh/id_ed25519_new). Authoritative checks:

• PRs: gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt
• UrlResolver queue: gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:10){nodes{position pullRequest{number} state}}}}}'
• DO rate limit: on the host, curl https://api.digitalocean.com/v2/account with the token from the droplet-service process env; read ratelimit-remaining.

A fresh query always wins over this document.

Mission summary

Same standing directives as the parent handoff: (1) fix the droplet-service memory leak at the root with the 128MB cap restored; (2) merge UrlResolver #713; (3) working (not just filed) solutions for BTSS#180, BTSS#181, ContainerNursery#500, NetLabManagerServer#98; (4) investigate CI flakes deeply, never just retry; (5) codex gpt-5.5 for heavy lifting (its quota returned this session); (6) merge-when-genuinely-green pre-approved.

What was found and done (the chain)

1. All four infra issues are FIXED, MERGED, CLOSED-WITH-EVIDENCE, and DEPLOYED.
   • BTSS#180 via #184 (merged 05:29Z).
   • BTSS#181 via #182 (merged 09:12Z after 5 queue attempts, each eviction root-caused). Includes a review-caught severe fix: the reaper originally treated absent run.json as orphaned — on the shared DO account it could have deleted other instances' droplets; now only locally-recorded terminal runs are reaped.
   • NetLab#98 via #99 (merged 09:25Z); manager deployed to prod url:netlab-hosted: at 21:17Z (bounded-FIFO acquire queueing live).
   • CN#500 via #509 (merged 15:11Z; all 5 tests verified red-on-main with quoted failures). Deployed to prod CN by the parallel claude-main-session at 20:51Z (their PR 508 deploy carried merged main incl. #509).
2. UrlResolver #730 merged (11:59Z) — the onServiceAnnouncement resurrection race fix, plus a second root fix found during eviction forensics: active-query results leaked into peer-registry advertisements, letting relay cleanup synthesize withdrawals that evicted cached entries (testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup flake, 4–5/50 repro → 0/50 fixed).
3. UrlResolver #713 merged (17:02Z) — the sandbox ObjRefImpl leak fix (SandboxInvalidationRegistry, semantic-merged with #727's close-owning handle), the launchSurfaceErrors test-race fix (test-side job.join(), proven pre-existing), and two relay-bytecode root fixes (UNKNOWN_METHOD flake: 7/30 on main → 0/30 fixed; retry across candidates + forward-to-owning-peer). Resolver foundation.url:resolver:0.0.603 auto-published by merge CI.
4. Droplet-service leak fixed at the root and deployed: DODSS#81 (resolver 0.0.599→0.0.603) merged 17:44Z; 0.0.44 live (see item 6 for why 0.0.44 not 0.0.43). -Xmx128m verified in the live cmdline; multi-hour RSS plateau ~410–450MB total (heap+metaspace+native; the old leak grew 24–87MB/min) with zero OOMs. The morning keepalive kill-loop was retired at 06:22Z (it had been killing a healthy parkfixed service — see [issues/82 evidence and droplet-service-parkfix-128mb memory]).
5. Buildtest CI reliability chain — five defects fixed and deployed, discovered by refusing to blind-retry:
   • Lease knife-edge (renewal interval == 15-min staleness): BTE#272/#273 (threshold/3 + retry) merged 09:44Z; server deployed 12:17Z (BTSS#192); consumer cadence aligned in kotlin-build-ci#171 merged 18:45Z, kotlin-build-ci 0.0.60 deployed 18:5xZ (jar >200MB exceeds CN upload limit — scp to image path + CN restart).
   • Oversized BTR event lines failing all-pass runs (4,194,423 chars vs 4 MiB cap): BuildTestRunner#33 (chunking, byte-exact continuations) + BTE#274 merged ~11:00Z; buildtest-server-service 0.0.152 deployed 14:55Z carries both (note: kotlin.directory publication is MANUAL for BTE — community-kotlin-maven-artifact-publishing launcher; UrlResolver merge-CI auto-publishes; verify per repo).
   • Adoption listDroplets stampede (prod regression from the 0.0.151 deploy, self-caught in 30 min): BTSS#194 (adoption only on retry/failure paths + single-flight TTL cache shared with reaper) merged 14:50Z, deployed in 0.0.152.
   • #707's testRemoteCommandObservableCleanupAfterTerminal: deterministic semantic conflict — COMMAND_STATUS fell back to the idempotency outcome store after the retention sweep, resurrecting swept records as SUCCEEDED. Fixed on the #707 branch (status bounded by retention; idempotent COMMAND_INVOKE still consults the store).
   • DO 429 storms (hourly self-sustaining cycles; even GETs 429'd; whole-fleet CI outage windows): root cause = no account-level pacing, filed as DODSS#79, fixed by #80 (merged 17:45Z, deployed as 0.0.43)…
6. …but the deployed pacer caused a new regression (verified): createConcurrencyLimit=1 start-serialization pushed sharded provisioning past the 5-minute watchdog and convoyed the single-threaded SJVM sandbox channel (interrupted listDroplets/renewDroplet; runs 2d919922, a3ed1e19). Reverted via #83 (merged 20:32Z), 0.0.44 deployed 20:37Z (keeps the resolver leak fix). The corrected re-land (#84, 10 in-flight creates / 750ms smoothing / gate-front 429 retry, all CI green, full suite 87/87) is deliberately held unmerged for user review given the same-day regression. Follow-up filed: DODSS#82 (starved renewals let auto-termination delete droplets mid-run; renewals deserve budget priority).
7. UNRESOLVED FLAKE (hypothesis, not root-caused): windowed SSH-unreachable fresh droplets. In windows (observed 18:33–19:18 and 21:18–21:44Z), 10-shard runs fail with Failed to connect to root@<ip>:22 after 12 attempts … socket is not established on ALL shards, while single/3-shard runs and other windows pass — including after the pacer revert and with DO remaining≈1900. NOT the pacer, NOT rate limiting, NOT the adoption stampede (all ruled out by timeline). Hypothesis: DO boot-latency spikes vs the fixed 12-attempt SSH budget in BuildTestEmbedded. A direct boot-timing experiment failed on a malformed raw create (no droplet was created — no conclusion). This is what kept evicting #707/#680 late in the session.

Relevant PRs / refs (write-time state, verified 22:03Z)

Merged (16): UrlResolver #730, #713 · BTSS #184, #182, #192, #193, #194 · NetLab #99 · CN #509 · BTE #273, #274 · BTRunner #33 · kotlin-build-ci #171 · DODSS #81, #80, #83.

| Open item | State @ 22:03Z | Next action | |---|---|---| | UrlResolver #707 — idempotent command dispatch | MERGED 2026-07-11T19:20:26Z (see 2026-07-12 addendum) | none | | UrlResolver #680 — resilience fixture + reconnect fail-fast | OPEN @ d98bd3da; remote check GREEN, bld-build re-running (see 2026-07-12 addendum — this is the live tail) | follow the 2026-07-12 addendum | | DODSS #84 — pacer re-land; fix/pacer-inflight-release; CI fully green | OPEN, CLEAN, held for user review | user decision; if approved: merge, deploy, soak with the item-6 failure modes in mind | | Item-7 SSH-window flake | unresolved hypothesis | proper boot-timing experiment (create via the droplet-service API, not raw curl), then calibrate the SSH budget in BuildTestEmbedded red-first or file with evidence |

Prod deploys this session (all audit-logged in /root/cn-watchdog.audit): buildtest-server-service 0.0.151→0.0.152 (12:17, 14:55), kotlin-build-ci 0.0.60 (18:5x), droplet-service 0.0.43→0.0.44 (17:51, 20:37), netlab-manager w/ #99 (21:17). Issues closed with evidence: #180, #181, #98, #500, #272, #79. No unsaved work: all checkouts under /code/workspace/ are clean/pushed (the only dirty entry is an untracked scratch clone under UrlResolver-730-rebase/workspace/).

Next steps (re-verify current state first)

1. Re-verify #707's queue entry; if MERGED, rebase #680 (changelog only), re-drive CI, enqueue, merge. Both PRs' content is reviewed and branch-green; only merge-group infra has blocked them.
2. If #707 keeps evicting with the item-7 signature: run the boot-timing experiment properly (create via url://digitalocean-droplets/ createDropletAsync so the request shape is right; time /dev/tcp/<ip>/22), then either calibrate BuildTestEmbedded's SSH attempt budget red-first or file the issue with the timing evidence.
3. User decision: merge+deploy DODSS #84 (pacer re-land). Recommend a supervised deploy window and watching for the item-6 signatures.
4. Optional cleanups: delete boottest-observables-* droplets if the failed experiment left any (it should not have — create returned no id); prune /code/workspace/*.jar build artifacts.
5. The Observables epic (Observable#87) tail (docs-alignment pass, wss transport, JS/Wasm) is unchanged by this session and remains tracked there.

Reusable / operational knowledge

• Publication is per-repo: UrlResolver merge-CI auto-publishes to kotlin.directory; BuildTestEmbedded does NOT (manual community-kotlin-maven-artifact-publishing launch). Probe .pom (200 taken / 401 free) before consumer bumps.
• CN upload-jar caps at 200MB — bigger jars: scp to the route's image path + restart --route-key; url-route keys are url:<name>:; restart prints "Failed: Unknown response format" even on success; "Container not found" on restart = cold (next request starts new jar).
• Deploy builds need dedicated clones — a codex job's dirty working tree blocked a git pull mid-deploy.
• codex exec background wedge: rarely parks in futex_wait right after the stdin banner (0% CPU, 39-byte log). Foreground probes still work. Detect via log-size tripwire (<200B after 3 min), kill, relaunch from a clean cwd.
• DO 429 storms are demand-driven and hourly-sliding: after recovery, re-drive checks ONE at a time; a "sustained headroom" trigger (3 consecutive samples >2500) beats single-sample judgment. Serialized re-drives; never three at once.
• Merge-queue eviction forensics: check the merge-group commit's check-runs (both kotlin.build (remote) AND bld-build can differ), then /root/buildtest-data/runs/<id>/{run.json,build.log,test-events.jsonl} on the host. jq is NOT installed on the host — use python3 -c.
• Parallel-session coordination via /root/cn-watchdog.audit works — write DEPLOY INTENT/DONE lines before/after prod changes; this session both averted a conflict (parkfix supersession) and explained a mystery restart (CN deploy at 20:51) through it.

Addendum (23:20Z): #707's last eviction hit the ORIGINAL stress test — do not blind-re-enqueue

Run dcd534b4 (merge group with #707, finished 23:14Z) failed on stressTestRegistrationAfterWithdrawalRaceUnderContention — the workstream's original target — with:

1/32 parallel-pair register/unregister/re-register cycles failed (3.1%) — the broadcastAnnouncement-after-unregister resurrection race fired under contention. Pre-fix baseline for this configuration is ~88% per run; post-fix expected ~0%.

(plus testNetlabLargeSwarmGossip, netlab-infra family). Two hypotheses, both unverified: (a) #730's fix has a residual ~3% tail under contention (its own amplified verification was 50/50 on the branch — a 3% rate could evade 50 iterations with P(miss)≈22%); (b) #707's idempotent-dispatch changes interact with the withdrawal/announcement path. Next session: amplified A/B (≥200 iterations) of that stress test on plain main vs main+#707 before any further #707 enqueue. Also note the earlier boot-timing datum: single buildtest-spec droplet had port 22 open in 46s at 22:37Z, so the SSH-window failures likely involve 10-concurrent-boot contention or episodic DO slowness — a 10-parallel-create timing experiment is the outstanding decisive test for item 7.

Addendum (2026-07-12 ~16:50Z): #707 MERGED; #680 is the sole remaining item — code fully validated, shepherding bld-build through serial environmental failures

Re-verify everything below before acting (gh pr view 680 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt); this supersedes the 07-09 table and the 23:20Z addendum.

State at write time (verified 16:48Z)

• UrlResolver #707 MERGED 2026-07-11T19:20:26Z. The 23:20Z addendum's A/B question was answered by the tail session's evidence (run eab27546, main-without-#707 reproducing the resurrection flake): the residual tail was in MAIN, #707 exonerated; root fix landed separately as UrlResolver #749 (issue 746).
• UrlResolver #680 OPEN, branch live-projection-resilience-fixture, head d98bd3da = 9bedcb2a (reconnect fail-fast + pin alignment) + 999270c7 (issue-717 test-topology fix) + d98bd3da (a parallel session's coordinate bump to the unclaimed 0.0.634 — legitimate; 0.0.626–633 were already published).
• kotlin.build (remote) is GREEN on d98bd3da (run 234ba4f7, full droplet-sharded suite). bld-build is on its 5th run (job 86670585878, in_progress at write time) after four environmental failures — none of them the PR's code (details below). The identical code content also went bld-build-green on the pre-bump head 999270c7.
• A build-watchman --to-merged (0.0.11) is armed from the driving session; on green it auto-enqueues via GraphQL enqueuePullRequest under the workstream's standing merge-when-green approval. If that session is gone, re-arm: cs launch buildwatchman:build-watchman:0.0.11 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 680 --to-merged (tracked background task, one per PR).

What happened on the way here (each verified, not guessed)

1. The branch's reconnect fail-fast had a real shipped defect caught by CI: testPersistentRpcReconnectDoesNotBlockConcurrentCaller constructs exactly the no-channel topology the fail-fast fires on (no relay, dead provider = only bootstrap peer) and asserts the caller is still inside the reconnect window 1.5s after provider death. Reproduced deterministically on an idle machine. Fix (commit 999270c7): the test's client now gets a live second bootstrap peer that does not serve the service, restoring a legitimate late-discovery channel; fixed test + both sibling fail-fast tests verified 3/3 locally.
2. bld-build failure #1 (05:43Z): 3 netlab tests, No service registered for identifier: digitalocean-droplets — DO-gateway peer unreachable window. Infra.
3. bld-build failure #2 (15:14Z): testReconnectFactoryWaitsForGossipWhenNoAlternativePeersAvailable, 16.3s duration = the full 10s gossip window WAS honored (fail-fast did NOT fire — a fail-fast exit would have finished ~7s), delayed registration happened, gossip simply didn't propagate in time under load. This is the pre-existing gossip-propagation flake class other sessions documented; NOT this PR's defect (same test green on the remote check, same head).
4. bld-build failure #3 (16:29Z): 6 netlab tests, url://netlab-hosted/ health RPC timing out 30s — the netlab-manager was unresponsive and restarted at 16:32Z (verified healthy afterward: 55 environments loaded, DO-gateway connection established, reconcile clean).
5. Also weathered on the remote-check side before it went green: buildtest-host disk-full #2 (5.8G regrown CN heapdumps — the OOM-looping CN child is STILL undiagnosed, flagged in the audit log — plus 861 accumulated run dirs; pruned), a load-110 starvation (HardwareControlFabric daemon metaspace GC death-spiral, restarted by another session 01:50Z), and two DO-quota zero-test kills (quota bursts drain 5000→0 in ~2 min; the kotlin-build-ci reconciler self-healed the second one).
6. BTE#284 hardening handover resolved: BuildTestEmbedded main@018dd8f (0.0.266) audited as a behavioral superset of the closed BTE#284's defect-class items (delete-vs-renewal resurrection race, renewal coalescing, Throwable capture); branch dropped; optional follow-ups (service close() lifecycle, porting deleteBuildRunRacingAsyncRenewalDoesNotResurrectRun.kts) logged in /root/cn-watchdog.audit for the BTE owners.

Next steps (re-verify first)

1. gh pr view 680 … — if MERGED: this workstream is DONE. Delete this handoff (and the parent 2026-07-09 handoff if still present) per the close-out procedure; deletion PRs inherit the handoff pre-approval.
2. If bld-build is green but the PR is not yet merged: confirm the merge-queue entry exists (gh api graphql -f query='{repository(owner:"CodexCoder21Organization",name:"UrlResolver"){mergeQueue{entries(first:10){nodes{position pullRequest{number} state}}}}}'); if absent, enqueue via GraphQL enqueuePullRequest (auto-merge is disabled repo-wide). Watch the queue-ref checks the same way; UNMERGEABLE = real speculative-base conflict, not a stale flag.
3. If bld-build failed AGAIN: read the job log first and classify — netlab acquisition (could not acquire a READY netlab environment) = check url:netlab-hosted:/url:digitalocean-droplets: containers via the CN CLI and re-run --failed once healthy; gossip-wait flake (item 3's signature, full-window duration) = re-run --failed; a genuinely new code failure = investigate before any re-run. gh run rerun 29176780753 --repo CodexCoder21Organization/UrlResolver --failed is the re-fire; re-arm the watchman after every rerun (it exits on terminal states).
4. Standing user-review item unchanged: DODSS #84 pacer re-land remains held for the user.
5. Recurring infra debt worth an owner (audit-logged, not this workstream's scope): the OOM-looping CN child regenerating heapdumps, no buildtest run-dir GC (861 dirs), DO-quota burst consumer (~40 req/s spikes), and the gossip-propagation-under-load flake class.