Handoff: buildtest run dee74ca2 lease-kill incident — four defects root-caused (2 fixed in green unmerged PRs, 2 deployed), user-ordered UrlResolver PR 653 merge in final merge-queue candidate

Written: 2026-07-12 ~21:30 UTC by claude-dee74ca2-leasekill-session.

RE-VERIFY BEFORE ACTING. Every state claim below is a write-time snapshot. PRs merge, merge queues shuffle, CI re-runs, services restart. Authoritative re-checks: gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt for every PR named here; https://buildtest.kotlin.build/api/runs?limit=100&page=1 for run state; the GraphQL mergeQueue(branch:"main") query for queue membership; /root/cn-watchdog.audit on 198.199.106.165 (ssh -p 23) for what other sessions have done since. A fresh query always wins over this document.

Mission summary

User request (2026-07-12 ~06:05Z): "Some sort of infra failure or something, it seems this build failed, we need to understand why and fix it: https://buildtest.kotlin.build/run?id=dee74ca2" — that run was the merge-queue CI candidate for UrlResolver PR 653 ("Fail loud on nested unmarshalable values instead of silent mkString").

Expanded by explicit user orders during the session: "YES, MERGE https://github.com/CodexCoder21Organization/UrlResolver/pull/653 — ensure it gets merged." The user has NOT yet approved merging the two incident-fix PRs (176/307 below) — they are green and waiting for the user's word.

What was found and done (the chain — all verified with evidence at the time)

1. Why dee74ca2 failed: the buildtest server's BuildWatchdog killed it at 06:03:35Z as "build lease not renewed for 16 minutes; consumer presumed abandoned", destroying its 10 droplets — while the run was demonstrably alive (buildMaven PASSED 05:51:08, a test PASSED 06:00:43, shards actively compiling per on-host build.log/test-events.jsonl).
2. Defect 1 — kotlin-build-ci OOM crash-loop (the root of the kill): the githubci kotlin-build-ci instance (DATA_DIR /root/ContainerNursery/data/kotlin-build-ci-githubci) OOM-crash-looped every ~2 min from ~05:12Z: each startup resumed all tracked builds concurrently and fetched FULL unbounded getTestResults collections (1000+ tests each) through one 128MB-heap SJVM connection — live jmap -histo:live showed ~300k net.javadeploy.sjvm.util.ObjRefImpl + ConcurrentHashMaps (~110MB) within 36s of start; -XX:+ExitOnOutOfMemoryError exited, ContainerNursery relaunched, identical replay. Because the consumer's first renewBuildLease fires only after 5 continuous minutes in its poll loop (timer reset at process start), zero lease renewals were sent fleet-wide after 05:10:20Z (verified in /root/ContainerNursery/data/container-logs/url_buildtest_.stdout.log) → every run >15 min was reaped. Side effect: an ~195MB hprof every 2–3 min filled the host disk to 100% (I deleted 26 redundant dumps, kept 2 newest + preserved/, audit-logged).
   • Operational mitigation (done, verified): preserved then pruned in-progress-builds.json from 16 to the 5 genuinely-live entries (copies at /root/preserved-in-progress-builds-*.json), killed the looper; instance stabilized (hours of uptime), renewals resumed at 06:41:22Z and have flowed since.
   • Permanent fix (green, unmerged): kotlin-build-ci PR 176 — paginated getTestResultsPaginated (≤500/page; failed-list cap 200; 4KiB/message UTF-8-safe truncation; 256KiB aggregate; one-shot legacy fallback only before any page consumed; renew-lease-immediately at poll-loop entry; buildtest-api 0.0.43→0.0.49; v0.0.67, head 98eb0ecd). Implemented + twice-reviewed (test-comprehensiveness and adversarial codex passes, both of which found and fixed real defects), 353/353 local, remote CI green.
3. Defect 2 — BuildWatchdog kills progressing runs: the stale-lease check (Check 4) considers only consumer heartbeat + admission activity; the watchdog's own droplet keepalives deliberately do NOT refresh the consumer lease (renewDropletLeases callback returns false), and server-observed run progress was ignored. Fix (green at review time, now needs rebase): BuildTestEmbedded PR 307 — Check 4 uses max(consumer heartbeat, admission activity, server-observed run activity from event ingestion/log flushes), lock-gated activity map cleared on terminal persistence, 3-signal forensic kill message (v0.0.267). Twice-reviewed; at write time its mergeStateStatus is DIRTY because BuildTestEmbedded main moved (see defect 4's hotfix) — needs a rebase before merge.
4. Defect 3 — DigitalOcean droplet-service rate-limit gate self-embargo (caused the parallel provisioning failures + the 11:22Z stall): root-caused by claude-admission-redesign-session; fix DigitalOceanDropletServiceServer PR 89 MERGED 08:04Z but was never deployed. I built main (a233948, PR-89 marker releaseCanaryAfterFailedAdmission verified in the jar) and deployed it to url:digitalocean-droplets: at 12:14Z (sha256 1d3bdaa6…, audit-logged). Holding since.
5. Defect 4 — watchdog-killed sharded runs leak shard threads + admission reservations: jcmd virtual-thread dump proved 20 shard threads of two killed runs still alive 50 min later in provisionShardWithRetry retry-backoff, pinning 2×10 admission reservation units → zero admissions 11:30–12:30Z (7 runs PENDING, none with build.log). I restarted url:buildtest: (12:2xZ, audit-logged) — startup-is-recovery cleared it; the whole backlog admitted within 4 minutes. Upstream fix was claimed and hot-deployed by claude-admission-redesign-session (BuildTestEmbedded 0.0.268, branch terminate-shard-workers-on-stale-kill, deployed 15:08Z per audit) — their lane; check whether their PR merged.
6. PR 653 merge shepherding (user-ordered): long queue saga — multiple candidate rebuilds from queue churn, one merge_conflict eviction (real conflict with UrlResolver PR 749, which touches the same build.kts region and merged 17:35Z), a queue-wide dissolution when ContainerNursery was restarted at 16:17Z by another session (CN metaspace-leak — their lane). After 749 merged, a codex agent rebased 653 onto main (single conflicted file build.kts, both change-sets preserved; 3 targeted tests passed incl. testNestedUnmarshalableValueFailsLoudNotStringified), force-with-lease pushed new head 3aaedeca. Branch checks: bld-build PASS; kotlin.build (remote) failed once on a provisioning-stall infra flake (0 tests ran), re-requested with verified new dispatch, PASSED. build-watchman auto-enqueued at ~20:45Z.

Relevant PRs / refs (state AT WRITE TIME ~21:28Z — re-verify)

| Ref | Write-time state | |---|---| | UrlResolver PR 653 (head 3aaedeca) | OPEN/CLEAN, merge-queue position 1, AWAITING_CHECKS (~47 min into final candidate; queue buildtest run b1e946e7); build-watchman 0.0.12 --to-merged shepherd RUNNING in this session | | kotlin-build-ci PR 176 (OOM fix, v0.0.67) | OPEN/CLEAN — awaiting USER merge approval | | BuildTestEmbedded PR 307 (watchdog fix, v0.0.267) | OPEN/DIRTY — needs rebase onto moved main (0.0.268 shard-fix landed), then USER merge approval | | DigitalOceanDropletServiceServer PR 89 | MERGED 08:04Z; deployed to prod 12:14Z by this session | | Incident audit trail | /root/cn-watchdog.audit on 198.199.106.165, entries tagged [claude-dee74ca2-leasekill-session] (06:25, 06:35, 12:11, 12:14, 12:25, 16:39Z) | | Session memory | ~/.claude/projects/-code/memory/buildtest_lease_kill_kbc_oom_loop_2026_07_12.md (diagnosis playbook for this failure class) |

Next steps (ordered; re-verify current state first)

1. Re-verify everything above (commands in the banner). In particular: did PR 653 reach MERGED? At write time it is queue head with ~10–20 min of candidate suite left and a live shepherd; most likely it merges without help. If the shepherd exited non-zero instead: diagnose per the watch-build skill (its output file is in this session only — start a fresh coursier launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 653 --to-merged). Known residual flake class: buildtest provisioning-stall kills (5-min bound) — remediation is warm https://github-webhooks.wasmserver.com/ (3 curls), gh api -X POST repos/CodexCoder21Organization/UrlResolver/check-suites/<id>/rerequest, verify the check's details_url run id CHANGED, relaunch shepherd.
2. Get user approval, then merge kotlin-build-ci PR 176 (CLEAN at write time). After merge: build fat jar (repo README) and deploy the shared kotlin-build-ci.jar to BOTH routes (kotlin.build:443 + githubci.kotlin.build:443) via CN upload-jar; log DEPLOY INTENT/DONE to /root/cn-watchdog.audit. Until deployed, the OOM defect is latent — another mass-failure night with a big tracked-build backlog would re-trigger the crash-loop (the pruned tracker is the only thing holding it at bay).
3. Rebase BuildTestEmbedded PR 307 onto main (conflicts likely with the 0.0.268 shard-worker-termination change in the same watchdog/service area — resolve preserving both; bump to 0.0.269 if convention requires), re-run its targeted watchdog tests (scripts/test.bash --test watchdog..., names in the PR), re-green CI, then get user approval and merge. After merge, coordinate a buildtest-server redeploy with whoever owns that lane (note: prod already runs a 0.0.268 hotfix build from a branch — merging 307 + redeploying from main supersedes it cleanly).
4. Confirm claude-admission-redesign-session's shard-leak fix PR merged (search BuildTestEmbedded PRs for terminate-shard-workers-on-stale-kill); if it stalled, that lane owns it — coordinate via the audit log rather than duplicating.
5. Optional hardening follow-ups observed but not filed: (a) buildtest provisioning-stall watchdog (5 min) is aggressive vs. real droplet-burst latency — consider raising ONLY via a proper fix at the owning layer (measure real p99 droplet-boot under bursts; do NOT blindly bump the bound); (b) build-watchman treats "enqueue rejected: already in queue" at startup as terminal — file against build-watchman.

Reusable / operational knowledge

• Two-lease model (the core confusion of this incident): DigitalOcean droplet auto-termination leases (~1h) are renewed BY the buildtest server's watchdog every stale/3=5 min; the build lease (15 min) is a consumer↔buildtest-server heartbeat that ONLY renewBuildLease from kotlin-build-ci refreshes — watchdog droplet keepalives intentionally don't count (consumerLeaseRenewed=false).
• Diagnosis playbook for "watchdog killed a live run": (1) compare the kill time against on-host stat/tail of /root/buildtest-data/runs/<id>/{build.log,test-events.jsonl}; (2) grep url_buildtest_.stdout.log for method='renewBuildLease' — a gap since time T = consumer heartbeat blackout since T; (3) ls /root/ContainerNursery/data/container-logs/ | grep kotlin-build-ci — ~70KB stdout files rotating every 1–3 min = crash loop; tail one for Terminating due to java.lang.OutOfMemoryError; (4) jmap -histo:live the youngest instance — SJVM ObjRefImpl explosion = unbounded RPC materialization (healthy instance has ~zero).
• Virtual threads hide from jstack — use jcmd <pid> Thread.dump_to_file -format=text <file> for BuildTestEmbedded ≥0.0.263 (orchestration on virtual threads). This is how the zombie shard threads were proven.
• Merge-queue mechanics (UrlResolver): gh pr merge --auto is disabled; enqueue via GraphQL enqueuePullRequest. Queue candidates are stacked: a conflict with an entry AHEAD shows as UNMERGEABLE/merge_conflict eviction even when the PR is CLEAN vs main. build-watchman 0.0.12 --to-merged auto-re-enqueues once per instance and correctly classifies buildtest executor queue-wait, the 100-result API display cap, and collection-phase log growth.
• Deploys: droplet service = build dropletserviceserver.buildFatJar() → CN upload-jar --route "url:digitalocean-droplets:" (auto-restart; verify a fix-specific class/method in the jar first — stale-cache trap). Always log DEPLOY INTENT/DONE with sha256 to /root/cn-watchdog.audit.
• Webhook receiver must answer in ~10s or GitHub drops the event forever — warm https://github-webhooks.wasmserver.com/ with 2–3 curls before any enqueue/rerequest that must trigger checks; after a rerequest, trust it only when the check's details_url run id changes.