← Priority list

Handoff: ContainerNursery flaky-test fixes — HttpClient-per-request (MERGED) + startup tmp-dir sweep deletes live dirs (PR open), plus two investigated-but-unstarted follow-ups

1. Original request (verbatim intent): *"Use your flakey test skills: https://github.com/CodexCoder21Organization/ContainerNursery/runs/84144960635"* — the `kotlin.build (remote)` run that failed on **`testHttpFacade_xForwardedHeadersReachBackendOverWire`** with `java.net.SocketTimeoutException: Read timed out`. 2. Then: *"Ensure https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 gets merged."* 3. Then: *"proceed with next steps"* — the follow-up items I had listed (two other intermittent droplet flakes seen on #474's first `kotlin.build` run, plus a `kotlin-build-ci` webhook-reliability issue).

hf-2026-06-29-containernursery-httpclient-flake-fix-and-tmpdir-sweep-followups

Edit handoff

Title Summary Markdown body

# Handoff: ContainerNursery flaky-test fixes — HttpClient-per-request (MERGED) + startup tmp-dir sweep deletes live dirs (PR open), plus two investigated-but-unstarted follow-ups

**Written:** 2026-07-06 (workstream began 2026-06-29).

> **⚠️ RE-VERIFY BEFORE ACTING.** Every state claim below (PR states, CI conclusions, `main` HEAD, droplet health) was true only at write time and goes stale fast. Re-check before doing anything:
> - PRs: `gh pr view <num> --repo CodexCoder21Organization/ContainerNursery --json state,mergeStateStatus,statusCheckRollup,mergedAt`
> - A specific commit's checks: `gh api repos/CodexCoder21Organization/ContainerNursery/commits/<sha>/check-runs --jq '.check_runs[] | {name,status,conclusion}'`
> - `kotlin.build (remote)` build detail: open the `buildtest.kotlin.build/run?id=...` URL in the check's summary.
> A fresh query always wins over this document.

## Mission summary

1. Original request (verbatim intent): *"Use your flakey test skills: https://github.com/CodexCoder21Organization/ContainerNursery/runs/84144960635"* — the `kotlin.build (remote)` run that failed on **`testHttpFacade_xForwardedHeadersReachBackendOverWire`** with `java.net.SocketTimeoutException: Read timed out`.
2. Then: *"Ensure https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 gets merged."*
3. Then: *"proceed with next steps"* — the follow-up items I had listed (two other intermittent droplet flakes seen on #474's first `kotlin.build` run, plus a `kotlin-build-ci` webhook-reliability issue).

## What was found and done (the chain)

**1. Root cause of the reported flake (verified from the failed run's own telemetry, not speculation).**
The failed run's `[ThreadMonitor]` log shows the forward coroutine `ContainerNursery-request-3` started at `20:15:20.690`, but the Ktor CIO engine threads (`DefaultDispatcher-worker @ktor-cio-context`) did not appear until `20:15:27.355` — a **~6.7s gap whose only work was constructing a per-request `HttpClient(CIO)`**. Each construction spins up a full CIO engine (selector manager, dispatcher thread pool, connection pool) plus first-use class loading; on the shared 2-core build droplet under contention that cold-start overran the test's 10s read budget. It is the only `tests/*.kts` that performs a *real* per-request CIO forward (the other X-Forwarded tests capture the header map without a wire forward), which is why only it flaked.

**2. Fix for the reported flake — MERGED.** Made `DockerBackedContainer`, `DangerJarFileBackedContainer`, and `CloudRunBackedContainer` each reuse **one** lazily-created `HttpClient(CIO)` (preserving each one's existing config), closed at `shutdown()`/`kill()`; the failing test's fake mirrors that (shared client built at setup, X-Forwarded assertions unchanged). Delivered as **[PR #474](https://github.com/CodexCoder21Organization/ContainerNursery/pull/474)**, squash-merged as `fe10612`. `kotlin.build (remote)` passed **285/285** on the merge.
   - Side note: I first added a 60-way concurrency guard test; it was itself timing-flaky on the droplet (`1/60` forwards hit the shared client's first-use cold-start), so I **removed it** — the fixed `testHttpFacade_xForwardedHeadersReachBackendOverWire` is the real regression guard.
   - Operational: the `kotlin.build (remote)` check on #474 got **stuck `queued`** (the `kotlin-build-ci` App created the check-suite but never dispatched the build — a timed-out webhook delivery). Unstuck it with `gh api --method POST repos/CodexCoder21Organization/ContainerNursery/check-suites/<suite_id>/rerequest`.

**3. "Next steps" — two other flakes that appeared on #474's first `kotlin.build` run were investigated (via code-reading subagents; neither is locally reproducible — the amplifier is the droplet's 2-core + memory pressure + many-JVM contention).**

**3a. `testDangerJarFile_killCleansUpContainerTmpDir` → REAL ContainerNursery defect. Fixed; PR open (see below).**
   `ContainerNurseryServer.cleanupStaleContainerTmpDirs()` (in `src/main/kotlin/org/example/ContainerNurseryServer.kt`) runs at **every** CN startup and unconditionally `deleteRecursively()`-ed **every** `/tmp/container-nursery-jar-*` directory, with **no liveness/ownership check**. During any overlap of two CN instances (a restart while the old instance is still draining, or two CN processes sharing `/tmp` — e.g. the sibling test `testZombieCleanup_dropletE2eMatchesProductionTopology` boots a real `ContainerNurseryServer` subprocess that inherits the real `/tmp`) the newly-starting instance's sweep destroys the **running** instance's containers' temp dirs, corrupting their native extractions (SQLite's `libsqlitejdbc.so`, etc.). The test symptom: `getContainerTmpDir()` returned a non-null dir that no longer existed on disk right after `start()` (its `assertNotNull` passed, its `assertTrue(dir.exists())` failed). This is a **production hazard**, not just a CI flake.
   Fix: the sweep now reads `/proc/*/cmdline` and **skips any candidate dir whose name appears in a live process's command line** (a running container's child JVM is launched with `-Djava.io.tmpdir=<that dir>`, so the dir path is always in its cmdline). Genuinely-orphaned dirs are still cleaned. Added `tests/testCleanupStaleContainerTmpDirs_preservesDirsInUseByLiveProcess.kts` (a real `tail -f` process holds the "live" dir open; sweep must delete only the orphan). I verified the `/proc` detection mechanism independently (a `tail -f` process's `/proc/<pid>/cmdline` does contain the referenced dir path); the fix is the sole code path that skips live dirs, so the test fails-first without it by construction. **I could not run the full kompile build locally** — local builds were stalling in dependency resolution against `kotlin.directory` (registry flakiness on 2026-06-30 / the rebased-workspace `WithArtifactVersionMismatch` phase). Relying on CI: `bld-all-tests` (full kompile suite on a clean GHA runner) **PASSED** on this change.

**3b. `testUrlFacade_autoStartPersistentRouteAndRestartResilience` → NOT a CN defect; upstream dependency latency. NOT started.**
   The test serially performs **two full libp2p host cold-starts** (`UrlProtocol2(bootstrapPeers = emptyList())`) + **four synchronous `registerGlobalService` calls** + two host closes inside one kompile 30s per-test budget; under droplet CPU contention it overran (last log line was registering the 2nd service, blocked in `ensureNetworkJoined`/announce). No deadlock or unbounded loop in `UrlFacadeProvider`/`ContainerNursery`. The real fix is **upstream**: make `registerGlobalService` non-blocking / bounded for a zero-peer node (UrlResolver / `UrlProtocol2`). Owning layer = UrlResolver/UrlProtocol/jvm-libp2p (sources not in the CN repo).

**4. `kotlin-build-ci` webhook reliability — documented, NOT started.** The `kotlin.build (remote)` check intermittently never dispatches because the `kotlin-build-ci-test` GitHub App's webhook deliveries (`push` / `check_suite:requested`) time out at GitHub's 10s limit (the receiver `github-watchman-integration-server.jar` is a lazy CN jarfile at `-Xmx128m`). Durable fix (in `github-watchman` / the integration server): ACK-200-then-async handler + a reconciliation poll. Immediate workaround: re-request the stuck check-suite (command above) or a `/retry` PR comment.

## Relevant PRs / branches / refs (write-time state — RE-VERIFY)

| Item | State @ write time | Link / ref |
|---|---|---|
| PR #474 — reuse one HttpClient per container | **MERGED** (squash `fe10612`, `2026-06-30T01:02:10Z`) | https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 |
| PR #479 — don't delete in-use container temp dirs in the startup sweep | **OPEN**, `mergeStateStatus=BEHIND` (~9 commits behind `main`). Checks on head `a18ae0c`: `bld-all-tests`=**SUCCESS**, `bld-build-fat-jars`=SUCCESS, `Build and test with Gradle`=SUCCESS, **`kotlin.build (remote)`=FAILURE**, **`bld-build-release`=FAILURE** | https://github.com/CodexCoder21Organization/ContainerNursery/pull/479 |
| ↳ branch / head | `fix/cleanup-tmpdir-skip-live-process` @ `a18ae0c541a88814213d8317bb75df8fd57bdb82` (branched off `fe10612`) | — |
| `ContainerNursery` `origin/main` @ write time | `b2be858` (#491); all three checks (`bld-all-tests`, `bld-build-release`, `kotlin.build (remote)`) **SUCCESS** on `main` right now | — |

**Important read on PR #479's red checks:** they are stale (from 2026-06-30, a droplet-infra bad day) and almost certainly **NOT caused by the change**:
- `kotlin.build (remote)` = "142 passed, **144 failed**", but ~all 144 are *"Test did not complete during build execution"* — the signature of the droplet **reaping/OOMing the build mid-run** (infra), not real assertion failures. `buildtest.kotlin.build/run?id=71efd3aa`.
- `bld-build-release` failed at the "Build release zip" step (a packaging/download step, unrelated to a `ContainerNurseryServer.kt` + test change).
- The authoritative code signal, `bld-all-tests` (full kompile suite on a clean runner), **PASSED**. And the same three checks are **green on current `main`** — i.e. the infra is healthy now.

## Next steps (ordered — start by re-verifying)

1. **Re-verify current state first** (banner commands above): PR #479 state/checks, `main` HEAD, and that `main`'s `kotlin.build (remote)` is still green (droplet healthy).
2. **Drive PR #479 to merge** (this is the main open thread):
   a. `cd` into a fresh ContainerNursery clone, `git fetch origin`, `git checkout fix/cleanup-tmpdir-skip-live-process`, `git rebase origin/main`, resolve any conflicts (unlikely — the change is one method in `ContainerNurseryServer.kt` + one new test file), `git push --force-with-lease`.
   b. That triggers fresh CI. Watch `kotlin.build (remote)`; if it sticks `queued`, re-request the check-suite (get the id via `gh api repos/.../commits/<head>/check-suites --jq '.check_suites[]|select(.app.slug=="kotlin-build-ci-test")'`, then `POST .../check-suites/<id>/rerequest`).
   c. If `kotlin.build (remote)` or `bld-build-release` fail again, first check whether it's the **"did not complete during build execution"** mass-reap (infra — re-run) vs a real failure of `testCleanupStaleContainerTmpDirs_preservesDirsInUseByLiveProcess` (real — debug). Cross-check `bld-all-tests`. Do **not** merge with `kotlin.build (remote)` red (absolute rule).
   d. When `kotlin.build (remote)` is green, enqueue: `enqueuePullRequest` GraphQL mutation with the PR node id (`gh pr view 479 --json id`). The `main` ruleset is a merge queue (SQUASH, ALLGREEN, required check = `kotlin.build (remote)`); `gh pr merge --auto` is rejected. Poll `gh pr view 479 --json state` until `MERGED`. Do **not** use `--delete-branch`. **This one needs the user's merge approval** (it is a code PR, not a handoff).
3. **(Optional, upstream) Fix `testUrlFacade_autoStartPersistentRouteAndRestartResilience` at the owning layer.** Clone `UrlResolver` and `UrlProtocol` (use the `github-repos` skill). Confirm whether `registerGlobalService` → `ensureNetworkJoined`/announce blocks on a zero-peer node waiting for an ACK/routing update that never arrives. Make it bounded/non-blocking (fail-first unit test in UrlResolver first), `publish-maven-artifact`, bump the dep in ContainerNursery, verify the CN test passes. This is speculative until the upstream source is read — flag as hypothesis.
4. **(Optional, infra) Fix `kotlin-build-ci` webhook reliability** in `github-watchman-integration-server`: ACK the webhook 200 immediately then process asynchronously, and add a reconciliation poll so a dropped delivery self-heals. Consider not hosting the webhook endpoint as an idle-reaped `-Xmx128m` CN jarfile.

## Reusable / operational knowledge

- **`kotlin.build (remote)` stuck `queued`** (suite created, `updated_at == created_at`, never dispatched, while `buildtest.kotlin.build` and `kotlin.build` are both HTTP 200) → `gh api --method POST repos/CodexCoder21Organization/ContainerNursery/check-suites/<suite_id>/rerequest` dispatches it within ~20s. (Also `/retry` PR comment, or `POST /app/hook/deliveries/{id}/attempts` with the app JWT.)
- **"Test did not complete during build execution"** across dozens of tests = the droplet reaped/OOMed the build mid-run (infra), **not** real failures. Cross-check the GHA `bld-all-tests` job (full kompile suite on a clean runner) for the real code signal.
- **ContainerNursery merge mechanics:** `main` ruleset enforces a **merge queue** (`SQUASH`, `ALLGREEN`, required status check = `["kotlin.build (remote)"]`; classic branch-protection API 404s "Branch not protected"). `gh pr merge --auto` / `--squash --auto` are rejected ("Auto merge is not allowed"). Enqueue via GraphQL `enqueuePullRequest(input:{pullRequestId})`; it refuses with *"Required status check 'kotlin.build (remote)' is expected"* until that check is **posted + green** on the PR head (a `BEHIND` branch is otherwise fine — the queue integrates with `main`). The queue builds `gh-readonly-queue/main/pr-<n>-<basesha>` and squash-merges on green.
- **Local kompile builds are unreliable right now:** they stall for many minutes in dependency resolution against `kotlin.directory` (and the post-#472 workspace hits a slow `WithArtifactVersionMismatch` phase). Validate via CI (`bld-all-tests` builds reliably on a clean runner) or via targeted independent checks (e.g. I validated the `/proc` cmdline detection with a standalone `tail -f` + `/proc/<pid>/cmdline` grep).
- **Memory written this workstream** (in the operator's auto-memory): `cn_per_request_httpclient_coldstart_flake_shared_client.md`, `kotlin_build_remote_check_stuck_queued_rerequest.md`.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: ContainerNursery flaky-test fixes — HttpClient-per-request (MERGED) + startup tmp-dir sweep deletes live dirs (PR open), plus two investigated-but-unstarted follow-ups

Written: 2026-07-06 (workstream began 2026-06-29).

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below (PR states, CI conclusions, main HEAD, droplet health) was true only at write time and goes stale fast. Re-check before doing anything:

PRs: gh pr view <num> --repo CodexCoder21Organization/ContainerNursery --json state,mergeStateStatus,statusCheckRollup,mergedAt

A specific commit's checks: gh api repos/CodexCoder21Organization/ContainerNursery/commits/<sha>/check-runs --jq '.check_runs[] | {name,status,conclusion}'

kotlin.build (remote) build detail: open the buildtest.kotlin.build/run?id=... URL in the check's summary. A fresh query always wins over this document.

Mission summary

Original request (verbatim intent): "Use your flakey test skills: https://github.com/CodexCoder21Organization/ContainerNursery/runs/84144960635" — the kotlin.build (remote) run that failed on testHttpFacade_xForwardedHeadersReachBackendOverWire with java.net.SocketTimeoutException: Read timed out.
Then: "Ensure https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 gets merged."
Then: "proceed with next steps" — the follow-up items I had listed (two other intermittent droplet flakes seen on #474's first kotlin.build run, plus a kotlin-build-ci webhook-reliability issue).

What was found and done (the chain)

1. Root cause of the reported flake (verified from the failed run's own telemetry, not speculation). The failed run's [ThreadMonitor] log shows the forward coroutine ContainerNursery-request-3 started at 20:15:20.690, but the Ktor CIO engine threads (DefaultDispatcher-worker @ktor-cio-context) did not appear until 20:15:27.355 — a ~6.7s gap whose only work was constructing a per-request HttpClient(CIO). Each construction spins up a full CIO engine (selector manager, dispatcher thread pool, connection pool) plus first-use class loading; on the shared 2-core build droplet under contention that cold-start overran the test's 10s read budget. It is the only tests/*.kts that performs a real per-request CIO forward (the other X-Forwarded tests capture the header map without a wire forward), which is why only it flaked.

2. Fix for the reported flake — MERGED. Made DockerBackedContainer, DangerJarFileBackedContainer, and CloudRunBackedContainer each reuse one lazily-created HttpClient(CIO) (preserving each one's existing config), closed at shutdown()/kill(); the failing test's fake mirrors that (shared client built at setup, X-Forwarded assertions unchanged). Delivered as PR #474, squash-merged as fe10612. kotlin.build (remote) passed 285/285 on the merge.

Side note: I first added a 60-way concurrency guard test; it was itself timing-flaky on the droplet (1/60 forwards hit the shared client's first-use cold-start), so I removed it — the fixed testHttpFacade_xForwardedHeadersReachBackendOverWire is the real regression guard.
Operational: the kotlin.build (remote) check on #474 got stuck queued (the kotlin-build-ci App created the check-suite but never dispatched the build — a timed-out webhook delivery). Unstuck it with gh api --method POST repos/CodexCoder21Organization/ContainerNursery/check-suites/<suite_id>/rerequest.

3. "Next steps" — two other flakes that appeared on #474's first kotlin.build run were investigated (via code-reading subagents; neither is locally reproducible — the amplifier is the droplet's 2-core + memory pressure + many-JVM contention).

3a. testDangerJarFile_killCleansUpContainerTmpDir → REAL ContainerNursery defect. Fixed; PR open (see below). ContainerNurseryServer.cleanupStaleContainerTmpDirs() (in src/main/kotlin/org/example/ContainerNurseryServer.kt) runs at every CN startup and unconditionally deleteRecursively()-ed every /tmp/container-nursery-jar-* directory, with no liveness/ownership check. During any overlap of two CN instances (a restart while the old instance is still draining, or two CN processes sharing /tmp — e.g. the sibling test testZombieCleanup_dropletE2eMatchesProductionTopology boots a real ContainerNurseryServer subprocess that inherits the real /tmp) the newly-starting instance's sweep destroys the running instance's containers' temp dirs, corrupting their native extractions (SQLite's libsqlitejdbc.so, etc.). The test symptom: getContainerTmpDir() returned a non-null dir that no longer existed on disk right after start() (its assertNotNull passed, its assertTrue(dir.exists()) failed). This is a production hazard, not just a CI flake. Fix: the sweep now reads /proc/*/cmdline and skips any candidate dir whose name appears in a live process's command line (a running container's child JVM is launched with -Djava.io.tmpdir=<that dir>, so the dir path is always in its cmdline). Genuinely-orphaned dirs are still cleaned. Added tests/testCleanupStaleContainerTmpDirs_preservesDirsInUseByLiveProcess.kts (a real tail -f process holds the "live" dir open; sweep must delete only the orphan). I verified the /proc detection mechanism independently (a tail -f process's /proc/<pid>/cmdline does contain the referenced dir path); the fix is the sole code path that skips live dirs, so the test fails-first without it by construction. I could not run the full kompile build locally — local builds were stalling in dependency resolution against kotlin.directory (registry flakiness on 2026-06-30 / the rebased-workspace WithArtifactVersionMismatch phase). Relying on CI: bld-all-tests (full kompile suite on a clean GHA runner) PASSED on this change.

3b. testUrlFacade_autoStartPersistentRouteAndRestartResilience → NOT a CN defect; upstream dependency latency. NOT started. The test serially performs two full libp2p host cold-starts (UrlProtocol2(bootstrapPeers = emptyList())) + four synchronous registerGlobalService calls + two host closes inside one kompile 30s per-test budget; under droplet CPU contention it overran (last log line was registering the 2nd service, blocked in ensureNetworkJoined/announce). No deadlock or unbounded loop in UrlFacadeProvider/ContainerNursery. The real fix is upstream: make registerGlobalService non-blocking / bounded for a zero-peer node (UrlResolver / UrlProtocol2). Owning layer = UrlResolver/UrlProtocol/jvm-libp2p (sources not in the CN repo).

4. kotlin-build-ci webhook reliability — documented, NOT started. The kotlin.build (remote) check intermittently never dispatches because the kotlin-build-ci-test GitHub App's webhook deliveries (push / check_suite:requested) time out at GitHub's 10s limit (the receiver github-watchman-integration-server.jar is a lazy CN jarfile at -Xmx128m). Durable fix (in github-watchman / the integration server): ACK-200-then-async handler + a reconciliation poll. Immediate workaround: re-request the stuck check-suite (command above) or a /retry PR comment.

Relevant PRs / branches / refs (write-time state — RE-VERIFY)

| Item | State @ write time | Link / ref | |---|---|---| | PR #474 — reuse one HttpClient per container | MERGED (squash fe10612, 2026-06-30T01:02:10Z) | https://github.com/CodexCoder21Organization/ContainerNursery/pull/474 | | PR #479 — don't delete in-use container temp dirs in the startup sweep | OPEN, mergeStateStatus=BEHIND (~9 commits behind main). Checks on head a18ae0c: bld-all-tests=SUCCESS, bld-build-fat-jars=SUCCESS, Build and test with Gradle=SUCCESS, kotlin.build (remote)=FAILURE, bld-build-release=FAILURE | https://github.com/CodexCoder21Organization/ContainerNursery/pull/479 | | ↳ branch / head | fix/cleanup-tmpdir-skip-live-process @ a18ae0c541a88814213d8317bb75df8fd57bdb82 (branched off fe10612) | — | | ContainerNursery origin/main @ write time | b2be858 (#491); all three checks (bld-all-tests, bld-build-release, kotlin.build (remote)) SUCCESS on main right now | — |

Important read on PR #479's red checks: they are stale (from 2026-06-30, a droplet-infra bad day) and almost certainly NOT caused by the change:

kotlin.build (remote) = "142 passed, 144 failed", but ~all 144 are "Test did not complete during build execution" — the signature of the droplet reaping/OOMing the build mid-run (infra), not real assertion failures. buildtest.kotlin.build/run?id=71efd3aa.
bld-build-release failed at the "Build release zip" step (a packaging/download step, unrelated to a ContainerNurseryServer.kt + test change).
The authoritative code signal, bld-all-tests (full kompile suite on a clean runner), PASSED. And the same three checks are green on current main — i.e. the infra is healthy now.

Next steps (ordered — start by re-verifying)

Re-verify current state first (banner commands above): PR #479 state/checks, main HEAD, and that main's kotlin.build (remote) is still green (droplet healthy).
Drive PR #479 to merge (this is the main open thread): a. cd into a fresh ContainerNursery clone, git fetch origin, git checkout fix/cleanup-tmpdir-skip-live-process, git rebase origin/main, resolve any conflicts (unlikely — the change is one method in ContainerNurseryServer.kt + one new test file), git push --force-with-lease. b. That triggers fresh CI. Watch kotlin.build (remote); if it sticks queued, re-request the check-suite (get the id via gh api repos/.../commits/<head>/check-suites --jq '.check_suites[]|select(.app.slug=="kotlin-build-ci-test")', then POST .../check-suites/<id>/rerequest). c. If kotlin.build (remote) or bld-build-release fail again, first check whether it's the "did not complete during build execution" mass-reap (infra — re-run) vs a real failure of testCleanupStaleContainerTmpDirs_preservesDirsInUseByLiveProcess (real — debug). Cross-check bld-all-tests. Do not merge with kotlin.build (remote) red (absolute rule). d. When kotlin.build (remote) is green, enqueue: enqueuePullRequest GraphQL mutation with the PR node id (gh pr view 479 --json id). The main ruleset is a merge queue (SQUASH, ALLGREEN, required check = kotlin.build (remote)); gh pr merge --auto is rejected. Poll gh pr view 479 --json state until MERGED. Do not use --delete-branch. This one needs the user's merge approval (it is a code PR, not a handoff).
(Optional, upstream) Fix testUrlFacade_autoStartPersistentRouteAndRestartResilience at the owning layer. Clone UrlResolver and UrlProtocol (use the github-repos skill). Confirm whether registerGlobalService → ensureNetworkJoined/announce blocks on a zero-peer node waiting for an ACK/routing update that never arrives. Make it bounded/non-blocking (fail-first unit test in UrlResolver first), publish-maven-artifact, bump the dep in ContainerNursery, verify the CN test passes. This is speculative until the upstream source is read — flag as hypothesis.
(Optional, infra) Fix kotlin-build-ci webhook reliability in github-watchman-integration-server: ACK the webhook 200 immediately then process asynchronously, and add a reconciliation poll so a dropped delivery self-heals. Consider not hosting the webhook endpoint as an idle-reaped -Xmx128m CN jarfile.

Reusable / operational knowledge

kotlin.build (remote) stuck queued (suite created, updated_at == created_at, never dispatched, while buildtest.kotlin.build and kotlin.build are both HTTP 200) → gh api --method POST repos/CodexCoder21Organization/ContainerNursery/check-suites/<suite_id>/rerequest dispatches it within ~20s. (Also /retry PR comment, or POST /app/hook/deliveries/{id}/attempts with the app JWT.)
"Test did not complete during build execution" across dozens of tests = the droplet reaped/OOMed the build mid-run (infra), not real failures. Cross-check the GHA bld-all-tests job (full kompile suite on a clean runner) for the real code signal.
ContainerNursery merge mechanics: main ruleset enforces a merge queue (SQUASH, ALLGREEN, required status check = ["kotlin.build (remote)"]; classic branch-protection API 404s "Branch not protected"). gh pr merge --auto / --squash --auto are rejected ("Auto merge is not allowed"). Enqueue via GraphQL enqueuePullRequest(input:{pullRequestId}); it refuses with "Required status check 'kotlin.build (remote)' is expected" until that check is posted + green on the PR head (a BEHIND branch is otherwise fine — the queue integrates with main). The queue builds gh-readonly-queue/main/pr-<n>-<basesha> and squash-merges on green.
Local kompile builds are unreliable right now: they stall for many minutes in dependency resolution against kotlin.directory (and the post-#472 workspace hits a slow WithArtifactVersionMismatch phase). Validate via CI (bld-all-tests builds reliably on a clean runner) or via targeted independent checks (e.g. I validated the /proc cmdline detection with a standalone tail -f + /proc/<pid>/cmdline grep).
Memory written this workstream (in the operator's auto-memory): cn_per_request_httpclient_coldstart_flake_shared_client.md, kotlin_build_remote_check_stuck_queued_rerequest.md.