← Priority list

Handoff: UrlResolver netlab network-scenario tests (PR #631) — tests correct & once-green, now red again on systemic fleet-capacity flakiness

The user asked (verbatim): *"We want to have a bunch of comprehensive tests for UrlResolver repository which utilize the netlab to simulate more realistic network scenarios. Especially situations around high network latency, high packet loss, peers going offline, etc."*

hf-2026-06-11-urlresolver-netlab-network-scenario-tests-pr631

Edit handoff

Title Summary Markdown body

# Handoff: UrlResolver netlab network-scenario tests (PR #631) — tests correct & once-green, now red again on systemic fleet-capacity flakiness

## Written 2026-07-06 (effort began 2026-06-11)

> **⚠️ RE-VERIFY BEFORE ACTING.** Every state claim below was true only at write time (2026-07-06). PRs move, branches get force-pushed, CI re-runs, and the shared host's load/droplet picture changes hour to hour. Re-verify before doing anything:
> - PR: `gh pr view 631 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid`
> - Host/DO capacity (the actual gating factor): fabric snapshot of `198.199.106.165` load + DigitalOcean droplet count (see **Reusable knowledge** for the exact command). **A fresh query always wins over this document.**

---

## Mission

netlab = the managed `url://netlab-hosted/` service (a ContainerNursery app on `198.199.106.165`) that provisions a fresh DigitalOcean droplet per environment and runs Docker network topologies on it, with `tc netem` link impairment. The deliverable is netlab-based kompile tests in the UrlResolver repo that drive **real** `UrlResolver` nodes under adverse network conditions.

## What was found and done (the chain)

1. **Wrote three tests** (branch `netlab-scenario-tests-v2`, single commit, currently head `396d11c6`), each one provisioned droplet / multiple sequential phases (matching the repo convention of bundling scenarios to limit concurrent droplet count):
   - `tests/testNetlabSevereImpairmentScenarios.kts` — severe provider latency (800ms±100ms), impaired **client** link (400ms±50ms + 10% loss), impaired **relay hub** on a cross-NAT path (300ms±30ms + 10% loss), 512 kbit/s rate-limited **sandboxed bytecode fetch**, then recovery after clearing.
   - `tests/testNetlabGossipUnderDegradedLinks.kts` — service **announcement** and **withdrawal** gossip must propagate through a hub degraded *before* registration (250ms±25ms + 15% loss via `SERVE_SIGNAL`). The negative withdrawal check runs on a **healthy** network (impairment cleared first) so infra flakiness can't false-pass it.
   - `tests/testNetlabPeersGoingOffline.kts` — abrupt container removal (`apply(prune=true)`) of a NAT'd provider; **HA takeover** by a standby registering the *same* URL after the original dies; survival of **root-bootstrap death** via a client bootstrapping off a surviving peer. Every offline assertion is gated reachability-must-break-**first**.

2. **Added `NetlabTestEnv` helpers** (`src/foundation/url/resolver/NetlabTestEnv.kt`), re-applied cleanly onto current main (main *already* had upload-resilience and a no-shared-worker `defaultNetlabCandidates`, so those were **not** re-added):
   - tc-netem impairment: `setLatency` / `setPacketLossRate` / `setRateLimit` / `clearImpairment` (composing per-dimension RPCs via `rpcWithRetry`, assert `exitCode == 0`).
   - `apply(prune: Boolean = false)` — deterministic peer removal (a deleted container can't be resurrected by a restart policy and false-pass an offline assertion).
   - **Log-offset-scoped resolve polling** — the central anti-false-pass mechanism: `logLength` (snapshot the host's captured stdout length *before* launching a resolve) + `awaitResolveResultFrom(fromIndex)` + `pollResolveRepeatedly`. Each attempt is judged **only** by the OK/FAIL marker printed at/after its own offset, so a stale marker from a prior phase can never satisfy the wait.
   - `acquireOnce` now wraps the `create` RPC in try/catch so a transient create-timeout counts as a *candidate* failure (retried by `acquire()`'s 6 outer rounds) instead of crashing the test.

3. **Verified all three pass locally** against the managed `url://netlab-hosted/` (each provisions a real droplet), built against `foundation.url:protocol:0.0.303`. VERIFIED — I ran them via `NETLAB_SERVICE_URL=url://netlab-hosted/ scripts/test.bash --test <name>` and saw `ALL TESTS PASSED`. The peers-offline test took 4 attempts *only* because of infra: every failure was `could not acquire a READY netlab environment` / `create timed out` — it **never reached test logic**. Test-file protocol pins were bumped `0.0.281 → 0.0.303` to match main (protocol PR is [UrlResolver #628](https://github.com/CodexCoder21Organization/UrlResolver/pull/628)).

4. **Root-caused a ContainerNursery OOM crash-loop** encountered mid-work (evidence: on-host heap-dump histogram of CN pid 2078831 — ~31,074 `MuxChannel` + 247,446 `ChannelHandlerContext` + 29,887 inbound `Negotiator$ResponderHandler` + 15,468 `broadcastGossipMessage…jobs`; object-count leak, byte[] only 38 MB). This was the inbound mux-substream leak fixed **upstream** in `foundation.url:protocol` #302/#303 and pulled into UrlResolver via [#628](https://github.com/CodexCoder21Organization/UrlResolver/pull/628) — that fix (`0.0.303`) is what stabilized CN enough to get a green run. Fix-at-the-owning-layer: this was a protocol bug, not a test or CN-config issue.

5. **Achieved BOTH checks GREEN once** on 2026-06-16: bld-build SUCCESS (all 892 tests incl. my 3 + the 5 pre-existing netlab tests) and kotlin.build (remote) SUCCESS (a full ~47-min run, 08:29→09:16Z). The green run happened only after I waited for the shared host to drain to load ~5 with ~4-7 droplets active, then retried in that calm window.

6. **State regressed by 2026-07-06** (write time). The branch was rebased/force-pushed (new head `396d11c6`, same single commit, same 4 files) and CI re-ran red: kotlin.build FAILURE, bld-build CANCELLED (~90 min then cancelled). This is **not** a regression in the tests — build-rules for the failing kotlin.build run [`205f17d3`](https://buildtest.kotlin.build/run?id=205f17d3) show `foundation.url.resolver.buildMaven()` **PASSED**, so the failure is in the test phase, and **main itself is red today** for the same reason (see baseline below). VERIFIED at write time: DO had **32 droplets active (over the 25 limit), 22 of them buildtest** — the fleet was heavily over-capacity, so netlab droplet provisioning fails for everyone. (I could not read the exact failing test names — the buildtest `/api/test-results` endpoint was timing out under host load — but the mechanism is the well-established netlab-acquire flake, not a code defect.)

### The systemic mechanism (verified across multiple runs)

Each UrlResolver CI run provisions ~8 netlab droplets (one per netlab test; kompile runs them concurrently). bld-build + kotlin.build per PR ≈ 16 concurrent provisions; the DO account limit is 25. When other CI/PRs run at the same time, the account saturates and DigitalOcean rejects droplet creation (`provisioningStatus=CREATING_DROPLET → FAILED`) → netlab tests fail to acquire. The host `ubuntu-s-1vcpu-1gb` runs ~20 CN JVMs on **1 vCPU** (chronic load avg ~26, spikes to 58 under herd); kotlin.build's builder runs *on* that host, so under extreme load it can even fail at build-setup in ~3 min with zero tests run. Calm windows (low droplet count, load ~5) do exist and are when CI goes green.

## Relevant PRs / refs (write-time state)

| Ref | State @ write time | Notes |
|-----|--------------------|-------|
| [UrlResolver PR #631](https://github.com/CodexCoder21Organization/UrlResolver/pull/631) | **OPEN, BEHIND main, kotlin.build FAILURE / bld-build CANCELLED** | The deliverable. Branch `netlab-scenario-tests-v2`, head `396d11c6`. 4 files: `NetlabTestEnv.kt` + the 3 test `.kts`. Was BOTH-GREEN once (2026-06-16 09:16Z). Do NOT merge without user approval. |
| UrlResolver `main` | **RED today** (bld-build failure 17:50Z, kotlin.build failure 17:42Z; an earlier bld-build succeeded 16:28Z) | head `3c127de587c8`. Confirms the red is fleet-wide, not PR-specific. |
| [UrlResolver #628](https://github.com/CodexCoder21Organization/UrlResolver/pull/628) | MERGED | Bumped protocol → `0.0.303` (inbound mux-substream OOM fix #302/#303). The fix for the CN crash-loop found in step 4. |
| Old branch `netlab-network-scenario-tests` | abandoned | Based on stale `b7e00e0a`; superseded by `netlab-scenario-tests-v2`. Ignore. |

**Unsaved work:** none. All work is on `origin/netlab-scenario-tests-v2` (commit `396d11c6`). The local `/code/UrlResolver` checkout was cleaned up between sessions — that is expected; nothing was lost.

## Next steps

1. **Re-verify current state first** (see banner). Especially: is PR #631 still open? did main go green (clearing the baseline)? what is the *current* DO droplet count and host load?
2. **Rebase PR #631 onto current main.** It is BEHIND (`git fetch origin && git rebase origin/main`, force-push with `--force-with-lease`). The only files it touches are `NetlabTestEnv.kt` + 3 test `.kts`; if main changed `NetlabTestEnv.kt` again, re-apply the 4 helper additions (impair setters, `apply(prune=)`, the `logLength`/`awaitResolveResultFrom`/`pollResolveRepeatedly` trio, and the `acquireOnce` create try/catch) — main may already have upload-resilience and no-shared-worker candidates, so don't duplicate those. Keep the test protocol pins matching main's (`foundation.url:protocol` version).
3. **Retry CI ONLY in a calm window.** Before triggering, check host load + DO droplet count (command below). Wait until droplets are well under 25 (ideally ≤ ~10) and 1-min load is low (~5, not 26+). Then `gh pr comment 631 --repo CodexCoder21Organization/UrlResolver --body "/retry"` for kotlin.build; bld-build re-triggers on push. Early-UTC windows have historically been calmest (the green run was ~08:29Z; main's earlier green was 00:57Z). Consider the `watch-build` skill to shepherd it.
4. **When green, ask the user before merging** — this is a code PR, not the pre-approved handoff exception.
5. **If CI proves un-greenable across several genuine calm-window retries**, the principled root-cause fix (do NOT paper over) is a **cross-process provision-concurrency gate** in `NetlabTestEnv.acquire()` — a filesystem-lock semaphore bounding simultaneous droplet acquisitions under the DO limit, which would make *all* netlab tests CI-reliable, not just these. This was deliberately **not** done here because main's bld-build has gone green without it, so it is not *required* for green and would be scope-creep on shared infra. It becomes justified if calm windows stop being reliably reachable. This is a broader fleet-capacity concern (1-vCPU host, 25-droplet account, N concurrent droplet-provisioning tests) that may warrant a separate workstream and user input.

## Reusable / operational knowledge

- **Run a netlab test locally against the managed service:** `NETLAB_SERVICE_URL=url://netlab-hosted/ scripts/test.bash --test testNetlabSevereImpairmentScenarios`. Each provisions a real DO droplet (~15-20 min when infra is healthy; longer with acquire retries). Runs 1 droplet at a time (sequential) so it's a light marginal load.
- **Check host load + DO droplet count (the gating signal) via HardwareControlFabric** — the fabric CLI's `--args` is naively space-split and can't take dash-leading tokens, so ship a script file:
  ```bash
  FABRIC="coursier launch community.kotlin.hardwarecontrolfabric:cli:0.2.6 -r https://kotlin.directory -- -H 198.199.106.165 -c $HOME/.config/hardware-control-fabric/client_cert.pem -k $HOME/.config/hardware-control-fabric/fabric_key.pkcs8 --ca-cert $HOME/.config/hardware-control-fabric/server_cert.pem"
  # write a /tmp/load.sh that reads the do-api-key from the netlab-manager process args and curls the DO droplets API, then:
  $FABRIC write-file --remote-output-path /tmp/load.sh --local-input-path /tmp/load.sh
  $FABRIC launch --cmd /bin/sh --args "/tmp/load.sh" --wait
  ```
  The DO API key is embedded in the running `netlab-manager-server.jar` args: `ps -eo args | grep -o 'do-api-key [a-z0-9_]*'`. `launch --wait` hard-caps at 60s — for longer jobs run detached (`( … ) > out.txt &`) and poll the file with `read-file`. Never `pkill -f <pattern>` from a Claude Bash shell — its own command line matches and it self-kills; kill by explicit PID.
- **Read a kotlin.build run's outcome** (the WUI is a JS SPA; scrape the JSON APIs instead): build outcome `https://buildtest.kotlin.build/api/build-rules?id=<runId>` (shows per-rule PASSED/FAILED — an empty result or a fast ~3-min run means it failed at build-setup, i.e. host too loaded to build); test failures `https://buildtest.kotlin.build/api/test-results?id=<runId>`. Both time out when the host is overloaded — that timeout is itself a load signal.
- **`buildMaven()` build gotchas:** clear `~/.aibuildcaches/_code_UrlResolver` before re-running a test after editing `src/` (stale cached artifact otherwise); the produced jar is a directory containing `resolver-<ver>.jar` (verify hardened code made it in by unzipping and grepping the class).
- **Related handoffs / memory:** the CN OOM + protocol `0.0.303` rollout, the netlab netem impairment feature, and the netlab manager provisioning-concurrency history are all prior context; see the other `handoffs/*netlab*` and `*containernursery*` files in this directory.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: UrlResolver netlab network-scenario tests (PR #631) — tests correct & once-green, now red again on systemic fleet-capacity flakiness

Written 2026-07-06 (effort began 2026-06-11)

⚠️ RE-VERIFY BEFORE ACTING. Every state claim below was true only at write time (2026-07-06). PRs move, branches get force-pushed, CI re-runs, and the shared host's load/droplet picture changes hour to hour. Re-verify before doing anything:

PR: gh pr view 631 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt,headRefOid

Host/DO capacity (the actual gating factor): fabric snapshot of 198.199.106.165 load + DigitalOcean droplet count (see Reusable knowledge for the exact command). A fresh query always wins over this document.

Mission

The user asked (verbatim): "We want to have a bunch of comprehensive tests for UrlResolver repository which utilize the netlab to simulate more realistic network scenarios. Especially situations around high network latency, high packet loss, peers going offline, etc."

netlab = the managed url://netlab-hosted/ service (a ContainerNursery app on 198.199.106.165) that provisions a fresh DigitalOcean droplet per environment and runs Docker network topologies on it, with tc netem link impairment. The deliverable is netlab-based kompile tests in the UrlResolver repo that drive real UrlResolver nodes under adverse network conditions.

What was found and done (the chain)

Wrote three tests (branch netlab-scenario-tests-v2, single commit, currently head 396d11c6), each one provisioned droplet / multiple sequential phases (matching the repo convention of bundling scenarios to limit concurrent droplet count):
- tests/testNetlabSevereImpairmentScenarios.kts — severe provider latency (800ms±100ms), impaired client link (400ms±50ms + 10% loss), impaired relay hub on a cross-NAT path (300ms±30ms + 10% loss), 512 kbit/s rate-limited sandboxed bytecode fetch, then recovery after clearing.
- tests/testNetlabGossipUnderDegradedLinks.kts — service announcement and withdrawal gossip must propagate through a hub degraded before registration (250ms±25ms + 15% loss via SERVE_SIGNAL). The negative withdrawal check runs on a healthy network (impairment cleared first) so infra flakiness can't false-pass it.
- tests/testNetlabPeersGoingOffline.kts — abrupt container removal (apply(prune=true)) of a NAT'd provider; HA takeover by a standby registering the same URL after the original dies; survival of root-bootstrap death via a client bootstrapping off a surviving peer. Every offline assertion is gated reachability-must-break-first.
Added NetlabTestEnv helpers (src/foundation/url/resolver/NetlabTestEnv.kt), re-applied cleanly onto current main (main already had upload-resilience and a no-shared-worker defaultNetlabCandidates, so those were not re-added):
- tc-netem impairment: setLatency / setPacketLossRate / setRateLimit / clearImpairment (composing per-dimension RPCs via rpcWithRetry, assert exitCode == 0).
- apply(prune: Boolean = false) — deterministic peer removal (a deleted container can't be resurrected by a restart policy and false-pass an offline assertion).
- Log-offset-scoped resolve polling — the central anti-false-pass mechanism: logLength (snapshot the host's captured stdout length before launching a resolve) + awaitResolveResultFrom(fromIndex) + pollResolveRepeatedly. Each attempt is judged only by the OK/FAIL marker printed at/after its own offset, so a stale marker from a prior phase can never satisfy the wait.
- acquireOnce now wraps the create RPC in try/catch so a transient create-timeout counts as a candidate failure (retried by acquire()'s 6 outer rounds) instead of crashing the test.
Verified all three pass locally against the managed url://netlab-hosted/ (each provisions a real droplet), built against foundation.url:protocol:0.0.303. VERIFIED — I ran them via NETLAB_SERVICE_URL=url://netlab-hosted/ scripts/test.bash --test <name> and saw ALL TESTS PASSED. The peers-offline test took 4 attempts only because of infra: every failure was could not acquire a READY netlab environment / create timed out — it never reached test logic. Test-file protocol pins were bumped 0.0.281 → 0.0.303 to match main (protocol PR is UrlResolver #628).
Root-caused a ContainerNursery OOM crash-loop encountered mid-work (evidence: on-host heap-dump histogram of CN pid 2078831 — ~31,074 MuxChannel + 247,446 ChannelHandlerContext + 29,887 inbound Negotiator$ResponderHandler + 15,468 broadcastGossipMessage…jobs; object-count leak, byte[] only 38 MB). This was the inbound mux-substream leak fixed upstream in foundation.url:protocol #302/#303 and pulled into UrlResolver via #628 — that fix (0.0.303) is what stabilized CN enough to get a green run. Fix-at-the-owning-layer: this was a protocol bug, not a test or CN-config issue.
Achieved BOTH checks GREEN once on 2026-06-16: bld-build SUCCESS (all 892 tests incl. my 3 + the 5 pre-existing netlab tests) and kotlin.build (remote) SUCCESS (a full ~47-min run, 08:29→09:16Z). The green run happened only after I waited for the shared host to drain to load ~5 with ~4-7 droplets active, then retried in that calm window.
State regressed by 2026-07-06 (write time). The branch was rebased/force-pushed (new head 396d11c6, same single commit, same 4 files) and CI re-ran red: kotlin.build FAILURE, bld-build CANCELLED (~90 min then cancelled). This is not a regression in the tests — build-rules for the failing kotlin.build run 205f17d3 show foundation.url.resolver.buildMaven() PASSED, so the failure is in the test phase, and main itself is red today for the same reason (see baseline below). VERIFIED at write time: DO had 32 droplets active (over the 25 limit), 22 of them buildtest — the fleet was heavily over-capacity, so netlab droplet provisioning fails for everyone. (I could not read the exact failing test names — the buildtest /api/test-results endpoint was timing out under host load — but the mechanism is the well-established netlab-acquire flake, not a code defect.)

The systemic mechanism (verified across multiple runs)

Each UrlResolver CI run provisions ~8 netlab droplets (one per netlab test; kompile runs them concurrently). bld-build + kotlin.build per PR ≈ 16 concurrent provisions; the DO account limit is 25. When other CI/PRs run at the same time, the account saturates and DigitalOcean rejects droplet creation (provisioningStatus=CREATING_DROPLET → FAILED) → netlab tests fail to acquire. The host ubuntu-s-1vcpu-1gb runs ~20 CN JVMs on 1 vCPU (chronic load avg ~26, spikes to 58 under herd); kotlin.build's builder runs on that host, so under extreme load it can even fail at build-setup in ~3 min with zero tests run. Calm windows (low droplet count, load ~5) do exist and are when CI goes green.

Relevant PRs / refs (write-time state)

| Ref | State @ write time | Notes | |-----|--------------------|-------| | UrlResolver PR #631 | OPEN, BEHIND main, kotlin.build FAILURE / bld-build CANCELLED | The deliverable. Branch netlab-scenario-tests-v2, head 396d11c6. 4 files: NetlabTestEnv.kt + the 3 test .kts. Was BOTH-GREEN once (2026-06-16 09:16Z). Do NOT merge without user approval. | | UrlResolver main | RED today (bld-build failure 17:50Z, kotlin.build failure 17:42Z; an earlier bld-build succeeded 16:28Z) | head 3c127de587c8. Confirms the red is fleet-wide, not PR-specific. | | UrlResolver #628 | MERGED | Bumped protocol → 0.0.303 (inbound mux-substream OOM fix #302/#303). The fix for the CN crash-loop found in step 4. | | Old branch netlab-network-scenario-tests | abandoned | Based on stale b7e00e0a; superseded by netlab-scenario-tests-v2. Ignore. |

Unsaved work: none. All work is on origin/netlab-scenario-tests-v2 (commit 396d11c6). The local /code/UrlResolver checkout was cleaned up between sessions — that is expected; nothing was lost.

Next steps

Re-verify current state first (see banner). Especially: is PR #631 still open? did main go green (clearing the baseline)? what is the current DO droplet count and host load?
Rebase PR #631 onto current main. It is BEHIND (git fetch origin && git rebase origin/main, force-push with --force-with-lease). The only files it touches are NetlabTestEnv.kt + 3 test .kts; if main changed NetlabTestEnv.kt again, re-apply the 4 helper additions (impair setters, apply(prune=), the logLength/awaitResolveResultFrom/pollResolveRepeatedly trio, and the acquireOnce create try/catch) — main may already have upload-resilience and no-shared-worker candidates, so don't duplicate those. Keep the test protocol pins matching main's (foundation.url:protocol version).
Retry CI ONLY in a calm window. Before triggering, check host load + DO droplet count (command below). Wait until droplets are well under 25 (ideally ≤ ~10) and 1-min load is low (~5, not 26+). Then gh pr comment 631 --repo CodexCoder21Organization/UrlResolver --body "/retry" for kotlin.build; bld-build re-triggers on push. Early-UTC windows have historically been calmest (the green run was ~08:29Z; main's earlier green was 00:57Z). Consider the watch-build skill to shepherd it.
When green, ask the user before merging — this is a code PR, not the pre-approved handoff exception.
If CI proves un-greenable across several genuine calm-window retries, the principled root-cause fix (do NOT paper over) is a cross-process provision-concurrency gate in NetlabTestEnv.acquire() — a filesystem-lock semaphore bounding simultaneous droplet acquisitions under the DO limit, which would make all netlab tests CI-reliable, not just these. This was deliberately not done here because main's bld-build has gone green without it, so it is not required for green and would be scope-creep on shared infra. It becomes justified if calm windows stop being reliably reachable. This is a broader fleet-capacity concern (1-vCPU host, 25-droplet account, N concurrent droplet-provisioning tests) that may warrant a separate workstream and user input.

Reusable / operational knowledge

Run a netlab test locally against the managed service: NETLAB_SERVICE_URL=url://netlab-hosted/ scripts/test.bash --test testNetlabSevereImpairmentScenarios. Each provisions a real DO droplet (~15-20 min when infra is healthy; longer with acquire retries). Runs 1 droplet at a time (sequential) so it's a light marginal load.

Check host load + DO droplet count (the gating signal) via HardwareControlFabric — the fabric CLI's --args is naively space-split and can't take dash-leading tokens, so ship a script file:

FABRIC="coursier launch community.kotlin.hardwarecontrolfabric:cli:0.2.6 -r https://kotlin.directory -- -H 198.199.106.165 -c $HOME/.config/hardware-control-fabric/client_cert.pem -k $HOME/.config/hardware-control-fabric/fabric_key.pkcs8 --ca-cert $HOME/.config/hardware-control-fabric/server_cert.pem"
# write a /tmp/load.sh that reads the do-api-key from the netlab-manager process args and curls the DO droplets API, then:
$FABRIC write-file --remote-output-path /tmp/load.sh --local-input-path /tmp/load.sh
$FABRIC launch --cmd /bin/sh --args "/tmp/load.sh" --wait

The DO API key is embedded in the running netlab-manager-server.jar args: ps -eo args | grep -o 'do-api-key [a-z0-9_]*'. launch --wait hard-caps at 60s — for longer jobs run detached (( … ) > out.txt &) and poll the file with read-file. Never pkill -f <pattern> from a Claude Bash shell — its own command line matches and it self-kills; kill by explicit PID.

Read a kotlin.build run's outcome (the WUI is a JS SPA; scrape the JSON APIs instead): build outcome https://buildtest.kotlin.build/api/build-rules?id=<runId> (shows per-rule PASSED/FAILED — an empty result or a fast ~3-min run means it failed at build-setup, i.e. host too loaded to build); test failures https://buildtest.kotlin.build/api/test-results?id=<runId>. Both time out when the host is overloaded — that timeout is itself a load signal.
buildMaven() build gotchas: clear ~/.aibuildcaches/_code_UrlResolver before re-running a test after editing src/ (stale cached artifact otherwise); the produced jar is a directory containing resolver-<ver>.jar (verify hardened code made it in by unzipping and grepping the class).
Related handoffs / memory: the CN OOM + protocol 0.0.303 rollout, the netlab netem impairment feature, and the netlab manager provisioning-concurrency history are all prior context; see the other handoffs/*netlab* and *containernursery* files in this directory.