← Priority list

UrlResolver terminal-status landing: netlab gossip-leak fix chain (PRs 737/866/850) in flight, manager restarted, bld-build verdict pending

Core awaitTerminal() mission complete (3 PRs merged). Remaining: land UrlResolver PR 737 (all failures were netlab-infra flakes; a real flaky test was root-cause-fixed in commit 74a8efc9; merge pre-approved; bld-build verdict pending on head f0e110374), rebase+land PR 866 (the retention fix, head cb37f9e98), roll the fixed UrlResolver into the netlab manager via ContainerNursery (5-step checklist inside; manager currently embeds pre-fix 0.0.670), then re-request checks on PR 850 (head fb88b56bd) and merge, then complete this handoff. The netlab manager was restarted 2026-07-27 ~02:20 UTC for a clean-heap CI window and stayed healthy through a 2h run.

hf-2026-07-17-urlresolver-terminal-status-landing

Edit handoff

Title Summary Markdown body

# UrlResolver Terminal-Status Landing — netlab-leak fix chain in flight

**Written**: 2026-07-27 ~04:30 UTC

**RE-VERIFY BEFORE ACTING**: Everything below is a write-time snapshot. Re-check every PR with `gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt` and the netlab manager process on the host (ContainerNursery route `netlab-hosted` on 198.199.106.165, `ssh -p 23 root@198.199.106.165`) before trusting any state claim. In particular, the `bld-build` check on the first PR was still IN_PROGRESS at write time — its verdict decides the very next action (see Next steps).

## Mission

The original mission (deploy the `awaitTerminal()` hang fix through protected CI) is **complete** — three core PRs merged and deployed previously:

1. https://github.com/CodexCoder21Organization/BuildTestServerService/pull/245 — upload-finalization redesign (merged 2026-07-24, deployed, jar sha256 `b8996948…`)
2. https://github.com/CodexCoder21Organization/UrlResolver/pull/834 — CI budget expansion 90→150 min (merged 2026-07-25)
3. https://github.com/CodexCoder21Organization/UrlResolver/pull/790 — terminal-status fix pinning Observable to 0.3.15 (merged 2026-07-25, commit `b2ddd106b`)

The remaining work is the follow-up chain: land the two UrlResolver fixes for the gossip forwarding-state memory retention problem (which repeatedly restarts the netlab CI manager and makes ~43% of full CI runs fail their netlab tests), roll the fixed build into the netlab manager, then get the coverage PR green and merged, then complete this handoff.

## What was found and done (2026-07-26/27 session)

The session drove everything through codex gpt-5.6-sol subagents. Chain of findings, all evidence-backed:

1. **PR 737 diagnosis.** Both original failing checks on https://github.com/CodexCoder21Organization/UrlResolver/pull/737 (branch `bump-kompile-cli-0.0.79`) were netlab-only failures — non-netlab results were 1200/1206 and 1203/1206 passing, with the signature `Persistent RPC connection to service 'netlab-hosted' was closed while requests were still pending`. No source defect in the PR.

2. **Real flaky test fixed at root.** A retrigger exposed a genuine intermittent failure in `testGossipControlPlaneWorkScalesLinearlyUnderPeerReannouncementChurn`: one-time remote-identity Base58 encoding could land inside the measured window depending on connection scheduling, skewing the linear-scaling assertion. Commit `74a8efc901a4e6babcff58a0f98d9273ed9d21df` adds a second unmeasured real-transport warmup broadcast so the identity cache is populated before measurement (no threshold or product-behavior change). Verified locally twice, and the test then passed in a full `bld-build` run. **Do not lose this commit** — it is in PR 737's branch history.

3. **Three consecutive full runs failed netlab-only**, establishing that blind retriggering cannot land the PR at ~43% failure odds per run.

4. **Deploy-chain investigation** (full report at the "deploychain" section below): the live netlab manager embeds pre-fix UrlResolver **0.0.670** (manager jar sha256 `e8458c8308b59dab3a67e21542bd42de4fcb9e863c6a016787d642b503a3ee9a`), is supervised by ContainerNursery at route `netlab-hosted`, runs with `-Xmx384m` and exit-on-out-of-memory. No worker jars or droplet snapshots need updating — the manager is the only consumer on the critical path.

5. **Netlab manager restarted to create a clean-heap stability window.** At ~02:20 UTC 2026-07-27 the manager was cleanly restarted via ContainerNursery: old PID gone, exactly one new JVM (PID `517616` at the time), route repointed to its live port `35499`, health call passing. Caveat: the ContainerNursery restart CLI returned an ambiguous `Failed: Unknown response format` even though the restart succeeded — recorded as https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-0224-containernursery-cli-reports-an-unknown-response-after.md. Do NOT blindly retry the restart command on that error; verify by PID + route first (retry risks overlapping manager generations — ContainerNursery's reaper has previously leaked duplicate manager JVMs).

6. **CI retriggered inside the window** at commit `f0e110374266af7a1c8bf94396d8c7d9aa19ca52` (empty trigger commit; the flake-fix commit `74a8efc9…` is in its ancestry), ~3 minutes after verified-healthy restart.

7. **Remote check failed for an unrelated reason**: run `a81cbdd1` = 0/1206 tests run, all 10 shards failed with droplet-provisioning stalls (`createDropletAsync stalled for 30000ms`, SSH channel timeouts) — a buildtest droplet-provisioning outage, NOT the netlab manager and NOT the PR. It needs a plain rerun.

8. **At write time** `bld-build` on head `f0e110374` had been executing ~2 hours — host-level ground truth (fresh state writes in `/root/buildtest-data/runs/<id>`, live job step) showed it slow-but-alive, not frozen. The restarted manager stayed healthy throughout: same single PID, no heap dump, JVM heap ~216 MiB of 384 MiB at ~02:50, RSS drifting up 463→678 MB by ~04:08 (the leak visibly re-accumulating, but far from death).

## Current state (verified 2026-07-27 04:27 UTC)

- https://github.com/CodexCoder21Organization/UrlResolver/pull/737 — OPEN, BLOCKED. Head `f0e110374266af7a1c8bf94396d8c7d9aa19ca52`. Checks: `bld-build` IN_PROGRESS (~2 h, verified live), `kotlin.build (remote)` FAILURE (provisioning outage, 0 tests ran). **Merge is pre-approved once green.** Working clone with no unpushed commits: `/code/workspace/UrlResolver-pr737`.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/866 — OPEN, BLOCKED. Head `cb37f9e98db5343ccd04e0f92b0d8ae94597e27d` (branch `fix/gossip-forwarding-state-retention`). This is the retention fix itself (releases superseded peer-forwarding state; includes a retained-heap churn test). A concurrent session hardened its test (last commit "Wait for forwarding reservations in heap churn test", 00:23 UTC) — always fetch the latest head. Stacked on PR 737's old history: after PR 737 squash-merges, rebase by comparing PR 866's intended diff against new main (keep the forwarding-retention change, drop commits that landed via PR 737); do not replay the stacked history blindly.
- https://github.com/CodexCoder21Organization/UrlResolver/pull/850 — OPEN, BLOCKED. Head `fb88b56bd26e89e150f01fc3f0273923de071c7a` unchanged. Adds the concurrency test that fails on broken Observable 0.3.14 and validates the 0.3.15 fix. Only needs re-requested check suites once netlab CI is reliable.
- Netlab manager: freshly restarted generation, healthy at last observation; the clean-heap window decays over hours as the leak re-accumulates.

## Relevant artifacts on the session host (/code/workspace/, ephemeral — key facts duplicated above)

- `pr737-report.md` — full PR 737 diagnosis/fix report
- `deploychain-report.md` — full dependency/rollout chain for the netlab manager
- `netlab-stabilize-report.md` — restart evidence and health verification
- `codex-pr737.log`, `codex-stabilize-737.log`, `codex-deploychain.log` — full agent transcripts

## Next steps (in order)

1. **Check the `bld-build` verdict** on https://github.com/CodexCoder21Organization/UrlResolver/pull/737 (it will have concluded by pickup). If green: rerun the remote check (push an empty commit — this restarts BOTH checks, so only do it after bld-build concludes; the remote check-run has no reliable re-run button). If bld-build failed netlab-only again despite the fresh manager window, capture the manager's state at failure (PID continuity, heap dumps, `jstat -gc`) — that evidence justifies deploying an interim fixed-manager build BEFORE merge (build UrlResolver from the PR branch, pin into NetLabManagerServer, fat jar, ContainerNursery `upload-jar` to route `netlab-hosted`).
2. **If/when both checks are green, merge PR 737** via the merge queue (GraphQL `enqueuePullRequest`; `gh pr merge --auto` is disabled). Merge pre-approved. Consider restarting the netlab manager again immediately before the retrigger if hours have passed (fresh window).
3. **Rebase and land https://github.com/CodexCoder21Organization/UrlResolver/pull/866** per the rebase note above; drive to green (same netlab-window tactics) and merge.
4. **Roll the fix into the netlab manager** (from `deploychain-report.md`): publish a fresh UrlResolver release from main (expected 0.0.678 — do NOT reuse 0.0.677, it predates the retention fix); merge a NetLabManagerServer PR pinning that release plus paired protocol/SJVM versions; build the manager fat jar from merged main; deploy via ContainerNursery `upload-jar` to route `netlab-hosted` (restarts the route); verify the embedded version on-host.
5. **Re-request check suites on https://github.com/CodexCoder21Organization/UrlResolver/pull/850** (head unchanged `fb88b56bd`), merge on success.
6. **Complete this handoff** via handoff-cli once all of the above is verified done.

## Reusable/operational knowledge

- Netlab-only CI failures with `Persistent RPC connection to service 'netlab-hosted' was closed` = the manager memory problem, not the PR under test. Non-netlab failures deserve real diagnosis.
- A remote-check failure with `0 passed / N total` and `createDropletAsync stalled` = droplet-provisioning outage; rerun, don't debug the PR.
- The clean-heap window tactic: restart the manager, verify one PID + live route + health, then trigger CI within minutes. Window comfortably covers one ~52-min run; RSS drift ~200 MB over 2 h under CI load.
- kompile-cli 0.0.84 prints ordinary successful build progress as `[ERROR] Unhandled effect` stack traces (recorded challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-26-2323-kompile-cli-0-0-84-labels-normal-successful-build-effects.md) — a "failing" local build may have succeeded; check the exit code.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

UrlResolver Terminal-Status Landing — netlab-leak fix chain in flight

Written: 2026-07-27 ~04:30 UTC

RE-VERIFY BEFORE ACTING: Everything below is a write-time snapshot. Re-check every PR with gh pr view <n> --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt and the netlab manager process on the host (ContainerNursery route netlab-hosted on 198.199.106.165, ssh -p 23 root@198.199.106.165) before trusting any state claim. In particular, the bld-build check on the first PR was still IN_PROGRESS at write time — its verdict decides the very next action (see Next steps).

Mission

The original mission (deploy the awaitTerminal() hang fix through protected CI) is complete — three core PRs merged and deployed previously:

https://github.com/CodexCoder21Organization/BuildTestServerService/pull/245 — upload-finalization redesign (merged 2026-07-24, deployed, jar sha256 b8996948…)
https://github.com/CodexCoder21Organization/UrlResolver/pull/834 — CI budget expansion 90→150 min (merged 2026-07-25)
https://github.com/CodexCoder21Organization/UrlResolver/pull/790 — terminal-status fix pinning Observable to 0.3.15 (merged 2026-07-25, commit b2ddd106b)

What was found and done (2026-07-26/27 session)

The session drove everything through codex gpt-5.6-sol subagents. Chain of findings, all evidence-backed:

PR 737 diagnosis. Both original failing checks on https://github.com/CodexCoder21Organization/UrlResolver/pull/737 (branch bump-kompile-cli-0.0.79) were netlab-only failures — non-netlab results were 1200/1206 and 1203/1206 passing, with the signature Persistent RPC connection to service 'netlab-hosted' was closed while requests were still pending. No source defect in the PR.
Real flaky test fixed at root. A retrigger exposed a genuine intermittent failure in testGossipControlPlaneWorkScalesLinearlyUnderPeerReannouncementChurn: one-time remote-identity Base58 encoding could land inside the measured window depending on connection scheduling, skewing the linear-scaling assertion. Commit 74a8efc901a4e6babcff58a0f98d9273ed9d21df adds a second unmeasured real-transport warmup broadcast so the identity cache is populated before measurement (no threshold or product-behavior change). Verified locally twice, and the test then passed in a full bld-build run. Do not lose this commit — it is in PR 737's branch history.
Three consecutive full runs failed netlab-only, establishing that blind retriggering cannot land the PR at ~43% failure odds per run.
Deploy-chain investigation (full report at the "deploychain" section below): the live netlab manager embeds pre-fix UrlResolver 0.0.670 (manager jar sha256 e8458c8308b59dab3a67e21542bd42de4fcb9e863c6a016787d642b503a3ee9a), is supervised by ContainerNursery at route netlab-hosted, runs with -Xmx384m and exit-on-out-of-memory. No worker jars or droplet snapshots need updating — the manager is the only consumer on the critical path.
Netlab manager restarted to create a clean-heap stability window. At ~02:20 UTC 2026-07-27 the manager was cleanly restarted via ContainerNursery: old PID gone, exactly one new JVM (PID 517616 at the time), route repointed to its live port 35499, health call passing. Caveat: the ContainerNursery restart CLI returned an ambiguous Failed: Unknown response format even though the restart succeeded — recorded as https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-0224-containernursery-cli-reports-an-unknown-response-after.md. Do NOT blindly retry the restart command on that error; verify by PID + route first (retry risks overlapping manager generations — ContainerNursery's reaper has previously leaked duplicate manager JVMs).
CI retriggered inside the window at commit f0e110374266af7a1c8bf94396d8c7d9aa19ca52 (empty trigger commit; the flake-fix commit 74a8efc9… is in its ancestry), ~3 minutes after verified-healthy restart.
Remote check failed for an unrelated reason: run a81cbdd1 = 0/1206 tests run, all 10 shards failed with droplet-provisioning stalls (createDropletAsync stalled for 30000ms, SSH channel timeouts) — a buildtest droplet-provisioning outage, NOT the netlab manager and NOT the PR. It needs a plain rerun.
At write time bld-build on head f0e110374 had been executing ~2 hours — host-level ground truth (fresh state writes in /root/buildtest-data/runs/<id>, live job step) showed it slow-but-alive, not frozen. The restarted manager stayed healthy throughout: same single PID, no heap dump, JVM heap ~216 MiB of 384 MiB at ~02:50, RSS drifting up 463→678 MB by ~04:08 (the leak visibly re-accumulating, but far from death).

Current state (verified 2026-07-27 04:27 UTC)

https://github.com/CodexCoder21Organization/UrlResolver/pull/737 — OPEN, BLOCKED. Head f0e110374266af7a1c8bf94396d8c7d9aa19ca52. Checks: bld-build IN_PROGRESS (~2 h, verified live), kotlin.build (remote) FAILURE (provisioning outage, 0 tests ran). Merge is pre-approved once green. Working clone with no unpushed commits: /code/workspace/UrlResolver-pr737.
https://github.com/CodexCoder21Organization/UrlResolver/pull/866 — OPEN, BLOCKED. Head cb37f9e98db5343ccd04e0f92b0d8ae94597e27d (branch fix/gossip-forwarding-state-retention). This is the retention fix itself (releases superseded peer-forwarding state; includes a retained-heap churn test). A concurrent session hardened its test (last commit "Wait for forwarding reservations in heap churn test", 00:23 UTC) — always fetch the latest head. Stacked on PR 737's old history: after PR 737 squash-merges, rebase by comparing PR 866's intended diff against new main (keep the forwarding-retention change, drop commits that landed via PR 737); do not replay the stacked history blindly.
https://github.com/CodexCoder21Organization/UrlResolver/pull/850 — OPEN, BLOCKED. Head fb88b56bd26e89e150f01fc3f0273923de071c7a unchanged. Adds the concurrency test that fails on broken Observable 0.3.14 and validates the 0.3.15 fix. Only needs re-requested check suites once netlab CI is reliable.
Netlab manager: freshly restarted generation, healthy at last observation; the clean-heap window decays over hours as the leak re-accumulates.

Relevant artifacts on the session host (/code/workspace/, ephemeral — key facts duplicated above)

pr737-report.md — full PR 737 diagnosis/fix report
deploychain-report.md — full dependency/rollout chain for the netlab manager
netlab-stabilize-report.md — restart evidence and health verification
codex-pr737.log, codex-stabilize-737.log, codex-deploychain.log — full agent transcripts

Next steps (in order)

Check the bld-build verdict on https://github.com/CodexCoder21Organization/UrlResolver/pull/737 (it will have concluded by pickup). If green: rerun the remote check (push an empty commit — this restarts BOTH checks, so only do it after bld-build concludes; the remote check-run has no reliable re-run button). If bld-build failed netlab-only again despite the fresh manager window, capture the manager's state at failure (PID continuity, heap dumps, jstat -gc) — that evidence justifies deploying an interim fixed-manager build BEFORE merge (build UrlResolver from the PR branch, pin into NetLabManagerServer, fat jar, ContainerNursery upload-jar to route netlab-hosted).
If/when both checks are green, merge PR 737 via the merge queue (GraphQL enqueuePullRequest; gh pr merge --auto is disabled). Merge pre-approved. Consider restarting the netlab manager again immediately before the retrigger if hours have passed (fresh window).
Rebase and land https://github.com/CodexCoder21Organization/UrlResolver/pull/866 per the rebase note above; drive to green (same netlab-window tactics) and merge.
Roll the fix into the netlab manager (from deploychain-report.md): publish a fresh UrlResolver release from main (expected 0.0.678 — do NOT reuse 0.0.677, it predates the retention fix); merge a NetLabManagerServer PR pinning that release plus paired protocol/SJVM versions; build the manager fat jar from merged main; deploy via ContainerNursery upload-jar to route netlab-hosted (restarts the route); verify the embedded version on-host.
Re-request check suites on https://github.com/CodexCoder21Organization/UrlResolver/pull/850 (head unchanged fb88b56bd), merge on success.
Complete this handoff via handoff-cli once all of the above is verified done.

Reusable/operational knowledge

Netlab-only CI failures with Persistent RPC connection to service 'netlab-hosted' was closed = the manager memory problem, not the PR under test. Non-netlab failures deserve real diagnosis.
A remote-check failure with 0 passed / N total and createDropletAsync stalled = droplet-provisioning outage; rerun, don't debug the PR.
The clean-heap window tactic: restart the manager, verify one PID + live route + health, then trigger CI within minutes. Window comfortably covers one ~52-min run; RSS drift ~200 MB over 2 h under CI load.
kompile-cli 0.0.84 prints ordinary successful build progress as [ERROR] Unhandled effect stack traces (recorded challenge: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-26-2323-kompile-cli-0-0-84-labels-normal-successful-build-effects.md) — a "failing" local build may have succeeded; check the exit code.