← Priority list

Handoff: HardwareControlFabric new-daemon onboarding — address-propagation root cause isolated, codex fix attempt is BROKEN (fails CI) and caused a prod outage; prod restored to known-good 0.0.630

Original user request (this multi-session workstream): *"Launch an HCF daemon locally and add it to https://hardware-control-fabric-wui.wasmserver.com; verify basic ops (call a function, modify the filesystem); test daemon registration BOTH via a CLI flag at startup AND by adding an already-running daemon via the WUI; ensure daemon registration and workstream features work; drive to completion using codex."*

hf-2026-07-11-hcf-onboarding-address-propagation-fix-and-prod-incident

Edit handoff

Title Summary Markdown body

# Handoff: HardwareControlFabric new-daemon onboarding — address-propagation root cause isolated, codex fix attempt is BROKEN (fails CI) and caused a prod outage; prod restored to known-good 0.0.630

**Written:** 2026-07-12 (~16:50 UTC). Effort/thread began 2026-07-11.

> ## ⚠️ RE-VERIFY BEFORE ACTING
> **Every state claim below is a write-time snapshot and goes stale.** PRs merge, CI re-runs, branches get force-pushed, and the droplet restarts. Before acting on anything here, re-verify:
> - **PRs:** `gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt`
> - **Prod WUI:** `curl -sS -o /dev/null -w "%{http_code}\n" https://hardware-control-fabric-wui.wasmserver.com/` (expect 200)
> - **Droplet:** `ssh -p 23 -i ~/.ssh/id_ed25519_new root@198.199.106.165 'cut -d" " -f1-3 /proc/loadavg; ss -tlnp | grep :8443'`
> A fresh query always wins over this document.

Related prior handoff: [`2026-07-07-hardware-control-fabric-netlab-e2e-blocked.md`](2026-07-07-hardware-control-fabric-netlab-e2e-blocked.md).

---

## Mission summary

Most of that is **DONE and deployed** (WUI, existing-daemon registration, function invocation, filesystem modification, daemon OOM fix, `-M` launch-command fix, function-creation + CLI). The **one unsolved piece** is **peer-scoped NEW-daemon onboarding** via `--register-with` (the `completeRegistration` ping). A later user directive narrowed the method: *"You need a real reproduction, in a hermetically sealed unit test"* and *"fully verify your claims."*

On 2026-07-12 the user chose the direction **"Fix address propagation"** (the isolated root cause). This handoff covers that attempt, which **failed** (broken fix + a prod outage), and hands off the corrected path.

---

## What was found and done (the chain)

1. **Root cause — PROVEN (live PEER-DIAL-TRACE), verified, not speculation.** When the HCF ServiceServer (peer A) resolves a freshly-registered daemon's peer-scoped alias `url://hardware.control.fabric.daemon.<peerId>/` (peer B), A's UrlResolver candidate set for B contains **ONLY `/p2p-circuit/` relay routes** (every candidate logged `containsDropletDirect=false`), and those relays hold **no circuit-v2 reservation** for B, so each dial throws `RelayException … not reachable` / `io.libp2p.etc.types.NothingToCompleteException`. B's own reachable **direct** address (`/ip4/198.199.106.165/tcp/<p2pPort>/p2p/<peerId>`, open and directly connectable from A on the same host) is **never a candidate**. Net effect: `completeRegistration … 'url://hardware.control.fabric.daemon.<peerId>/' did not respond to ping`. **This is an address-PROPAGATION defect** (B's direct address never propagates into the RESOLVING peer A's registry), **not a dial-ordering defect** — you cannot reorder to a direct address that was never learned. Five prior fixes (resolver 0.0.622/0.0.625/0.0.628/0.0.630 + an on-disk dial diff) were all dial-side and could not close it.

2. **Likely home of the correct fix (hypothesis, worth pursuing):** `UrlProtocol` `main` already has adjacent routing work — `#348 Prefer direct mesh paths before relay fallback`, `#343 link-state relay routing`, `#349 relay ACK routing race`. `#348` "prefer direct" is a no-op if the direct path was never LEARNED — the missing half is propagating B's direct/observed address into A's registry (via identify observed-addr / announce at registration / PEX), and/or making a NATed daemon hold a real circuit-v2 reservation.

3. **codex (gpt-5.5) attempt — BROKEN.** codex wrote a reproducer + an "inbound registration RPC seeds the daemon's direct addresses into the resolver" fix and pushed it to the **same 5 PRs** below. **Its CI FAILS in 3 of 5 repos** (verified via `gh pr checks`): UrlResolver, HCFDaemon, HCFServiceServer all red on `kotlin.build (remote)`; only HCFApi and HCFEmbedded passed. The fix is **broken as implemented, not merely unverified.** codex's hermetic reproducer files (kept — genuinely useful): `UrlResolver/tests/testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts` (white-box: asserts "the ping inside completeRegistration must see at least one non-circuit direct candidate") and `.../testPeerScopedResolutionIgnoresLegacyParentService.kts`.

4. **codex caused a ~40-minute prod outage (verified).** codex ran an **unconstrained live E2E**: deployed its **broken ServiceServer** into ContainerNursery (route `url:hardware-control-fabric:`) + a fix daemon to the droplet, and fired all 5 PRs' CI at once. This spiked the 8-vCPU droplet to **load 119** and took the WUI down (500 "All 0 peers for service 'hardware-control-fabric'"). **Same reckless-prod-deploy pattern as codex run 9.** GUARDRAIL for the next driver: **never let codex deploy to prod or run live E2E on the shared droplet unconstrained.**

5. **Prod RESTORED (verified: WUI 200, 6/6 hits, load falling 119→12).** The WUI "0 peers" 500 was **codex's broken ServiceServer in CN** — restoring the known-good ServiceServer jar fixed it. **Restoring the daemon alone does NOT fix "0 peers"; the ServiceServer jar is the fix.** Also restored the known-good daemon. codex was stopped by PID (verified nothing merged, no merge-queue entries). codex's broken jars are preserved on the droplet for analysis (see below) — diagnostic evidence deliberately not stripped.

6. **Separate CHRONIC bug found (verified from daemon log), independent of codex:** the HCF daemon busy-loops re-pulling a **dead configured bootstrap peer** `12D3KooWQ7YXLXhWLVPdJZ88eKqzDG8k6YgpLZqjsVLvPxrYVVxr` **every 5–11 ms with NO backoff** (`[PEX-EMPTY] re-pull of configured bootstrap … dial-failed`), burning ~1–3 cores. Present in **known-good 0.0.630 too**, and it **re-persists into `/root/.HardwareControlFabric/p2p_peers/` after quarantine** (re-learned, not a stale file). The real fix is exponential backoff in the daemon's PEX re-pull — **do NOT fix by deleting peer files.**

---

## Relevant PRs / refs (write-time state — RE-VERIFY)

All 5 carry codex's broken propagation fix. **None merged. DO NOT MERGE.**

| PR | Branch / head | Write-time state |
|----|---------------|------------------|
| https://github.com/CodexCoder21Organization/UrlResolver/pull/758 | `reproduce-hcf-circuit-only-onboarding` / `83984307` | OPEN, merge=BLOCKED, **CI kotlin.build(remote)=FAIL** |
| https://github.com/CodexCoder21Organization/HardwareControlFabricApi/pull/38 | `bump-urlresolver-0.0.630` / `5aefbeee` | OPEN, merge=CLEAN, CI pass |
| https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/113 | `bump-urlresolver-0.0.630` / `831e2c97` | OPEN, merge=BLOCKED, **CI=FAIL** |
| https://github.com/CodexCoder21Organization/HardwareControlFabricEmbedded/pull/6 | `bump-urlresolver-0.0.630` / `0ab02a0e` | OPEN, merge=CLEAN, CI pass |
| https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer/pull/7 | `bump-urlresolver-0.0.630` / `6c454c5b` | OPEN, merge=UNSTABLE, **CI=FAIL** |

**Deployed prod artifacts (write-time):**
- ServiceServer in CN: **known-good** `hcf-ss-0630.jar` (73176136 bytes, SHA256 `2107aa7fcf9c64d211c3d1e90309e903dbc6e39167130335fb1c7367db5185e4`), route `url:hardware-control-fabric:`.
- Droplet daemon `/root/.HardwareControlFabric/daemon.jar`: **known-good 0.0.630** (60317163 bytes). codex's broken fix jars preserved at `/root/hcf-daemon-codexfix-busyloop-20260712.jar` and `/root/.HardwareControlFabric/daemon.jar.before-address-propagation.20260712151758`.
- Known-good jars for restore live in the originating session scratchpad (ephemeral): `hcf-ss-0630.jar`, `hcf-daemon-0630.jar`, plus `hcf-ss-0628/0625`, `hcf-daemon-0628/0625`. If gone, rebuild from `origin/main` of each repo (see below).

Working clones (ephemeral): `/code/workspace/workspace/reproduce-first-20260711/{UrlProtocol,UrlResolver,HardwareControlFabric*}`. Local UrlResolver working copy is CLEAN and matches pushed head `83984307` — **no unsaved work at risk.**

---

## Next steps (ordered — start by re-verifying)

1. **Re-verify current state first** (PRs, WUI 200, droplet load) using the banner commands. Confirm prod is still on known-good 0.0.630 and the WUI is 200.
2. **Decision the user was asked and owes an answer to (as of write time): (A)** continue the propagation fix under a hard guardrail — codex does code + hermetic tests ONLY, **no prod deploys / no live E2E on the shared droplet**; the human/next-agent reviews the diff and drives any prod touch in a controlled way only after CI is genuinely green — **or (B)** pause: keep prod on known-good, hold/close the 5 PRs, leave this handoff as the record. **Do not proceed to prod work without that steer.**
3. **If (A):** treat codex's current diff as a broken starting point. Get each repo's `kotlin.build (remote)` CI **genuinely green** and the hermetic reproducer (`testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts`) failing-before/passing-after, **entirely off prod**. Diagnose the CI failures first (start from the run logs on the 3 red PRs). The correct fix likely lives in `UrlProtocol` (propagate B's direct/observed address into the resolving peer's registry) — see chain item 2.
4. **Only after CI is green + reproducer passes:** publish bumped versions (never republish a version), bump pins UrlProtocol→UrlResolver→HCF Api/Embedded/Daemon/ServiceServer, then do a **single controlled** live droplet E2E (mint token via WUI `POST /add-peer action=create-token`, launch fresh daemon with `--ip 198.199.106.165`), watching droplet load. Never fire all repos' CI simultaneously (it overloads the shared buildtest droplet).
5. **Separately, fix the chronic daemon PEX no-backoff busy-loop** (chain item 6) at the owning layer — exponential backoff on PEX re-pull of an unreachable configured bootstrap. This is independent of onboarding and worth its own PR.
6. **Never merge any of the 5 code PRs without explicit user approval.**

---

## Reusable / operational knowledge

- **WUI "All 0 peers for service 'hardware-control-fabric'" 500 ⇒ the deployed ServiceServer jar is bad.** Restore: `/home/christian/bin/cs launch containernurserycli:container-nursery-cli:0.0.16 -r https://kotlin.directory -- upload-jar --file <known-good-ss.jar> --route "url:hardware-control-fabric:" --url https://api.nursery.wasmserver.com` (auto-restarts the container), then warm the WUI. Restarting the daemon alone does **not** fix it.
- **coursier on this host is wrong-arch** (`/usr/local/bin/coursier` fails "Exec format error"); use **`/home/christian/bin/cs`**.
- **codex model:** `gpt-5.6-sol` (CLAUDE.md's preference) is **unreachable** — installed codex CLI is 0.143.0 ("model requires a newer version of Codex"), `sudo` does not exist on this host, and the global npm install is root-owned so it can't be upgraded. Fallback that works: **`gpt-5.5`**.
- **Daemon restart (canonical):** kill by filtering `pgrep -x java` + `/proc/$p/cmdline` for `/root/.HardwareControlFabric/daemon.jar` (NEVER `pkill -f daemon.jar` — matches the ssh argv and self-kills), then `setsid nohup java -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/.HardwareControlFabric/logs/ -jar /root/.HardwareControlFabric/daemon.jar </dev/null >/root/hardware-control-fabric-daemon.log 2>&1 &`. The daemon has **no watchdog** — if it dies, the WUI 500s until restarted.
- **Droplet:** 8 vCPU / 16 GB, `ssh -p 23 -i ~/.ssh/id_ed25519_new root@198.199.106.165`. Never `rm` a live/open log; truncate via `/proc/<pid>/fd`.
- **`gh pr edit` is broken** in this org (Projects-classic GraphQL deprecation) — use `gh api -X PATCH repos/<o>/<r>/pulls/<n> -F body=@file`.
- **Never let codex deploy to prod or run an unconstrained live E2E on the shared droplet** — it caused prod outages twice (run 9 and the 2026-07-12 run). Constrain codex to code + hermetic tests; a human/driver controls deploys.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: HardwareControlFabric new-daemon onboarding — address-propagation root cause isolated, codex fix attempt is BROKEN (fails CI) and caused a prod outage; prod restored to known-good 0.0.630

Written: 2026-07-12 (~16:50 UTC). Effort/thread began 2026-07-11.

⚠️ RE-VERIFY BEFORE ACTING

Every state claim below is a write-time snapshot and goes stale. PRs merge, CI re-runs, branches get force-pushed, and the droplet restarts. Before acting on anything here, re-verify:

PRs: gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt

Prod WUI: curl -sS -o /dev/null -w "%{http_code}\n" https://hardware-control-fabric-wui.wasmserver.com/ (expect 200)

Droplet: ssh -p 23 -i ~/.ssh/id_ed25519_new root@198.199.106.165 'cut -d" " -f1-3 /proc/loadavg; ss -tlnp | grep :8443' A fresh query always wins over this document.

Mission summary

Original user request (this multi-session workstream): "Launch an HCF daemon locally and add it to https://hardware-control-fabric-wui.wasmserver.com; verify basic ops (call a function, modify the filesystem); test daemon registration BOTH via a CLI flag at startup AND by adding an already-running daemon via the WUI; ensure daemon registration and workstream features work; drive to completion using codex."

Most of that is DONE and deployed (WUI, existing-daemon registration, function invocation, filesystem modification, daemon OOM fix, -M launch-command fix, function-creation + CLI). The one unsolved piece is peer-scoped NEW-daemon onboarding via --register-with (the completeRegistration ping). A later user directive narrowed the method: "You need a real reproduction, in a hermetically sealed unit test" and "fully verify your claims."

On 2026-07-12 the user chose the direction "Fix address propagation" (the isolated root cause). This handoff covers that attempt, which failed (broken fix + a prod outage), and hands off the corrected path.

What was found and done (the chain)

Root cause — PROVEN (live PEER-DIAL-TRACE), verified, not speculation. When the HCF ServiceServer (peer A) resolves a freshly-registered daemon's peer-scoped alias url://hardware.control.fabric.daemon.<peerId>/ (peer B), A's UrlResolver candidate set for B contains ONLY /p2p-circuit/ relay routes (every candidate logged containsDropletDirect=false), and those relays hold no circuit-v2 reservation for B, so each dial throws RelayException … not reachable / io.libp2p.etc.types.NothingToCompleteException. B's own reachable direct address (/ip4/198.199.106.165/tcp/<p2pPort>/p2p/<peerId>, open and directly connectable from A on the same host) is never a candidate. Net effect: completeRegistration … 'url://hardware.control.fabric.daemon.<peerId>/' did not respond to ping. This is an address-PROPAGATION defect (B's direct address never propagates into the RESOLVING peer A's registry), not a dial-ordering defect — you cannot reorder to a direct address that was never learned. Five prior fixes (resolver 0.0.622/0.0.625/0.0.628/0.0.630 + an on-disk dial diff) were all dial-side and could not close it.
Likely home of the correct fix (hypothesis, worth pursuing): UrlProtocol main already has adjacent routing work — #348 Prefer direct mesh paths before relay fallback, #343 link-state relay routing, #349 relay ACK routing race. #348 "prefer direct" is a no-op if the direct path was never LEARNED — the missing half is propagating B's direct/observed address into A's registry (via identify observed-addr / announce at registration / PEX), and/or making a NATed daemon hold a real circuit-v2 reservation.
codex (gpt-5.5) attempt — BROKEN. codex wrote a reproducer + an "inbound registration RPC seeds the daemon's direct addresses into the resolver" fix and pushed it to the same 5 PRs below. Its CI FAILS in 3 of 5 repos (verified via gh pr checks): UrlResolver, HCFDaemon, HCFServiceServer all red on kotlin.build (remote); only HCFApi and HCFEmbedded passed. The fix is broken as implemented, not merely unverified. codex's hermetic reproducer files (kept — genuinely useful): UrlResolver/tests/testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts (white-box: asserts "the ping inside completeRegistration must see at least one non-circuit direct candidate") and .../testPeerScopedResolutionIgnoresLegacyParentService.kts.
codex caused a ~40-minute prod outage (verified). codex ran an unconstrained live E2E: deployed its broken ServiceServer into ContainerNursery (route url:hardware-control-fabric:) + a fix daemon to the droplet, and fired all 5 PRs' CI at once. This spiked the 8-vCPU droplet to load 119 and took the WUI down (500 "All 0 peers for service 'hardware-control-fabric'"). Same reckless-prod-deploy pattern as codex run 9. GUARDRAIL for the next driver: never let codex deploy to prod or run live E2E on the shared droplet unconstrained.
Prod RESTORED (verified: WUI 200, 6/6 hits, load falling 119→12). The WUI "0 peers" 500 was codex's broken ServiceServer in CN — restoring the known-good ServiceServer jar fixed it. Restoring the daemon alone does NOT fix "0 peers"; the ServiceServer jar is the fix. Also restored the known-good daemon. codex was stopped by PID (verified nothing merged, no merge-queue entries). codex's broken jars are preserved on the droplet for analysis (see below) — diagnostic evidence deliberately not stripped.
Separate CHRONIC bug found (verified from daemon log), independent of codex: the HCF daemon busy-loops re-pulling a dead configured bootstrap peer 12D3KooWQ7YXLXhWLVPdJZ88eKqzDG8k6YgpLZqjsVLvPxrYVVxr every 5–11 ms with NO backoff ([PEX-EMPTY] re-pull of configured bootstrap … dial-failed), burning ~1–3 cores. Present in known-good 0.0.630 too, and it re-persists into /root/.HardwareControlFabric/p2p_peers/ after quarantine (re-learned, not a stale file). The real fix is exponential backoff in the daemon's PEX re-pull — do NOT fix by deleting peer files.

Relevant PRs / refs (write-time state — RE-VERIFY)

All 5 carry codex's broken propagation fix. None merged. DO NOT MERGE.

| PR | Branch / head | Write-time state | |----|---------------|------------------| | https://github.com/CodexCoder21Organization/UrlResolver/pull/758 | reproduce-hcf-circuit-only-onboarding / 83984307 | OPEN, merge=BLOCKED, CI kotlin.build(remote)=FAIL | | https://github.com/CodexCoder21Organization/HardwareControlFabricApi/pull/38 | bump-urlresolver-0.0.630 / 5aefbeee | OPEN, merge=CLEAN, CI pass | | https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/113 | bump-urlresolver-0.0.630 / 831e2c97 | OPEN, merge=BLOCKED, CI=FAIL | | https://github.com/CodexCoder21Organization/HardwareControlFabricEmbedded/pull/6 | bump-urlresolver-0.0.630 / 0ab02a0e | OPEN, merge=CLEAN, CI pass | | https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer/pull/7 | bump-urlresolver-0.0.630 / 6c454c5b | OPEN, merge=UNSTABLE, CI=FAIL |

Deployed prod artifacts (write-time):

ServiceServer in CN: known-good hcf-ss-0630.jar (73176136 bytes, SHA256 2107aa7fcf9c64d211c3d1e90309e903dbc6e39167130335fb1c7367db5185e4), route url:hardware-control-fabric:.
Droplet daemon /root/.HardwareControlFabric/daemon.jar: known-good 0.0.630 (60317163 bytes). codex's broken fix jars preserved at /root/hcf-daemon-codexfix-busyloop-20260712.jar and /root/.HardwareControlFabric/daemon.jar.before-address-propagation.20260712151758.
Known-good jars for restore live in the originating session scratchpad (ephemeral): hcf-ss-0630.jar, hcf-daemon-0630.jar, plus hcf-ss-0628/0625, hcf-daemon-0628/0625. If gone, rebuild from origin/main of each repo (see below).

Working clones (ephemeral): /code/workspace/workspace/reproduce-first-20260711/{UrlProtocol,UrlResolver,HardwareControlFabric*}. Local UrlResolver working copy is CLEAN and matches pushed head 83984307 — no unsaved work at risk.

Next steps (ordered — start by re-verifying)

Re-verify current state first (PRs, WUI 200, droplet load) using the banner commands. Confirm prod is still on known-good 0.0.630 and the WUI is 200.
Decision the user was asked and owes an answer to (as of write time): (A) continue the propagation fix under a hard guardrail — codex does code + hermetic tests ONLY, no prod deploys / no live E2E on the shared droplet; the human/next-agent reviews the diff and drives any prod touch in a controlled way only after CI is genuinely green — or (B) pause: keep prod on known-good, hold/close the 5 PRs, leave this handoff as the record. Do not proceed to prod work without that steer.
If (A): treat codex's current diff as a broken starting point. Get each repo's kotlin.build (remote) CI genuinely green and the hermetic reproducer (testInboundRpcPropagatesDirectAddressesForPeerScopedRegistration.kts) failing-before/passing-after, entirely off prod. Diagnose the CI failures first (start from the run logs on the 3 red PRs). The correct fix likely lives in UrlProtocol (propagate B's direct/observed address into the resolving peer's registry) — see chain item 2.
Only after CI is green + reproducer passes: publish bumped versions (never republish a version), bump pins UrlProtocol→UrlResolver→HCF Api/Embedded/Daemon/ServiceServer, then do a single controlled live droplet E2E (mint token via WUI POST /add-peer action=create-token, launch fresh daemon with --ip 198.199.106.165), watching droplet load. Never fire all repos' CI simultaneously (it overloads the shared buildtest droplet).
Separately, fix the chronic daemon PEX no-backoff busy-loop (chain item 6) at the owning layer — exponential backoff on PEX re-pull of an unreachable configured bootstrap. This is independent of onboarding and worth its own PR.
Never merge any of the 5 code PRs without explicit user approval.

Reusable / operational knowledge

WUI "All 0 peers for service 'hardware-control-fabric'" 500 ⇒ the deployed ServiceServer jar is bad. Restore: /home/christian/bin/cs launch containernurserycli:container-nursery-cli:0.0.16 -r https://kotlin.directory -- upload-jar --file <known-good-ss.jar> --route "url:hardware-control-fabric:" --url https://api.nursery.wasmserver.com (auto-restarts the container), then warm the WUI. Restarting the daemon alone does not fix it.
coursier on this host is wrong-arch (/usr/local/bin/coursier fails "Exec format error"); use /home/christian/bin/cs.
codex model: gpt-5.6-sol (CLAUDE.md's preference) is unreachable — installed codex CLI is 0.143.0 ("model requires a newer version of Codex"), sudo does not exist on this host, and the global npm install is root-owned so it can't be upgraded. Fallback that works: gpt-5.5.
Daemon restart (canonical): kill by filtering pgrep -x java + /proc/$p/cmdline for /root/.HardwareControlFabric/daemon.jar (NEVER pkill -f daemon.jar — matches the ssh argv and self-kills), then setsid nohup java -Xmx128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/.HardwareControlFabric/logs/ -jar /root/.HardwareControlFabric/daemon.jar </dev/null >/root/hardware-control-fabric-daemon.log 2>&1 &. The daemon has no watchdog — if it dies, the WUI 500s until restarted.
Droplet: 8 vCPU / 16 GB, ssh -p 23 -i ~/.ssh/id_ed25519_new root@198.199.106.165. Never rm a live/open log; truncate via /proc/<pid>/fd.
gh pr edit is broken in this org (Projects-classic GraphQL deprecation) — use gh api -X PATCH repos/<o>/<r>/pulls/<n> -F body=@file.
Never let codex deploy to prod or run an unconstrained live E2E on the shared droplet — it caused prod outages twice (run 9 and the 2026-07-12 run). Constrain codex to code + hermetic tests; a human/driver controls deploys.