Handoff: resolver 0.0.600 lane A blocked at NetLabWorkerServer by full CI host disk

Written 2026-07-11 UTC for work begun 2026-07-10.

RE-VERIFY: This is a write-time snapshot. Before acting, run gh pr view 60 --repo CodexCoder21Organization/NetLabWorkerServer --json state,mergeStateStatus,statusCheckRollup,mergedAt, inspect /home/helena/workspace/campaign-lane-a/progress.txt, and verify host disk with ssh -p 23 root@198.199.106.165 'df -h /; stat -c "%s %y %n" /root/ContainerNursery/container-nursery.log.1'. Fresh state wins.

Mission summary

Execute /home/helena/workspace/campaign-lane-a/lane.md under GATE AMENDMENT v2 and /home/helena/workspace/campaign-common.md, skipping supervisor-owned items 1–2 and processing item 3 onward strictly one repository/CI run at a time. Bump only foundation.url:resolver in each build.kts to 0.0.600, merge green campaign PRs, and use build-watchman 0.0.11 only.

What was found and done

1. Reconciled existing clones and remote PRs. No campaign PR existed for items 3–16. NetLabWorkerServer had the prior attempt's exact uncommitted one-line change; it was preserved, rebased onto origin/main, committed, pushed, and opened as https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/60.
2. The PR head is 11910358678250907682e01432bbe454aa8b38b7 on campaign/resolver-0.0.600. The diff changes only resolver 0.0.599 to 0.0.600. The GitHub bld-build check passed.
3. Local ./scripts/test.bash --test . could not complete because of pre-existing cross-test pin mismatch warnings and repeated Coursier workspace-artifact resolution timeouts. No test files were changed.
4. build-watchman 0.0.11 shepherded multiple remote attempts. Initial runs c83e0d06 and 1cf139c5 were evicted after acquiring 0 droplets (lease watchdog). Required 10-minute backoffs were observed.
5. Run bd194212 reached 27/29 but failed two real-droplet tests because the DigitalOcean gateway returned 404 for droplets missing from its full-list snapshot. This exact problem was already recorded in PlanRepository.
6. Run a9efef1e failed three infrastructure-dependent tests. Cross-process forensics proved ContainerNursery PID 1068832 OOMed at 01:13 UTC and restarted at 01:14, killing url://digitalocean-droplets/ and url://webcron/ routes during the tests. Eclipse MAT analysis of preserved heap /root/ContainerNursery/heapdumps/preserved/gen-20260711-011404Z/java_pid1068832.hprof found aggregate byte-array/coroutine/thread/EventStore/libp2p retention rather than one leak suspect. Incident: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-11-0124-containernursery-oom-restart-killed-dependency-routes.md.
7. After dependency recovery and actual successful fresh droplet timer creation, run a6e6f00b failed before tests with No space left on device.
8. Verified root filesystem /dev/vda1 is 49G/49G, 100%, 0 available (inodes only 5%). Dominant file is /root/ContainerNursery/container-nursery.log.1, logical size 17,912,543,338 bytes. /root/buildtest-data is only 2.0G. Incident: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-11-0237-containernursery-17-9-gb-rotated-log-filled-build-host.md.
9. No production file was deleted, truncated, compressed, or otherwise mutated: the campaign authorizes source-pin changes only. HardwareControlFabric credentials were absent in this runner, forcing documented read-only SSH fallback; see https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-10-2314-hardwarecontrolfabric-skill-credentials-absent-in-campaign.md.

Relevant PRs / refs

| Resource | Write-time state | |---|---| | https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/60 | OPEN, BLOCKED; bld-build success, remote check failure from ENOSPC | | campaign/resolver-0.0.600 | pushed; head 11910358678250907682e01432bbe454aa8b38b7 | | lane progress file | no completed item-3 line yet; append only after merge or campaign-authorized skip |

There is no unsaved NetLab source work. The checkout is clean and tracks origin/campaign/resolver-0.0.600.

Next steps

1. Re-verify current state first using the commands in the banner.
2. Obtain explicit operational authority to preserve/compress/truncate or rotate /root/ContainerNursery/container-nursery.log.1. Do not destroy the OOM evidence blindly. Free enough disk for a build and confirm df -h / has durable headroom.
3. Fix bounded ContainerNursery log rotation at the owning layer; do not treat manual cleanup or the current 1 GB ContainerNursery heap as the durable fix.
4. Probe https://kotlin.directory/ for 200 in under one second and confirm droplet/webcron processes and ports are stable. Re-request only check suite 78822117457, then launch exactly coursier launch buildwatchman:build-watchman:0.0.11 -r https://kotlin.directory -- --repo CodexCoder21Organization/NetLabWorkerServer --pr 60 --to-merged.
5. If green and merged, append NetLabWorkerServer https://github.com/CodexCoder21Organization/NetLabWorkerServer/pull/60 MERGED to /home/helena/workspace/campaign-lane-a/progress.txt.
6. Continue lane items 4–16 in order, one repository and one CI run at a time. Item 4 is ContainerNursery. Do not touch supervisor-owned items 1–2, UrlProtocol PR 351, or UrlResolver PR 737.

Reusable / operational knowledge

• Use build-watchman 0.0.11, never 0.0.9.
• The gate amendment forbids DO account-limit arithmetic. Classify capacity from actual run/service failure text.
• Host ground truth for a build is /root/buildtest-data/runs/<runId>/build.log plus test-events.jsonl; do not delete a run unless both are frozen.
• Current Fabric mTLS credentials are absent, so SSH was the last-resort read-only path. Do not expand that fallback into production mutations without authority.
• Exact failed run URLs: https://buildtest.kotlin.build/run?id=c83e0d06, https://buildtest.kotlin.build/run?id=1cf139c5, https://buildtest.kotlin.build/run?id=bd194212, https://buildtest.kotlin.build/run?id=a9efef1e, and https://buildtest.kotlin.build/run?id=a6e6f00b.