Handoff: UrlResolver merge campaign — one PR left (822, two 826-interaction test failures) plus the deferred buildtest/fabric/W3Wallet backlog; paused on gpt-5.6-sol quota

Written: 2026-07-24 ~12:25 UTC. RE-VERIFY: Everything below is a write-time snapshot. PRs merge, get force-pushed, and re-run CI constantly in this org; re-check every PR with gh pr view <n> --repo CodexCoder21Organization/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt and every buildtest run against host ground truth (ssh -p 23 -i ~/.ssh/cn_host_diag root@198.199.106.165, run dirs under /root/buildtest-data/runs/<id>/) before acting. A fresh query always wins over this document.

Mission summary

A three-day campaign (2026-07-21 → 07-24) to merge a user-specified train of production PRs across seven repos, fixing every infrastructure root cause in the way. The campaign is ~95% complete: all headline deliverables are merged and deployed (details below). What remains is (1) one final UrlResolver PR — https://github.com/CodexCoder21Organization/UrlResolver/pull/822 — failing exactly two tests due to an interaction with recently-merged main, (2) two supporting UrlResolver PRs (838, 834), and (3) a deferred backlog of buildtest-scheduler, HardwareControlFabric-consumer, and W3Wallet/LambdaServer PRs inherited from earlier handoffs. Work is paused because the operator's delegation target (codex gpt-5.6-sol) hit its usage limit until 2026-07-28 17:02 UTC — per the operator's global CLAUDE.md, substantial implementation work is delegated to that model, so the next session should either wait for quota, or get explicit authorization to proceed hands-on.

What was found and done (compressed chain — the "why" behind current state)

Full detail lives in the status issue https://github.com/CodexCoder21Organization/PlanRepository/issues/2752 and the PlanRepository challenges from 2026-07-21..24. Summary:

1. Merge queue wedge root-caused and fixed. buildtest's coordinator adjudicated run-lease death against a liveness snapshot captured before blocking provider calls, mass-failing healthy runs ("false lease-death"). Fixed in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/359 (reproducer-first), deployed to production. https://github.com/CodexCoder21Organization/UrlResolver/pull/819 then merged, ending a ~38 h wedge.
2. /bypass fixed + deployed (https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/193, live as kotlin-build-ci 0.0.79): GitHub reports private-org owners as CONTRIBUTOR in comment payloads, so authorization now queries the collaborator-permission API (accepts write/maintain/admin, fails closed).
3. Six more production root causes fixed en route (each with its own PR/challenge): runner_heartbeat events crashing coordinator ingestion (https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/364); merge-queue check_response_timeout_minutes 60→180 org-wide (silent checks_timed_out evictions of healthy queued runs); a dead-peer redial storm from contaminated P2P transport pins (emergency deploy of resolver 0.0.653/protocol 0.0.397 into ContainerNursery and buildtest-server); CN hardcoded -XX:MaxMetaspaceSize=128m OOM crash-loop; GitHub-Actions bld-build 90-min job timeout killing passing runs (raise to 150 in https://github.com/CodexCoder21Organization/UrlResolver/pull/834, still unmerged, see below); a coordinator run.json persistence failure under disk pressure (transient, diagnosed).
4. The original train merged: UrlResolver https://github.com/CodexCoder21Organization/UrlResolver/pull/819, https://github.com/CodexCoder21Organization/UrlResolver/pull/801, https://github.com/CodexCoder21Organization/UrlResolver/pull/821, https://github.com/CodexCoder21Organization/UrlResolver/pull/680; plus PerformanceTestWui 15, BuildTestEmbedded 357, BuildTestWui 136, kompile-core 234, PerformanceTestEmbedded 23 (all merged 07-21).
5. The withdrawal/relay flake family was root-caused and fixed in https://github.com/CodexCoder21Organization/UrlResolver/pull/826 (merged 07-23, 1136/1136 queue run): close-before-unregister-commit loss + snapshots resurrecting withdrawn services. This de-flaked the gate for everything after it.
6. Observables ceremony complete: https://github.com/CodexCoder21Organization/UrlResolver/pull/680 (Phase C resilience fixture) merged 07-24 10:45 UTC after two real harness fixes (readiness anchored at child-liveness signal, not JVM spawn; poll-fallback restart gated on an observed stale-window read). Epic https://github.com/CodexCoder21Organization/Observable/issues/87 checkboxes synced with evidence comments; handoff hf-2026-06-22-observables-command-idempotency-and-resilience completed/archived in the Handoff service; the false-lease-death handoff likewise closed (https://github.com/CodexCoder21Organization/PlanRepository/pull/2961, https://github.com/CodexCoder21Organization/PlanRepository/pull/2962 merged).

Current state (verified 2026-07-24 12:22 UTC)

The one remaining campaign PR

• https://github.com/CodexCoder21Organization/UrlResolver/pull/822 — branch fix/timeout-units-seconds, head b89331aa. kotlin.build (remote) SUCCESS; bld-build FAILURE at 1135/1137 on exactly two tests that main gained from PR 826:
  • testPeerAnnouncementAntiEntropyReplaysAfterInitialPathWasUnavailable — "The unverified provider must not leak through the verified late-join snapshot path" (fails in 4.0 s, deterministic-looking)
  • stressTestFreshJoinWithdrawalDeliveryUnderConcurrentReannounce — exit code 1
  • The branch's residual commits: (a) circuit-breaker @Timeout(90) annotation, (b) whole-node-close withdrawal causal test setup, (c) relay-registration single-flight guard (801162c3), (d) deadline-test lost-completion fix (b89331aa — a test-side race where the request completed before provider dispatch and the test waited forever on a handler-entry latch; now a first-of-either future). Suspect for the two failures is (c): main's 826 serialized registration and gated snapshot authority at the owning layer, so the resolver-side single-flight may suppress a registration pass the new anti-entropy replay/withdrawal-delivery paths rely on, or double-guarding changes candidate-budget semantics.
  • Next step (the exact task in flight when quota ran out): rebase onto origin/main; run the two failing tests locally; if (c) is redundant/conflicting, drop it and verify the two tests pass PLUS stressTestHealthySameTierRelayWinsUnderContention (the test (c) originally fixed — if that fails without (c), fix the interaction properly instead of dropping); keep (a),(b),(d); push --force-with-lease; enqueue via GraphQL enqueuePullRequest; shepherd with build-watchman --to-merged.
  • Note: 822's original purpose (millisecond-style @Timeout values) was already delivered to production by the merged https://github.com/CodexCoder21Organization/UrlResolver/pull/788 — what's left is genuinely-valuable flake-fix residue, not a production emergency.
  • Local workspace /code/ws/pr822fix (clone, detached at b89331aa = remote head; nothing unpushed). Branch backup/pr822-pre-rescope-8b0df191 there is an intentionally-abandoned backup of the pre-07-23 over-broad gossip-fanout-gating version (it broke 9 tests and was rescoped away) — historical reference only, do not resurrect.

Supporting UrlResolver PRs

• https://github.com/CodexCoder21Organization/UrlResolver/pull/838 — real root-cause fix for the org-wide testGossipStormToStalledPeersStillDeliversToHealthyPeer flake (synchronous relay send waited up to 200 ms backpressure and could reconnect inline; 48×200 ms explains CI's 9.6–20 s blocking; fix = zero-wait/non-reconnecting relay admission with misses handled by the existing background path). Both checks currently FAILURE — undiagnosed; diagnose before re-running (could be the fix, could be infra).
• https://github.com/CodexCoder21Organization/UrlResolver/pull/834 — workflow-only diff raising bld-build timeout-minutes 90→150 (queue bld-builds were being killed while passing). bld-build SUCCESS, kotlin.build FAILURE — a workflow-only diff cannot cause suite failures, so its kotlin.build failures have all been infra draws (verified twice: provisioning-killed-by-restart, netlab starvation). Rerequest when the executor is calm, then merge; until it lands, queue bld-build runs die at 90 min under load.

Deferred backlog (from earlier handoffs; states verified 12:22 UTC)

• Keystone/buildtest chain (H3): https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/363 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/358 both have kotlin.build SUCCESS (enqueueable); https://github.com/CodexCoder21Organization/BuildTestServerService/pull/238 and https://github.com/CodexCoder21Organization/BuildTestServerService/pull/244 both FAILURE (diagnose first). After landing: rebuild+redeploy the buildtest backend from merged main (deploy discipline below) and run orphaned-run recovery.
• Fabric consumers (H5): https://github.com/CodexCoder21Organization/HardwareControlFabricApi/pull/40, https://github.com/CodexCoder21Organization/HardwareControlFabricEmbedded/pull/7, https://github.com/CodexCoder21Organization/HardwareControlFabricServiceServer/pull/9 are all CLEAN with green checks — ready to enqueue right now; https://github.com/CodexCoder21Organization/HardwareControlFabricDaemon/pull/119 is BLOCKED/FAILURE (diagnose; note the fabric daemon in production reportedly still runs contaminated transport 0.0.395 — landing+deploying this chain retires that risk).
• Misc: https://github.com/CodexCoder21Organization/GithubProxyServerService/pull/26 FAILURE; https://github.com/CodexCoder21Organization/BuildTestWui/pull/134 has green kotlin.build but is DIRTY (merge conflict — rebase needed). W3Wallet/LambdaServer chain (H4) untouched by this session; see its own handoff in the service.

Production health (verified 12:10–12:22 UTC)

https://buildtest.kotlin.build/ answers HTTP 200 in <1 s. UrlResolver main tip = PR 680's merge. Host 198.199.106.165: CN stable under the cron watchdog /root/cn-watchdog.sh (relaunches CN within ~1 min; audit log /root/cn-watchdog.audit), disk ~92% (chronic; check df before uploads). Known live degradation: the droplet service (url://digitalocean-droplets/, /root/ContainerNursery/apps/droplet-service-server.jar) times out listDroplets over SJVM when the fleet is large (~100+ droplets — response conversion outruns clients' 30 s timeout; also hits the WebCron listJobs "cumulative value limit"). This is what starves netlab environment provisioning during busy periods and cost ~10 merge-queue evictions on 07-23/24. It self-heals as the fleet shrinks; the durable fix is pagination/limiting in DigitalOceanDropletServiceServer — nobody owns that yet.

Blocker

codex gpt-5.6-sol usage limit reached; resets 2026-07-28 17:02 UTC (error text: "You've hit your usage limit… try again at Jul 28th, 2026 5:02 PM"). The operator's global CLAUDE.md mandates delegating implementation-heavy work to that model. Options for the next session: wait for reset; purchase credits; or get the user's explicit go-ahead to do the work hands-on.

Next steps (in order)

1. When able to execute: finish 822 per the exact brief above (drop-or-fix the single-flight commit), land it.
2. Diagnose + land https://github.com/CodexCoder21Organization/UrlResolver/pull/834 (removes the queue bld-build 90-min killer) and https://github.com/CodexCoder21Organization/UrlResolver/pull/838 (removes the gossip-storm flake).
3. Enqueue the three green fabric-consumer PRs (Api 40 / Embedded 7 / ServiceServer 9); diagnose Daemon 119; deploy the fabric daemon off contaminated 0.0.395.
4. Enqueue BuildTestEmbedded 363 + 358; diagnose BuildTestServerService 238/244; then backend redeploy + orphaned-run recovery per the keystone handoff.
5. Rebase BuildTestWui 134; diagnose GithubProxyServerService 26; pick up the W3Wallet/Lambda chain handoff.
6. Durable fixes worth opening: droplet-service listDroplets pagination; netlab-manager resilience to slow listDroplets.

Reusable/operational knowledge

• Merge queues here reject gh pr merge — enqueue with GraphQL: gh api graphql -f query='mutation($id:ID!){enqueuePullRequest(input:{pullRequestId:$id}){mergeQueueEntry{position state}}}' -f id=$(gh pr view <n> -R <o/r> --json id --jq .id). Shepherd with build-watchman 0.0.13 --to-merged (coursier launch buildwatchman:build-watchman:0.0.13 -r https://kotlin.directory -- --repo <o/r> --pr <n> --to-merged); one watchman per PR, tracked in background, re-arm after every rerun.
• Never re-run a failed queue draw blind — diagnose from host ground truth first (/root/buildtest-data/runs/<id>/run.json + test-events.jsonl). Infra classes seen: netlab starvation (testNetlab* env-acquisition), createBuildDroplet provisioning, all-shards "Shared workspace module preparation failed", run.json persistence failure, coordinator-restart kills. A repeat failure of the same real test means stop and fix.
• /usr/local/bin/coursier on this box is wrong-arch — use the JVM wrapper at /tmp/claude-1000/-code/792c6c44-a997-453a-b7c5-da8096dfac44/scratchpad/coursier.jar (java -jar coursier.jar launch …) or fetch a fresh one.
• handoff-cli fatjar built at /code/ws/handoffcli/build-fats/handoff-cli.jar.
• Deploy discipline for buildtest-server: any local build/deploy MUST include the still-unmerged PRs' fixes or production regresses (this bit once already); prefer building from merged main after landing the chain.
• codex invocation gotchas: always < /dev/null (else it hangs on "Reading additional input from stdin"), 90-min timeout kills long CI-watch lanes — keep watching in the orchestrator, have codex end at "push and report".
• Session memories with the durable versions of all of the above live in the operator's memory directory (buildtest-false-lease-death-fixed, urlprotocol-large-frame-drop-outages, cn-host-disk-chronically-full, etc.).