Handoff: GithubWatchman → lambdaserver callback failures + the systemic awaited-write DELIVERY fix + deploy chain

Thread started: 2026-06-18. Handoff written: 2026-07-05 by Claude (Opus 4.8, 1M ctx).

⚠️ Re-verify everything before acting. This thread paused ~13 days; main and CI have moved on both UrlProtocol and UrlResolver (e.g. resolver main is now on protocol 0.0.332 / resolver 0.0.589, well past the versions below). Parallel efforts landed overlapping fixes (see the sibling 2026-07-05 handoffs on netlab/#650/#656/#669). Treat gh pr view <n> --json state,mergeStateStatus,statusCheckRollup and the live code as authoritative, not this doc.

---

Mission (original request)

Investigate and fix, end-to-end and in production, the GithubWatchman → url://lambdaserver/functions/{id} callback failures (first observed as "Stream closed", later NoSuchMethodError: lambdaserver/LambdaServiceClientImpl.invoke(List)Object). Ensure the fixes merge, publish, and are deployed so the callbacks actually succeed. Mid-thread the user added the higher-priority directive:

"Ultrathink — are the tests comprehensive? Add end-to-end integration tests which identify bugs (the tests should initially fail), then fix the bug in the implementation."

…and a standing rule (verbatim): "The server is not overloaded, the server is NEVER overloaded. An overloaded server would never result in a 30-second block — something more systemic is happening there, which needs to be resolved."

The final in-thread instruction was to run the deploy chain: (1) merge UrlProtocol #319 + publish protocol 0.0.314; (2) re-pin resolver #641 onto 0.0.314, merge + publish resolver; (3) redeploy ContainerNursery + github-watchman onto the new protocol/resolver. Then, when netlab e2e tests blocked step 2, the user asked me to investigate the prod netlab outage before gating any tests — that investigation is complete (below).

---

The 3-layer callback bug (root-cause diagnosis)

The watchman → url://lambdaserver/functions/{id} callback traverses ContainerNursery's UrlFacade proxy → the resolver's bytecode fetch → the lambda-server provider. Three independent defects, each on a different layer:

1. Transport "Stream closed" = the deployed lambda-server.jar was a stale pre-url-bind Jetty build (answered HTTP, not url-bind RPC). Fix = deploy the correct url-bind build. See LambdaServer branch below (fix-fatjar-stale-lambdaserver-shadow — the fat-jar was shadowing the url-bind classes with a transitive lambda-server:0.0.1).
2. NoSuchMethodError — proxy dropped request.metadata = ServiceRoutingHandler.handleBytecodeRequest proxy fallback forwarded emptyMap() metadata. FIXED + MERGED: UrlProtocol #316 (protocol 0.0.309).
3. NoSuchMethodError — resolver never attached resourcePath = requestBytecodeConditionallyFromPeer threaded resourcePath in as a param but built the RpcRequest with metadata = emptyMap(); the relay path omitted it too. So the provider always got empty metadata → served the generic LambdaServiceClientImpl instead of the per-resource FunctionDefinitionClientImpl → invoke(List)Object NoSuchMethodError in the sandbox. FIX WRITTEN but NOT LANDED — see UrlResolver #641 (CLOSED, unmerged). ⚠️ As of 2026-07-05, resolver main does NOT contain this fix (grep 'mapOf("resourcePath" to' in UrlResolver.kt = 0 hits). This layer is still open.

---

The systemic root cause (the big win): awaited RPC writes dropped on transient backpressure

Mechanism (end-to-end): Libp2pHostFactory.writeFramedBufferWithBackpressure dropped an awaited framed RPC write after only WRITE_BACKPRESSURE_WAIT_MS (200 ms) of connection unwritability — even when the stall was transient and the connection recovered a moment later. A multi-MB payload (a 1.7 MB bytecode response, a 2 MB uploadChunk) legitimately keeps the connection unwritable until the peer drains it (sub-second at datacenter rates, a few seconds for a slow-but-alive peer). The 200 ms drop killed these healthy-but-busy transfers, and the awaiting caller then hung its full 30 s/120 s invocation timeout. This is the one mechanism behind lambdaserver "Stream closed", netlab-hosted uploadFile/apply 30 s timeouts, and buildtest uploadChunk 120 s. #308/#314 only surfaced the drop (hang → fast-fail); they never delivered the message — that coverage gap (asserting surfacing, not delivery) is what let it survive.

Fix — UrlProtocol #319 — protocol 0.0.314 — ✅ MERGED (2026-06-22) + PUBLISHED (kotlin.directory, verified 200).

• New 3-arg overload writeFramedBufferWithBackpressure(channel/stream, buf, maxWaitMs). Awaited paths (writeRpcMessageReturningDelivered, writeStreamingBytecodeMessageReturningDelivered, the NettyOutputStream request path, and the effect-request path) pass WRITE_BACKPRESSURE_DELIVERY_WAIT_MS (5 s) → a transiently-unwritable connection is ridden out and the message delivered on recovery.
• Best-effort gossip unchanged (2-arg overload, 200 ms drop). Binary compatible (2-arg overloads preserved, delegate to the new 3-arg).
• Memory stays bounded exactly as the 200 ms-drop design intended: writeAndFlush is still only called while writable (the event-loop WriteTask queue never grows — the 2026-04-27/06-10 CN OOM root cause), and at most maxConcurrentParkedWritesPerConnection (16) writes park per connection (a Netty channel-attribute Semaphore on the root connection, GC'd with the connection). Beyond the cap → drop fast.
• 5 s is deliberately under the awaited-write fast-fail contract (testSendRequestFastFailsWhenConnectionUnwritable: dropped awaited write must surface within 8 s).
• TDD tests (all in UrlProtocol/tests/): testRpcWriteDeliveredAfterTransientBackpressure (the bug-identifier — confirmed FAILING on pre-fix main, received=[phase1/baseline], message lost; passes with fix), testParkedWritesBoundedPerConnectionUnderSustainedBackpressure (budget bounds parked writes under a flood, still delivers the parked ones), testAwaitedWriteFailsFastWhenConnectionStaysWedged (bounded wait, no infinite park). 18 pre-existing write-path tests re-verified. Full 872-test CI green.
• Downstream: resolver #656 re-pinned resolver onto protocol 0.0.314 ("fixes netlab Stream closed at the owning layer"); resolver main has since advanced to protocol 0.0.332. So this fix is live in the resolver line.

Build-system gotcha that cost time (worth knowing)

@MavenArtifactCoordinates("foundation.url:protocol:") is empty-version, but buildSimpleKotlinMavenArtifact(coordinates = "foundation.url:protocol:0.0.309", …) hardcodes it. When that exact version is already published and in the coursier cache, buildMaven() resolves the cached published jar for the test-compile classpath instead of your local build — so tests compile against stale bytecode and your new symbols look "unresolved." Fix = bump the coordinate to an unpublished version (this is why #319 shipped as 0.0.314, not a same-version edit). Same trap applies to the resolver coordinate. Purge ~/.aibuildcaches/_tmp_work_<repo> + ~/.cache/coursier/v1/bldbinary between src edits, but the version bump is the real fix.

---

Prod netlab investigation (why the 5 live-netlab e2e tests were red) — COMPLETE

The user asked me to investigate before gating. Findings (SSH to root@198.199.106.165:23; the HardwareControlFabric daemon on :8443 was down, so SSH was the justified fallback):

• url://netlab-hosted/ = netlab-manager-server.jar and it is UP and serving (active apply/uploadFile/__bytecode_request, serving a live W3Wallet browser-nat e2e uploading W3WalletDaemon + relay-harness parts to two hosts).
• The host rebooted cleanly at 06:45 UTC (uptime 1:03 at probe time). No kernel OOM, no panic, no oom-kill in dmesg. Prior boot ran 4 days.
• The failure is shared-host contention, not a code defect. The 2-vCPU/4 GB host simultaneously runs the CI test-runner control plane, url://netlab-hosted/ (which itself provisions droplets), the W3Wallet e2e, and ~12 cold-starting CN lazy-service JVMs after the reboot. Load hit 9.64 on 2 vCPU and the netlab-hosted health/connect RPC blew past the resolver's 30 s window → could not acquire a READY netlab environment. Load was already self-relieving (9.64 → 6.50 as the e2e + cold-start herd drained).
• NOT a retry storm / not the drop bug (checked): the repeated Serving bytecode (impl: 2020297 bytes) were legitimate distinct requests (2× per uploadFile part = two browser-nat target hosts, not retries); no Stream closed/reset/drop signatures in the manager log.
• Main's own #644 CI failed identically on the same 5 tests → recurring shared-infra flake, independent of my protocol re-pin (898/903 logic tests passed).

Conclusion: the 5 live-netlab e2e tests (testNetlabP2PIntegration, testNetlabDiscoveryScenarios, testNetlabDynamicFailureRecovery, testNetlabLargeSwarmGossip, testNetlabResolutionUnderLatencyAndPacketLoss) gate every resolver PR's merge on the instantaneous load of a contended shared 2-vCPU host. That's a CI-architecture defect. See the sibling 2026-07-05 handoffs — parallel work (#650 nested-interface ClassCast, #669 SJVM-sandbox-leak OOM) has since fixed other real netlab root causes, so re-check whether these 5 still flake before acting.

My recommendation (pending user decision — they had NOT yet approved when the thread paused): env-gate the 5 live-netlab e2e tests behind RUN_NETLAB_LIVE_E2E (skip in PR CI, run in a dedicated/nightly job with the live infra) as a focused separate PR — do not delete them, do not bump their timeouts. The user's last selection was "investigate prod netlab first"; the investigation is done and supports gating, but the explicit go-ahead to implement the gate was never given. Confirm with the user before gating.

---

PR / branch inventory (all work preserved & pushed)

| Repo | PR / branch | State | What it is | |---|---|---|---| | UrlProtocol | #319 | ✅ MERGED | Awaited-write transient-backpressure DELIVERY fix → protocol 0.0.314 (published). The big win. | | UrlProtocol | #316 | ✅ MERGED | Layer-2: proxy fallback forwards request.metadata (protocol 0.0.309). | | UrlResolver | #641 · branch attach-resourcepath-bytecode-metadata | ⚠️ CLOSED (unmerged) | Layer-3: attach resourcePath to __bytecode_request metadata (direct + relay) + re-pin protocol 0.0.307→0.0.314 + resolver coord →0.0.561. NOT in main. Needs re-landing (rebase onto current main @ protocol 0.0.332 / resolver 0.0.589; renumber coord; 891 protocol test-pins were bulk-updated on this branch). Two TDD tests: testBytecodeRequestCarriesResourcePathMetadata, testBytecodeRequestOmitsResourcePathForRootUrl (both pass locally against 0.0.314). | | ContainerNursery | #466 · branch bump-protocol-0.0.309-resourcepath-fix | 🟡 OPEN, STALE | Bump protocol 0.0.305→0.0.309. Needs update to current protocol (0.0.332+) + new resolver + CN coord bump, then rebuild + redeploy (step 3). | | GithubWatchman | branch wip/bump-resolver-protocol-callback-fix | 🟡 WIP branch (pushed, no PR) | 50 files: bump resolver 0.0.556→0.0.560 + protocol 0.0.303→0.0.309 + coord 0.0.25→0.0.26 (49 test-pins + build.kts). STALE — re-pin to current protocol/resolver before building. Preserved so the bulk edit isn't lost. | | LambdaServer | branch fix-fatjar-stale-lambdaserver-shadow | 🟡 branch (pushed, no PR) | Layer-1: buildFatJar fix so local url-bind classes aren't shadowed by a stale transitive lambda-server:0.0.1. Needs a PR + the correct url-bind jar deployed to prod. |

---

Next steps (to drive to completion)

1. Re-land the resolver resourcePath metadata fix (Layer 3). #641 is CLOSED and the fix is absent from main. Rebase the attach-resourcepath-bytecode-metadata branch onto current resolver main (now protocol 0.0.332 / resolver 0.0.589 — expect conflicts in build.kts changelog/coordinate and the 891 protocol test-pins; verify UrlResolver.kt hasn't refactored the bytecode-request builder). Renumber the coordinate to the next free version. Open a fresh PR, get CI green, merge, publish. Verify the lambdaserver NoSuchMethodError is actually still live first — parallel work may have addressed it another way.
2. Decide + (if approved) implement the netlab e2e gating (Task B). Confirm with the user. If gating: RUN_NETLAB_LIVE_E2E env-gate on the 5 tests, separate PR, plus a dedicated/nightly workflow that sets the flag. First re-check whether the 5 still flake given #650/#669 landed.
3. Step 3 deploy chain (only after 1): update ContainerNursery #466 + the GithubWatchman WIP branch to the current published protocol + resolver, rebuild, and redeploy CN + github-watchman (prefer the ssh-restart-containernursery RestartCli 0.0.5 for CN; HardwareControlFabric daemon on :8443 was down at last check — may need an SSH restart of the daemon first, see the hardware-fabric-processes skill). Then verify the watchman → url://lambdaserver/functions/{id} callback actually succeeds (the whole point).
4. LambdaServer: open a PR for fix-fatjar-stale-lambdaserver-shadow, and confirm the deployed lambda-server.jar is the url-bind build (not the stale Jetty one).
5. Systemic host follow-up (large, separate): the 2-vCPU prod host co-locates CI runners, url://netlab-hosted/, and ~12 CN lazy services with no admission control / cold-start throttling → recurring load spikes. Real fix is CN admission control / not co-locating CI with prod services (CN #458/#459). Per the standing rule, this is a software problem (no backpressure/admission), not a capacity problem — do not upsize the host.

Access / environment notes

• Prod host: ssh -i ~/.ssh/id_ed25519 -p 23 root@198.199.106.165. CN deploy: coursier launch community.kotlin.ssh.restart.containernursery:ssh-restart-containernursery:0.0.5 …. Publish: the publish-maven-artifact skill (coursier launch community.kotlin.maven.artifact.publishing:…:0.0.5 -- --workspace <abs> --buildrule <pkg>.buildMaven --api-url https://api.kotlin.directory/upload --force). coursier is at /usr/local/bin/coursier (not cs).
• kompile tests: bash scripts/test.bash --test <name> (multiple --test flags supported). Never --test . on these repos (800+ tests). WithArtifactVersionMismatchWarningEffect lines are non-fatal warnings, not failures.
• Checkouts live under /tmp/work/<repo>/ (not /code/workspace, which is permission-restricted in this sandbox).