HHandoff
← Priority list

Handoff: netlab `uploadFile` relay reply-drop — all resolved except final merge of UrlResolver PR 632

Original mission (2026-07-05): root-cause and durably fix the netlab CI failure `RPC request 'uploadFile' to service 'netlab-hosted' timed out after 30000ms`, then land the netlab scenario-test PRs it was blocking ([UrlResolver #631](https://github.com/CodexCoder21Organization/UrlResolver/pull/631), [UrlResolver #632](https://github.com/CodexCoder21Organization/UrlResolver/pull/632)). The root cause was a UrlProtocol relay transport bug: the relay wrote awaited RPC replies with the 200 ms best-effort `writeFramedBufferWithBackpressure` overload and ignored a dropped write, so the client hung its full 30 s timeout. Fixed TDD-style in [UrlProtocol #329](https://github.com/CodexCoder21Organization/UrlProtocol/pull/329) (merged 2026-07-05).

hf-2026-07-05-netlab-uploadfile-relay-reply-drop-root-cause-fix-and-deploy-chain

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Handoff: netlab uploadFile relay reply-drop — all resolved except final merge of UrlResolver PR 632

Written: 2026-07-12 ~16:42 UTC (this snapshot fully replaces the 2026-07-05 version of this file — the deploy chain it described is long done). RE-VERIFY before acting: gh pr view 632 --repo CodexCoder21Organization/UrlResolver --json state,mergeStateStatus,statusCheckRollup,mergedAt is authoritative, never this doc. PRs merge, the merge queue shuffles constantly, and buildtest infra false-reds under load.

Mission summary

Original mission (2026-07-05): root-cause and durably fix the netlab CI failure RPC request 'uploadFile' to service 'netlab-hosted' timed out after 30000ms, then land the netlab scenario-test PRs it was blocking (UrlResolver #631, UrlResolver #632). The root cause was a UrlProtocol relay transport bug: the relay wrote awaited RPC replies with the 200 ms best-effort writeFramedBufferWithBackpressure overload and ignored a dropped write, so the client hung its full 30 s timeout. Fixed TDD-style in UrlProtocol #329 (merged 2026-07-05).

Everything in the original handoff is resolved except one item: UrlResolver #632 is validated but not yet merged. The only remaining work is shepherding it through UrlResolver's busy merge queue, then deleting this handoff.

What was found and done (verified 2026-07-12 ~16:42 UTC)

  1. Relay fix deployed to prod — done by parallel fleet-redeploy sessions before 2026-07-12. Verified directly: the deployed /root/ContainerNursery/bin/container-nursery.jar on 198.199.106.165 contains StreamRelayMessageSender.sendMessageAwaitingDelivery (the fix), the main CN process runs it, https://kotlin.build returns 200, netlab-manager is up. The original handoff's user-gated "redeploy prod CN" step is moot.
  2. UrlProtocol #331 (coordinate bump to 0.0.324) — closed as obsolete 2026-07-07; main's coordinate is now ≥0.0.341 and the relay fix is in every published protocol ≥0.0.329.
  3. UrlResolver #673 (bump to protocol 0.0.324) — closed as superseded 2026-07-11 (main already pins protocol 0.0.337 + jvm-libp2p snapshot-10). The Yamux-regression disambiguation it was gating is moot.
  4. UrlResolver #631 (3 netlab scenario tests) — rebased onto main 2026-07-11, passed branch + queue suites (100/100 each; the netlab tests pass end-to-end, live confirmation the relay fix holds), MERGED 2026-07-12T00:33:48Z as 5f84db1b.
  5. UrlResolver #632 (Round-4 netlab impairment tests) — de-duplicated after #631's merge. Head d24887ba = main + one commit touching only NetlabHarness.kt (+17/−3) and adding tests/testNetlabImpairedNatRelayScenarios.kts + tests/testNetlabLinkFlapAndProviderMigration.kts. NOTE: a first restack (1e67d4c4) cherry-picked the stale handoff commit and was corrected by a parallel session to d24887ba (reconciled WithArtifact pins, setPacketLossRate→setPacketLoss rename, added assertions) — do not "restore" the older content; d24887ba is canonical.
  6. #632 validation state: branch checks passed overnight (2026-07-12, after three infra false-reds on 2026-07-11 night documented in this merged challenge: host load-110 storm, orchestrator getTestResults 120 s timeout, DO rate-limit gate self-embargo). Enqueued 14:28Z; one queue candidate was evicted ~16:33Z ("manual" removal ahead of it — shared-queue churn); build-watchman auto-re-enqueued once. At write time the new candidate (queue ref c3b8c4ba) has queue:kotlin.build (remote) already SUCCESS and queue:bld-build in progress (GHA run job 86670603289, typically ~67 min).
  7. A build-watchman 0.0.12 instance with --to-merged was shepherding #632 in the writing session (auto re-enqueue on eviction, exits at MERGED). That watcher dies with its session — re-arm your own (see next steps).

Relevant PRs / refs (state at write time — RE-VERIFY)

| Item | State 2026-07-12 16:42 UTC | |---|---| | UrlResolver #632 | OPEN, head d24887ba, in merge queue (ref c3b8c4ba), queue suite ✅ / queue build in progress | | UrlResolver #631 | MERGED 2026-07-12T00:33:48Z | | UrlProtocol #329 | MERGED (the relay fix); deployed to prod CN (bytecode-verified) | | UrlProtocol #331 / UrlResolver #673 | CLOSED, obsolete/superseded | | Infra challenge report | merged |

No uncommitted/unpushed work anywhere: the session's UrlResolver checkout is clean at d24887ba (matches the PR head).

Next steps

  1. Shepherd UrlResolver #632 to MERGED. Preferred: GH_TOKEN=$(gh auth token) cs launch buildwatchman:build-watchman:0.0.12 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr 632 --to-merged (run as a tracked background job; note /usr/local/bin/coursier on the /code box is wrong-arch — use a working cs, e.g. ~/bin/cs). It auto-enqueues if the PR fell out of the queue and auto-re-enqueues one eviction. The PR's code needs nothing: every test run has passed; failures so far have all been infra (diagnose against the challenge report's signatures before blaming the diff, and ground-truth runs on the host: ssh -p 23 -i ~/.ssh/id_ed25519_20260706_035316 root@198.199.106.165 'stat -c "%y" /root/buildtest-data/runs/<runId>/build.log').
  2. If evicted a second time, diagnose the queue-ref conclusion by hand before re-enqueueing (watchman only auto-remediates once), and check /root/cn-watchdog.audit for whoever is manipulating the shared queue.
  3. After #632 merges, delete this handoff (git rm + PR, pre-approved per handoffs/README.md) — the workstream is then fully complete.

Reusable / operational knowledge

  • When restacking an inherited PR, rebase the live remote tip, never a commit id remembered from a handoff — handoff ids go stale the moment a parallel session reconciles the branch (burned here on 1e67d4c4 vs b04cd9f7).
  • UrlResolver's merge queue is shared and churny: "manual removal" evictions discard in-flight candidates; queue waits and candidate rebuilds are normal, and queue scheduling wait is not run time.
  • buildtest false-red signatures seen on this PR (all infra, none code): netlab uploadFile 30 s timeouts during host load ~110; SJVM ... getTestResults timed out after 120s failing a healthy run; DO droplet-provisioning ratelimit-remaining=0 insta-fail (the rate-limit gate can self-embargo while the provider is healthy — a gate-container restart cleared it on 2026-07-12 ~03:09Z).