Make the UrlProtocol main branch reliably green: land the reviewed fix PRs once the build service provisions again, then finish the flake queue

Written 2026-10-02 ~21:45 UTC by session fable-code6d-2026-10-02 (explicit stop by the operator). RE-VERIFY before acting: everything below is a write-time snapshot (PR states and checks verified at 2026-10-02 21:40 UTC). Re-check with gh pr list --repo CodexCoder21Organization/UrlProtocol --state open --json number,headRefOid,mergeStateStatus,statusCheckRollup, gh api repos/CodexCoder21Organization/UrlProtocol/commits/main, the merge queue GraphQL query, and https://buildtest.kotlin.build/api/health. Previous campaign state (2026-09-20 to 2026-10-02 morning) is in the earlier versions of this handoff's reports (handoff-cli reports <id>) and in the artifacts branch below.

Mission (operator's /goal, verbatim)

"ensure the main branch of https://github.com/CodexCoder21Organization/UrlProtocol is reliably green. Also triage the open pull requests, decide which are bad/obsolsete and close them or which are good/salvagable and fix/merge/salvage the good parts and drive to completion. Parallelize to the extent possible and race PRs."

Where it stands (verified)

• main = 78deea8c61311e47596dad9c99b72763292e70e8 (unchanged since 2026-10-01; nothing landed today). Merge queue: PR622:AWAITING_CHECKS. Two entries (PR 622 by a sibling session and PR 626 by this one) were removed by the merge-queue bot at 20:10 UTC after 622's queue run exceeded the build coordinator's 180-minute deadline.
• The shared build service (kotlin.build (remote), the required check) cannot currently finish or even start runs. Since ~17:50 UTC every UrlProtocol head fails before tests run with "Provisioning failed before tests ran: Provisioning deadline exceeded for runId=... currentAttempt=4" (runs 3d6c934b/629, cbbb7a50/633, f550dd4d/636, 3a562f7b/618); run 98bf14af (634) lost all shards to "Provisioning stopped ... method=createBuildDroplet" after 2241 passes; run d9c01208 (621) "Upload of build failed after 5 attempts". Earlier (14:49-16:30 admissions) runs ran but exceeded the 180-minute overall deadline while TESTING at 94-97 percent (fleet oversubscribed: 38 TESTING runs at 17:10; 120 active droplets at 21:24). Challenge filed: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-02-1812-remote-check-runs-on-urlprotocol-exceed-the-ci-coordinator.md (and the morning one: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-02-1040-build-service-infrastructure-not-test-flakes-is-the-main.md). A read-only investigation lane was started at 21:27 and stopped 15 minutes later with no conclusion (findings in the artifacts branch: findings/invprov-findings.md). Nothing can land until provisioning works; do not re-request checks into a saturated or non-provisioning fleet (each re-request burns ten droplets for three hours).
• Decision: no per-PR coordinate bumps. Every PR keeps main's coordinates = "foundation.url:protocol:0.0.569" line (see REBASE-TEMPLATE.md in the artifacts branch); one publish PR bumps the coordinate after the batch lands (precedent https://github.com/CodexCoder21Organization/UrlProtocol/pull/625). PRs that still carry a bump (636 at 0.0.602, 634 at 0.0.595, 621 at 0.0.592, 629 at 0.0.598, 598 at 0.0.596, 616 at 0.0.597) get it dropped at their no-bump rebase.
• Two decisions are open with the operator (asked in the status reports, unanswered): (1) may the main-branch test testForwardedOutputRefusalPreservesAtomicBoundary be corrected? Its final clean-write assertion contradicts the refusal behaviour (a refusal closes the incomplete backend source, so the backend's remaining write gets Broken pipe); reproducer testForwardedOutputRefusalBeforeBackendTailWrite fails 3/3 on PR 616 and 3/3 on main (lane f616b). Recommendation A: fix the fixture in its own PR, then 616 lands behind it. (2) The remaining 30-second timeout families (candidate-prepare stress, materialization budget, peer-exchange admission, TCP-server-runs-every-request, large-relay-response, aggregate-refusal-throttle) are environment-bound: droplet CPU sharing after kompile-core PR 319 dropped the C1 compiler cap for test JVMs (memory note; partly mitigated by kompile-core PR 350). Recommendation A: restore the test-JVM compiler cap / thresholds in kompile-core; then B: partition the two heaviest tests. Investigations that refuted code-level hypotheses with numbers: lanes fdial, fdial2, f4, f5 (findings in the artifacts branch).

Open PRs (verified 2026-10-02 21:40 UTC)

Ownership and next action per PR of this campaign (all authored by codexcoder21 = this session's delegates unless noted):

• 626 (TcpServer lifecycle rewrite + effect-delivery deferral): final-reviewed by me, two delegate review rounds clean; was enqueued, evicted 20:10. Next: no-bump rebase (REBASE-TEMPLATE.md), re-enqueue when the service provisions.
• 636 (relay keepalive ticks and pong deadlines on the injected Clock; re-proposal of closed 585): opened 18:56 from branch fix/relay-keepalive-on-injected-clock at c7dde8b8; local proof 34/34; Actions suite red on this head: testCurrentGenerationFailedRelayRegistrationSchedulesNextReconnect and testCurrentGenerationThrowingRelayRegistrationSchedulesNextReconnect ("The initial registration must schedule exactly one reconnect check") - plausibly caused by the keepalive tick now sharing the Clock those tests count - plus testPendingRelayRegistrationPreservesSameRelayFailuresUntilAcknowledged (main's flake, fixed by 634). Full log: artifacts branch actions-logs/actions-636-110981530691.log. A fix lane (brief p-f636.md) ran 10 minutes and was stopped; nothing pushed. My own read of the source diff found nothing blocking; two points for the delegate review (brief p-rv636.md): a possible idle-tick leak if a connection is replaced without detaching, and the pong-timeout expression's readability. Next: fix the Actions failures (f636 brief), then the delegate review, then my final review.
• 634 (pending relay registration recovery): review MUST-FIX closed by e2eb21ef (failure ownership captured under the registration monitor; proof 28/28, 10/10). Delta re-review (brief p-rv634b.md) was running 15 minutes when stopped; no comment posted. Next: re-review, then no-bump rebase (drop 0.0.595), re-request, enqueue.
• 621 (jvm-libp2p snapshot pin; shared with the sibling "drain" session which once enqueued it): eaa993a3 (seven pause fixtures observe the pause) and dc832c02 (review finding 8: deadline guards closed at 63 s despite a 4,096 s declaration budget; fixed; mutations 7/7; validation 9/9). Actions red on this head was a curl: (22) ... 504 downloading tooling; I re-ran the failed job at 21:31 (check its result). Next: delta re-review (brief p-rv621c.md), then rebase/land; coordinate 0.0.592 to drop.
• 629 (mDNS startup stall isolation; upstream jvm-libp2p issue 43): final-reviewed; should-fix verified and pushed 449425ed (selection proof discriminates). Next: no-bump rebase (drop 0.0.598), re-request, enqueue.
• 630 (root-close reconciliation test barriers) and 633 (synchronised FakeFileSystem fixture): test-only, final-reviewed; remote red on infrastructure only (630's run: three environment timeouts, two assertion flakes listed in the inventory addendum, then SSH loss on all shards; 633: interrupted submission twice, then provisioning). Next: re-request when provisioning works; enqueue on green.
• 603 (admission probe cleanup), 598 (announcement snapshot + replay defect; three review rounds), 611 (awaited-response outbox grace; nine findings fixed): final-reviewed; remote red on the coordinator deadline / deleted-run upload. Next: no-bump rebases after the first landing, re-request, enqueue.
• 616 (receive lanes): fixed and final-reviewed but CI red on the pre-existing main test defect = decision 1.
• 594 (connection-listener isolation diagnostics): held on decision 2; review rv594 found five MUST-FIX items (see finals/rv594-final.md); staged fixes on wip/594-listener-isolation-fix-handoff-2026-10-02 (unverified).
• 628 (draft reproducer, this campaign), 622 (sibling session; rv622 review posted with a MUST-FIX on description references), 604, 574, 618, 617, 631, 632, 635, 637: other sessions'. 585, 580, 578 closed by triage (585 with nine re-proposals: M12 = 636, M2 = listener-failure tests below, M3 and six more pending; assessment in findings/s585-findings.md).

Flake inventory and what was fixed

• Fixed in open PRs: root-close reconciliation ticks (630), pending relay registration (634), FakeFileSystem fixture race (633), mDNS startup stall (629), admission probe (603), relay cache/replay (598), keepalive on injected Clock (636).
• New, unfixed, with evidence (see inventory-addendum.md in the artifacts branch; counts are over today's five full remote runs): testFiniteBackpressureWaitDoesNotRestartForOtherTrafficProgress 3/5 ("A finite 200 ms admission budget must not restart when unrelated channel writes drain. Expected <0>, actual <1>"); testRelayEmitsWithdrawalAtExpiryAfterProviderTransportClose 2/5 ("The observer must register with the real relay before the provider announces its service"); testNettyFrameDeadlineBeforeBodyIsTerminal 2/5 ("Expected ExecutionException ... but was completed successfully", testNettyFrameDeadlineBeforeBodyIsTerminal.kt:127). Briefs p-fbackpressure.md, p-fwithdrawal.md, p-fnettyframe.md are written; the first two ran 15 minutes and were stopped (scaffolding only on their wip branches).
• Root close racing event-loop shutdown (testProtocolRootCloseCallerRacingEventLoopShutdown, from PR 605, failed on 621's Actions run at iteration 42 "the root is closed"): a deterministic diagnostic on branch wip/root-close-racing-event-loop-shutdown-handoff-2026-10-02 reproduces 1/1 on main: the Netty event loop terminates without closing the root, the physical close is rejected (RejectedExecutionException wrapped in IOException), every caller shares that failed result, and the root stays open. Invariant to implement: after any caller's close completes, success or failure, the root is closed, and the shared failed result still carries the rejection. Brief p-frootclose2.md. Ten amplified copies all timed out at 30 s (not a usable reproducer).
• Listener failure closes its own transport (585 re-proposal M2): seven public-API tests fail 7/7 on unchanged main (three invariants: close transport before the listener exits; preserve the listener cause and suppress cleanup failures; fatal Error still propagates); on wip/relay-listener-failure-closes-transport-tests-handoff-2026-10-02; no fix yet. Sub-handoff: https://www.handoff.wasmserver.com/handoffs/hf-2026-10-02-finish-the-relay-listener-startup-and-fatal-failure-fix.
• Also pending from reviews: 585 M3 (listener settles before close observer), a stale-selector race noted by lane f2, fdial3 instrumentation (low priority).

Relevant PRs / refs (branches pushed at stop; all verified with git ls-remote)

Deployed or published but not merged: nothing. This campaign published no artifact and deployed nothing (checked: no publish lane ran; kotlin.directory was not written to; the coordinate stays 0.0.569 on main).

Next steps

1. Confirm the build service provisions again (https://buildtest.kotlin.build/api/health, recent runs reaching TESTING and COMPLETED); if not, that is the blocker and belongs to the service owner (invprov findings give the starting evidence; see memory notes on droplet-service RPC stalls and the droplet limit).
2. Re-run Actions on 621 if still red; fix 636's Actions failures (brief p-f636.md) and re-review 636 (p-rv636.md) and 634 (p-rv634b.md) and 621 (p-rv621c.md); delegates never merge - the final review and the enqueue stay with the orchestrator.
3. No-bump rebase (REBASE-TEMPLATE.md) and re-enqueue 626 first; then 629, 634, 598, 603, 611, 621 in any order (independent); re-request 630 and 633 and enqueue on green; after the batch, one publish PR bumping the coordinate.
4. Flake queue, ordered by recurrence: root-close fix (p-frootclose2.md), fbackpressure, fwithdrawal, fnettyframe, the listener-failure fix (M2), M3, stale-selector race.
5. Get the operator's answers to decisions 1 and 2; act on them (616 and 594 depend on them).

Reusable / operational knowledge

• Lane rules that held: at most three test-running lanes share the host's single kompile lock (flock SCRATCH/local-build.lock scripts/test.bash --local --test ...; batch selectors into one invocation; never hold the lock idle); 120-minute budgets; checkpoint unverified work as patches or wip branches; one push per lane; reviews run against the final head and are void if it moves.
• gh pr edit fails on this repo (deprecated Projects field): edit descriptions with gh api -X PATCH repos/CodexCoder21Organization/UrlProtocol/pulls/N -F body=@file.
• Build-service ground truth is on the host: ssh -p 23 root@198.199.106.165, /root/buildtest-data/runs/<id>*/{run.json,test-events.jsonl,build.log}; the API hides runs older than the newest window. Classify a red remote check from its summary first: "Provisioning deadline exceeded", "Provisioning stopped", "Upload ... failed", "interrupted before its run ID was recorded", "exceeded its overall deadline of 180 minutes", "SSH session is not connected" are all infrastructure; only named FAILED tests are the PR's.
• Re-request a check suite with gh api -X POST repos/.../check-suites/<id>/rerequest after warming https://github-webhooks.wasmserver.com/; when the coordinator stays bound to a deleted run, push an empty commit instead.
• build-watchman 0.0.21 as a tracked background task per PR (--pr plain, --until-merged for the queue); its raw-log probe against the host fails today, which blocks its stall verdict but not polling.
• Do not pkill -f a string that appears in your own shell command (it kills your shell); kill delegate lanes by the pid that owns their -o <lane>-final.md flag, then sweep their detached test loops.