Handoff: Finish the three UrlProtocol workload diagnoses and required Clock-repair verification

Written 2026-10-21 19:56:34 UTC. RE-VERIFY: This is an in-flight checkpoint; work continues under codex-s64. Re-check https://github.com/CodexCoder21Organization/UrlProtocol/pull/699 with gh pr view 699 --repo CodexCoder21Organization/UrlProtocol --json state,headRefOid,statusCheckRollup,mergedAt. Current head is fe721b763fe4efbab64e13b809b872658f3be144, OPEN and not merged. Required bld-build is SUCCESS, bld-all-tests is IN_PROGRESS. Current task time box ends about 21:53 UTC.

Design decisions in force

2026-10-21 lane brief: “fix the product so all six pass without raising any timeout, reducing load, or editing assertions — and keep the last change's goal: waits on a call's path end at the call's original deadline per the caller's Clock.” “Do not merge, enqueue or publish.”

The earlier Clock deadline requirement below remains in force. No deployment is authorized. No existing real-PR test, assertion, timeout, iteration count, payload or heap changed in s64.

Mission summary

The full suite on the s56 candidate failed 6/2,903: huge-timeout reader; partial-prefix and partial-body Clock failures; abandoned interrupted streams; outbox workload; beyond-heap streamed reply. The s64 task is to diagnose each, repair the product while retaining the original Clock deadline, push to the existing OPEN PR and obtain green bld-build/bld-all-tests. The first three are repaired and proven; the other three are explicitly unresolved pending full-suite context, not classified as unrelated.

What was found and done

1. Rebased 35 commits onto main 0ed6401dc012c0230d31bc914241d3082e071b58 without conflicts. No source change beyond the single byte-wait repair in fe721b763.
2. Established the three reader mechanisms. The injected-Clock branch had changed Condition.await(timeout) to an untimed await. The huge-timeout case explicitly observes TIMED_WAITING, so this was a wait-state regression, not arithmetic overflow (saturating arithmetic was already present). The partial-prefix/body cases make the Clock throw an exact IOException after consuming a fragment; an indefinitely parked reader never reads that Clock again without time advancement, so the test Future timed out instead of receiving the original failure.
3. Restored the existing min(remaining,100ms) condition check while retaining the original absolute Clock wake and post-callback byte/close predicate checks. Prefix/body ownership, interruption and parse-memory cleanup are unchanged. The repair is on the real PR head and duplicate wip/s64-clockfix.
4. Focused unchanged baseline failed those three cases 3/3 each: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39661435290 . Focused repair passed all six originals plus nineteen prior Clock guards, 25/25 first attempt, zero histories: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39661853615 . Separate test coverage and implementation review passes are in findings. No README change is needed for restored behavior.
5. The other three workloads pass first attempt in three unchanged-product focused runs. The diagnostic run https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39662191029 has 37 thread samples: outbox phases show about 20/21 genuine permit waiters alongside ten active reply readers and progress between phases; no lost wake or admission-helper wait was identified. Do not invent a fair-budget or zero-wait-admission fix from these healthy samples.
6. The first main comparison used main's shorter outbox test and is not equivalent for that case. Corrected comparison copies the EXACT existing PR outbox file, with unchanged assertions/counts/budgets, onto unchanged main production code: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39663479534 . All four selected tests pass first attempt; outbox 61.628s versus 64.147s in the PR's healthy diagnostic run. This is an evidence-only test replacement on the main comparison branch; no real-PR test changed.
7. Adding the snapshot-27-bearing existing FUP1 test plus the two failed-attempt neighbors did not reproduce the workload failures: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39663272639 . Only the three reader cases fail, 3/3 each; all workloads and neighbors pass first attempt. That specific dependency-context hypothesis is unsupported.
8. Original full-suite attempt history has all six terminal cases failing 3/3, plus buffered-heap failures on attempts 1/2 before attempt 3 passes and R671b interruption failure before attempt 2 passes. The beyond-heap console reveals the original child failure at chunk 1355/1536 at its now-original 60s deadline; a shutdown publication-executor rejection later masks it. This is not evidence of an identified shutdown fix.
9. Recent main full-suite artifacts https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39655081474 , https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39646894011 , https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39637390517 show no target failed attempts. Identical beyond-heap tests take 50.545/63.000/63.551s including setup/cleanup; ten-interruption tests take 49.994/80.741/80.193s. Hypothesis only: cumulative work that an old per-frame refreshed budget tolerated may now reach the intended original call deadline. Full-context profiles must distinguish active processing from genuinely stuck ownership; no throughput repair is established yet.

Relevant PRs and branches

Older s56 branches and their proofs are retained in the historical section below; their remote SHAs were re-verified unchanged at this checkpoint. All six local worktrees are clean; no stash or commit exists only locally. No code was merged, deployed, published, enqueued or uploaded to production. Runner build artifacts and dependency caches are deliberately excluded; forensic records are on the proof branch.

Next steps

1. Re-verify live head and checks first. Do not substitute old focused results for final-head required verification. Final real full suite: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39663008968 ; required build passed: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39663009002 .
2. Read exact-workload main full result https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39663479429 and diagnostic full result https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39664142217 . Diagnostic artifacts include s64-profiles and the ordinary test_log_file.xml. Both keep every original test/timeout. Preserve failed-attempt history even if the final check is green.
3. For each workload failure state the mechanism and evidence before any repair. No failure has been shown on main yet, so none is unrelated. The scoped dependency-context experiment and healthy focused samples do not explain the original full failures. Do not speculate by changing fairness, increasing a timeout, reducing workload or weakening an assertion.
4. Push any proven repair to the still-OPEN real branch after fetch/rebase and explicit force-with-lease; final required gates are bld-build and bld-all-tests, plain build-watchman only. Do not merge/enqueue/publish/deploy.
5. Stop and report at the task's 2.5-hour box if unfinished; stop owned background monitors, push all evidence, delete the lane's checkouts and matching aibuildcaches entries. Do not leave requested failures silently deferred or call incomplete verification complete.

Operational knowledge

• Shared-box local tests are prohibited by the current brief; use the GitHub runner verification process. All focused runs race local-on-runner and remote. Remote submission failed with '/root/buildtest-data: No space left on device'; the runner continued independently. No service mutation or retry was applied.
• App checks are informational even when watchman warns about a queued App suite. Do not warm/rerequest those suites. Required Actions jobs are dispatched and active.
• Evidence-only redundant full jobs were canceled; canceled jobs on those drafts are not test failures. The exact main full suite remains deliberately active. Closed positive/negative drafts were never merged.
• GitHub job logs are not downloadable while a job is active (BlobNotFound); do not classify that as a stopped run. Test output is buffered until the command finishes.
• No scheduling tools are exposed; status reports are produced during active work, uploaded to this handoff, and the claim remains codex-s64.

Historical s56 checkpoint (superseded current-state claims; preserve proof and branch provenance)

Handoff: Finish required CI verification for UrlProtocol original Clock deadline waits

Written 2026-10-21 18:17 UTC. RE-VERIFY: All state is a write-time snapshot. The PR is open at 1f1426f67848634628ee9a3adae2b98f961891b2. Re-check https://github.com/CodexCoder21Organization/UrlProtocol/pull/699 with gh pr view ... --json state,headRefOid,statusCheckRollup,mergeStateStatus; required checks are bld-build and bld-all-tests.

Design decisions in force

2026-10-21 lane brief: “every wait that is part of a call — first attempt, resend, and the wait for an accepted resend's answer — ends at the call's original deadline as measured by the caller's injected Clock, with the call's normal timeout error, and releases what it holds.”

“No new or raised timeout; no wall-clock waits that ignore the injected Clock on this path.” “Interruption still respected.” “Do not merge, enqueue or publish.”

Mission summary

The user assigned lane s56 a 90-minute Clock correction to https://github.com/CodexCoder21Organization/UrlProtocol/pull/699 after a reviewer proved that an accepted response-chunk resend remained in a wall-time CompletableFuture wait after ManualClock expiry. That PR belongs to the earlier extreme-condition UrlResolver effort, but this lane is limited to the Clock wait correction and green required checks, with no merge, enqueue, publication or deployment. The broader old handoff is already completed/archived and was read for context, not rewritten.

What was found and done

1. Rebased the existing branch on current main with no conflicts. Every original application/response-fetched sequence retains one deadline captured at call entry.
2. Proved seven deadline guards fail 3/3 before the fix: both accepted resend routes, both initial replies, real shared-permit waiting, initial transport admission, and the strengthened existing original-direct-deadline test. Evidence: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39643105988 (closed draft https://github.com/CodexCoder21Organization/UrlProtocol/pull/787).
3. Direct replies use the original absolute Clock deadline and remove exact pending futures on all exits. Forwarded reads receive the caller Clock and original operation deadline through prefix/body/parse admission. Shared fetch permits use fair tickets and Clock wakes. Transport admission uses only whole-frame nonblocking offers separated by Clock sleeps within unchanged 200ms/5s caps. Only explicit not-started chunk replies are resent; application requests stay outside resend loops.
4. Initial fixed validation passed 14/14 with zero failed attempts: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39643485031 (closed evidence draft https://github.com/CodexCoder21Organization/UrlProtocol/pull/788).
5. Expanded Clock-wrapper review found that a Clock can deliver real bytes/remote close before the new byte condition wait begins. Existing unedited testRelayBodyReadObservedRemoteCompleteClose failed 3/3 at https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39645866839. Added locked predicate checks after Clock calls; 18/18 then passed without retries at https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39647438560, including unchanged listener timer-count and admission/close neighbors.
6. Clock cancellation ownership: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39648751805 fails 3/3: expected ten fetch permits, actual nine after a delegated ManualClock cancellation throws. acquire had decremented available but its caller never received the grant or incremented outstanding. The final candidate returns that grant before propagating the original cancellation failure. The public test holds actual reply bytes, causes a real fair grant, propagates the original injected failure, settles the application forward, and proves the next call on the same physical root can issue ten simultaneous fetches. The strengthened baseline failed 3/3 (expected ten, actual nine) at https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39649679037. Final 19-case fixed verification succeeded at https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39649939407. No original test was changed for this ownership case.

Only one test from the starting PR was edited: testDirectRegisteredChunkFetchResendUsesOriginalCallDeadline. It still spends 29,000ms in the application and advances 1,000ms, retains 63 parked calls and exact full error/application-count assertions, but holds an accepted resend reply instead of the refusal sleep. The parked setup uses an independent SystemClock requester; observing six initial refusals strengthens its ordering. It fails without the fix in the baseline above. No timeout, heap, payload or iteration count was weakened.

Wait table

Relevant PRs / refs

Current required runs: https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39649885322 (build) and https://github.com/CodexCoder21Organization/UrlProtocol/actions/runs/39649885320 (full suite). Snapshot: bld-build=success, bld-all-tests=in_progress.

No deployment, Maven publication, merge or queue operation was issued in this lane. Builds only produced runner/local artifacts. Nothing live needs reconciliation with main.

Next steps

1. Re-verify the real PR head and effective required checks. The final focused runner succeeded. Read the remaining full-suite result and address any finding before calling the complete required gate green.
2. Watch final bld-build and bld-all-tests using plain ~/bin/coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlProtocol --pr 699 --idle-timeout-minutes 120. Never use --to-merged. One watcher per PR; stop it before handing off.
3. Download the full suite's test-attempt-history artifact and inspect final pass/fail and attempt histories, especially all Clock guards and the previously reported buffered-workload observation. Do not call a failure unrelated without evidence on main; do not blindly rerun a failed job.
4. Stop when the current head's required checks are green. No merge, enqueue, publication or deployment is authorized by this handoff.

Operational knowledge

• Current CI gates are GitHub Actions bld-build and bld-all-tests, both racing runner execution against remote. The two kotlin.build App checks are informational; do not rerequest them even if watchman prints lost-dispatch warnings.
• The shared /code box could not finish one bounded 600-second compile; no local tests were run. Runner verification is explicitly authorized by this lane's common.md override. Do not restart local compile/test loops here.
• The previous full suite took 73 minutes; pending CI is not evidence of a failure. Github PR status polling is at most once per 60 seconds.
• Old gh pr edit fails on deprecated projectCards; update a concrete body with REST PATCH using a JSON file and gh api --method PATCH repos/CodexCoder21Organization/UrlProtocol/pulls/699 --input <file>.
• A Clock gate inside the outgoing-response synchronized monitor can prevent other virtual handlers progressing; the permit tests instead hold actual Netty outbound bytes without blocking any thread and flush/release every retained message/promise during cleanup.
• All tests and code are on remote branches. Proof XML/logs are GitHub Actions artifacts linked above and can be fetched with gh run download <run-id> --repo CodexCoder21Organization/UrlProtocol --name s56-clock-evidence. Scratch generators are not authoritative; no build output or cache belongs in the repository.

All three checkout/worktree status sweeps were clean, no stashes and no branch commits existed only locally. Remote branch SHAs were verified with git ls-remote before writing. Proof XML/logs are already stored in the linked Actions run artifacts; local duplicates and generator scripts are not required for pickup. This lane published/deployed nothing. The old broader handoff is completed/archived: https://www.handoff.wasmserver.com/handoffs/hf-2026-10-14-land-the-urlresolver-extreme-condition-bug-fixes-fix-the-two-merge-queue-flakes-then-merge-the-reviewed-prs-and-finish-the-rest . This new record only covers the remaining Clock verification, so that archived multi-project record was not replaced.

Terminal lane status: the 90-minute time box ended. Final focused run succeeded; bld-build succeeded; final real-head bld-all-tests remains in progress. Source changes are finished and remotely checkpointed; only required full-suite verification remains. No user decision is pending. Local monitors and checkouts are being removed; remote Actions continues.