Finish UrlResolver timer cleanup verification

Written 2026-10-21T20:00:12.240549+00:00

RE-VERIFY: Live snapshot. Run gh pr view https://github.com/CodexCoder21Organization/UrlResolver/pull/1163 --json state,headRefOid,statusCheckRollup and inspect exact-head bld-build. Only bld-build is required; no merge/enqueue/publication/deployment is authorized.

Latest S65 checkpoint (2026-10-21T20:50:51.968575+00:00; supersedes all earlier S65 run/head numbers below)

Real source09f3753375ea881b5bf5c4f78c9f98ff992f16ee on OPEN https://github.com/CodexCoder21Organization/UrlResolver/pull/1163, rebased cleanly on maind6d3f5321f. No further source change. Exact full https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39667543450 is IN PROGRESS, test step; build succeeded. Required check bld-build only. The watcher is alive. Direct bounded GitHub checks at20:50 UTC confirm the job remains in its test step, started20:07:24, about43 minutes against roughly73 minutes usual duration. A watcher GitHub read timed out once (java.net.http.HttpTimeoutException); direct read immediately recovered and the watcher keeps polling. No evidence of a failed/stalled executor.

Source-identical focused459 https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39667580871 completed458/459. XML independently confirms459 unique names, no missing/unexpected selectors and both build rules successful. All seven new regression cases PASS. ALL322 prior focused cases PASS. Its only failure is accepted unchanged testParkedObservablePollsDoNotStallOtherRequests: unrelated response follows17 of80 poll expirations (expected0). This case was included by adding all Observable-named neighbors; do not remove it or re-run merely to obtain a green badge. Four explicit P2 files remain byte-identical to4645 and await https://github.com/CodexCoder21Organization/UrlProtocol/pull/772. Focused branch https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/s65-focused at29e753027f4f56ee385b5e4589824e575699b71a differs from real09 ONLY in workflow selection. Draft1393 CLOSED after evidence capture.

All seven final regression bodies have fail-first proof. Original five final bodies fail0/5 at exact lock assertions in https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39666352777 (draft1394 CLOSED). Final neighbors fail0/3 in https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39667542086: late-result scheduling, connection-getter callback reentry and strengthened SAME-connection projection rebind (draft1395 CLOSED). Both XML manifests are exact with successful source build. Neighbor branch https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/s65-late-result-negative at f85a8fac47e9aa9096389c989ec5a6d8e9110969 retains old handler/projection source plus latest main, no fix. Earlier5 branchc3f also retained. All evidence drafts1392/1393/1394/1395 closed, unmerged.

Seven regression scripts added, no pre-existing test body/limit edited by S65. Test and source reviews complete: observed early-arm and getter-reentry gaps corrected; no remaining concrete review finding on final09. Observable arming now occurs immediately after EACH outer association monitor, before user work; the intermediate dispatch wrapper was removed to preserve old failure flow. connection.isOpen is sampled outside those monitors, since its getter may invoke onClose inline. HandlerSweepTimer computes deadline lazily after reserving ownership, outside its lock, so active timers cause no extra clock reads. Complete call-site audit and invariant table are in the evidence branch.

Evidence branch https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/s65-artifacts/handoff-artifacts/s65 at def121bdeef4238691b38f893243b898883cd651 contains final negative/focused logs and XML, exact selector manifest and review findings. Historical source4645 full39658630071 completed2378/2382, exactly the four accepted P2 failures, successful build rules. XML and complete failure traces are archived; never substitute it for current09 full. Only final full attribution/terminal cleanup remain. No merge, enqueue, publication or deploy occurred. Timebox ends about21:47UTC.

Design decisions in force

2026-10-21 brief: "Rule: no lock (monitor) is held while calling Clock.schedule (or any API that may run a callback inline) or while any user/cleanup callback runs; e.g. compute what to schedule under the lock, release it, then schedule — making sure a callback that runs immediately sees consistent state and that cancel/reschedule races are handled. Check every other Clock.schedule call site in the handler/projection code for the same pattern and list them with their status."

The request-owned cleanup threading/deadline requirements and absolute prohibitions from S57 remain in force, quoted in the historical record below. Four accepted upstream-dependent tests stay unchanged. Timebox2h for S65. No agents.

Mission and current state

Source fix is pushed at19e466739d2ab258e6aded71b804eadee5a48e8e to OPEN https://github.com/CodexCoder21Organization/UrlResolver/pull/1163. Finish focused457 runner verification and exact-head full-suite attribution. No pre-existing test was edited by S65, and all four accepted P2 test files match4645 byte-for-byte. Main remainedc5bc50c6ca on latest fetch/rebase; no conflicts.

What was found and done

The projection grace scheduler held its state monitor over Clock.schedule, so an honestly due callback inherited that lock during dependency cleanup. The adopted public probe confirms this, and the original S57 negative is https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39662876223. The audit also found both effect-session and observable-orphan sweep schedulers calling Clock.schedule under their locks.

Projection grace now reserves a generation before scheduling outside the monitor. An older window stays valid until its replacement registers successfully; failure unlinks only that pending generation. Rebind/unsubscribe/shutdown invalidate pending ownership, late returned handles cancel once, and stale callbacks cannot reap replacement subscriptions. Both sweeps use HandlerSweepTimer to claim pending ownership under a private data-only monitor; schedule, callback action and cancel happen after releasing it. Observable orphaning only changes data under sweepLock; connection-close and request-completion boundaries arm the timer outside all enclosing locks. Observable shutdown attempts timer cancel, attachments and delegate separately outside the lock, retaining failures.

Negative runner https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39665391815 compiled source4645 and failed0/5 at the exact intended lock assertions. Evidence draft1392 is CLOSED. Four added probes cover pending sweep shutdown, projection shutdown and rebind; the fifth is the unchanged adopted due-inline test. Review then strengthened only NEW probe cleanup using nested finally, retaining every assertion/count/time limit. Exact final bodies re-run negative on c3f1196ee13f013ddf739b5aa527c9320ac00f39; source-identical focused87b89bcc0c2b3ac0053217ddbe49dee2b1c18215 selects457 names (prior322 + new5 + observable neighbors). Both and exact-full are pending; no passing result claimed yet.

Relevant PRs / refs

No code deployed or artifacts published by S65: only Git/PR/evidence/handoff actions were performed. No merge/enqueue command used.

Next steps

1. Complete dedicated test and adversarial source review; fix any concrete findings.
2. Capture final-body negative results and close evidence draft1394; capture exact457 XML and close draft1393 after success.
3. Attribute EVERY exact-head full-suite failure: four accepted P2 cases unchanged, anything else fixed or demonstrated on main. Do not call known flake names sufficient evidence without main reproduction.
4. Update real PR description and this handoff with exact head, scheduling call-site table and runner verdicts; preserve all evidence in artifact branch. Do not merge or enqueue.
5. Before final report stop own watcher processes and delete lane clone/worktree/cache. S65 started around19:47UTC; timebox ends21:47UTC.

Historical S57 record (source4645; superseded by current S65 section above)

Handoff: Fix due-inline projection grace cleanup and finish exact-head verification

Written 2026-10-21T19:44:07.974475+00:00

RE-VERIFY: This is a write-time snapshot. Before acting run gh pr view https://github.com/CodexCoder21Organization/UrlResolver/pull/1163 --json state,headRefOid,statusCheckRollup,mergeStateStatus and inspect the linked exact-head Actions runs. Only bld-build is required.

Design decisions in force

The original lane brief (2026-10-21) is preserved verbatim in the evidence branch. Its binding behavior:

any cleanup step that may run user code or block (closing streams/observables, releasing results, user callbacks) (a) never runs on the caller's thread or inside cancel()/the cancelling call stack, and (b) always runs while the worker's existing 30-second deadline still applies to it, so a blocking close is interrupted at the deadline and the worker then returns its slot. The caller gets its outcome (error or cancellation) promptly regardless of cleanup. Locks guard data only — none held while cleanup runs. No new or raised timeout.

Do not merge or enqueue. Time box 2.5 h.

Shared brief additionally forbids publication/deployment, skipping/deleting/weakening tests, increased timeouts, reduced iterations, reflection/mocks or spawning agents. No approval to merge has been given. Four named P2 tests must stay unchanged until UrlProtocol release linked below. Request work runs to completion after departure, with queue-then-retry admission.

Mission and current state

Fix the confirmed due-inline grace cleanup lock and finish verification and attribution of https://github.com/CodexCoder21Organization/UrlResolver/pull/1163 at source 4645ec6ded71bec44f18a534ad389d6b0bdedbf0. S57 added16 public failing-first scenarios, repaired two original review defects plus discovered deadline/lock edge cases, and pushed all source/evidence. Final exact-head full runner https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39658630071 needs its terminal result checked. Additional source-identical13 existing projection lifecycle/publication neighbors were outside309; run https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39661136831 PASSED13/13 with exact successful XML/build rule. The two focused gates cover322 distinct cases. The latest focused309 run PASSED309/309 with an independently verified exact successful XML manifest and both build rules successful. Earlier186 verification cannot substitute for it. The original2.5h lane ends near19:47 UTC; do not describe pending full verification as DONE.

No artifact has been published and no service was deployed. All mutations were local Git edits, evidence/source pushes, draft evidence PR creation/closure, and PR body updates. No merge/enqueue or production CLI was used. No active goal, agents or scheduler loop exists.

What was found and done

1. Original failed requests set complete and discarded physical-task authority before partial resources closed. Accepted failure now signals its original error early when cleanup is pending while retaining its actual worker and deadline until cleanup finishes. Normal successful outcomes and ordinary no-resource errors preserve old completion/timer ownership.
2. Final cancellation/refusal called resource discard on the cancelling/delivery stack. Discard now only claims registrations under its lock and queues actual closes to an owned pool bounded by maxConcurrentHandlerWorkers, separate from handler/delivery workers. It schedules the original absolute cutoff, not another30seconds; drains and shuts down each batch, and can accept later final handoffs after parent shutdown.
3. Per-action clock guards originally could skip actual release if the read threw. Every guard/action/report is attempted separately and original failures retained. Expiry is terminal after timer firing, including if a close consumes its interrupt and rolls the injected clock backward. Worker admission/resume establishes a cutoff once; delayed waiter registration reuses it and applies past expiry immediately.
4. Projection dependency listener removals recursively invoked actual service subscriber callbacks under the outer projection state monitor. Projection required-pad callbacks also held a separate serialization monitor. Both were proven with public controlled gates before changing source. Every departure path now collects listener/reap releases under state monitor and runs them outside it. Each attachment owns distinct listener identities; installation/release arbitrate once-only detach; stale callbacks cannot seed a new generation. Resource expiry is shared into nested projection releases. Required-pad serialization uses data-only owner/future state; snapshot-producing calls await outside the monitor, cleanup coalesces to the current owner, inline reentry does not self-wait, failures leave the pad retryable.
5. Dedicated review inspected all resource registration sites: StreamAware input-result and parameter-pull entries; ObservableAware result registrations; projection subscription handles; aliases; failure/refused/cancelled/expired result delivery; warnings/reporters; nested property listener detach and reap cancellation. Stream/observable release helpers already claim state before user close; projection was the additional lock defect fixed here. Global client-owned lifecycle shutdown has its prior public contract; this change governs request-owned undelivered resources and operation-owned retirement, rather than inventing another timer for explicit service shutdown.

New final review concern: due callback inside grace scheduling

The final monitor audit found startPushGraceForClosedConnection still calling clock.schedule under the projection monitor. Its callback can become genuinely due and execute before schedule returns; exiting the reap helper's nested monitor then leaves the outer scheduling monitor held during dependency-removal callbacks. This mechanism is CONFIRMED: runner39662876223 compiled and failed0/1 at the exact intended lock assertion in6.0seconds. XML has one unique failed test and successful buildMaven. Full log/XML are in the artifact branch. Evidence draft1391 is CLOSED; its branch is retained. This remaining defect is NOT FIXED on real source4645. New public probe testEffectAwareInlineProjectionGraceCleanupDoesNotHoldState is on https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/s57-inline-grace-failfirst at21d888f5f92582f4c36bba79cc1cde341128db1e, draft https://github.com/CodexCoder21Organization/UrlResolver/pull/1391, runner https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39662876223. Production matches4645; only test/workflow added. It starts a real pending final SUBSCRIBE, advances a real ManualClock by10ms during schedule to deliver a legitimately due callback, holds an actual dependency last-subscriber callback and measures an independent harmless UNSUBSCRIBE before releasing that gate. It uses no early callback, host-speed/load calibration or guessed source fix.

This runner compiled and failed the exact Projection grace-reap cleanup must not hold the state monitor when a due Clock callback runs inside schedule assertion. The structural cleanup rule remains violated and source4645 MUST NOT be described as fully fixed even if full suite only has the known4 P2 failures. Fix belongs at the outer scheduling owner: claim timer/token state under lock, schedule outside it, arbitrate pending timer install/cancel/stale callback/rebinding honestly. Write the timer invariant table and public hostile-caller cases before changing code; no timeout increase or guessed mitigation. Carry the new test into the real fix, preserve all322 prior focused cases, and rerun exact-head full after actual source change. No production fix was attempted near the time limit; do not describe this as merely pending CI. The new test is on its negative branch and must be carried into the real fix. Review original request-owned thread and deadline ownership too, rather than treating one moved synchronized block as the whole contract.

Evidence and verification

All new tests compile and fail at their intended public behavior assertion before their corresponding fix:

All16 new scripts are scenarios in tests/, public API only; real wrappers/dependency observables; no mocks/reflection. Clock/gates force exact ordering, cancellation-return is measured before gate release, actual deadline retirement/physical worker return is awaited before shutdown. Each callback fixture opens its gates in finally. Earlier local source-overlay19-case execution passed, but its final small clock-read ordering correction was verified only by the passing186 runner. Projection is runner-verified because local compile reached its unchanged180s limit; the shared common brief explicitly directs using runner evidence rather than blocking on this loaded box. Do not increase local/test limits.

Five old tests were adjusted only to await actual asynchronous completion, retaining their old failure oracles:

• CancelledFinalHandoffReleasesResultResources / RefusedFinalHandoffReleasesResultResources await actual observable detach before unchanged alias,close,subscriber,sibling counts.
• DiscardedProjectionSubscriptionReleasesHandle awaits release callback before unchanged sibling-pad assertions.
• DiscardedResultReleaseFailureKeepsSiblingCleanup awaits warning/detach and transfers captured effect context to worker; exact warning text,identity,count retained.
• ReleaseReporterFailurePreservesDelegateOutcome awaits executor afterExecute before unchanged suppressed failure identities.

Four accepted upstream-dependent failures are byte-identical to originald4 and must not be weakened: testEffectAwareStartedWaitsLeaveOtherServicesDispatchable, testEffectAwareResumedWaitsLeaveOtherServicesDispatchable, testEffectAwareClientConnectionCloseLetsRequestWorkComplete, testParkedObservablePollsDoNotStallOtherRequests. Their dependency is https://github.com/CodexCoder21Organization/UrlProtocol/pull/772. Originald4 full https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39630194381 had2357/2361 with exactly these4. The unchanged-main baseline https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39600492181 (source48e9; current mainc5bc only connection-publication change, not these paths/tests) also exhibits these failures. Do not call any fifth failure unrelated without showing it on main.

Relevant PRs and refs

All code is unmerged in the open real PR. Original negative drafts and focused1385 are CLOSED. New due-inline grace draft1391 is CLOSED after0/1 intended failure evidence captured. Additional projection13 draft1390 is CLOSED after its13/13 successful XML evidence was captured. Artifact branch is support only, never merge it. Rebase was conflict-free on mainc5bc50c6ca5763271e4294c1ea3f34073e4fb333 before source pushes. No stashes or other worktrees. Historical source commits are ancestors of4645; no local-only source work.

Next steps

1. Re-verify PR/head, latest runner statuses, and latest main. FIRST fix the CONFIRMED due-inline grace failure39662876223 at source, using the concern section above and unchanged public reproducer; this is still engineering work, not only waiting for CI. Additional projection13 run39661136831 passed13/13; branch082a6ba65acf95c2ce3ce4101aaf60ad727cf366 differs from4645 only by workflow selector. Clone fresh under your own workspace; read README/AGENTS, the saved brief/common/testing/philosophy and findings. Full findings/evidence: https://github.com/CodexCoder21Organization/UrlResolver/tree/wip/s57-artifacts/handoff-artifacts/s57.
2. Follow plain build-watchman on the real PR (all evidence drafts are closed), no --to-merged. Informational App lost-dispatch diagnostics must not be re-requested. Gate only bld-build. No polling PR status more than once60s, no open-ended shell waits.
3. Download terminal logs/XML using gh run view <id> --log and gh run download <id> --name test-attempt-records --dir <path>. Evidence helper s57-verify-run.py prints exact unique test/buildrule counts and failures; focused selector s57-focused-tests.txt has309 names. Verify every expected selected test once, zero missing/unexpected, both builds successful. Full source must be exact4645; expected final count derives fromactual XML, not a guess.
4. Attribute full-suite failures. Four named P2 failures are accepted. Any additional defect belongs to this lane unless proven unchanged-main; reproduce deterministically before fixing. No blind CI retries. Shared common permits one failed-job retry only for a proven unrelated flake, and it must be recorded.
5. If changes are needed, keep their failing-first evidence, update source/tests, fetch/rebase latestmain before build/push, check realPR OPEN, push with explicit lease. If main/head changed, don't overwrite others. Run source-identical focused and exact-head full again only for actual changes. No weakened old assertions or raised limits.
6. Update PR description with terminal focused/full counts and attribution; its first paragraph explains original user motivation, footer is correct. Copy new evidence/findings to artifactbranch, push and ls-remote verify. Complete this handoff only once source is fully verified with no unexplained failures. Leave real PR open, do not merge/enqueue/publish/deploy.
7. Kill your own watcher/process tasks, delete your disposable clone and matching ~/.aibuildcaches entry before final report. Never touch other lanes' processes/caches.

Infrastructure observation and verification invocation correction

The first13-neighbor evidence run https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39660494570 compiled but ran ZERO tests because names were incorrectly joined with commas in one --test value. Corrected082a uses repeated --test flags; source/test bodies never changed. Do not count old run as a failed test. The same old job's remote build half reported Execution-duration publication failed for '/root/buildtest-data' ... java.io.IOException: No space left on device. New publications are refused. Marker kotlin-build-ci-upload-request=84ba143c-34ce-47ec-8ef0-5228d3573851 could not be recovered; CLI refused duplicate submission and local racing build succeeded. Root cause of filled storage has not been investigated. The report-challenge skill's CLI auto-enqueues/merges, forbidden for this lane; instead the complete observation/log is in artifact branch s57-infrastructure-challenge.md for the storage owner. No production intervention.

Reusable knowledge

The workflow races local-on-runner and remote BuildTest via --local-and-remote. Only bld-build is required; App checks are informational even if watcher emits PROBLEM lost-dispatch lines. A superseded full runner cannot verify a newer source. Source-identical focused branch must differ ONLY in .github/workflows/bld-build.yml (git diff <realhead>..<focusedhead> --name-only). XML uses root tests with direct test/buildrule success attributes. Original timer expiry must be latched independently of later wall-clock readings; each sibling close/report can consume an interrupt, so guard them individually. Captured effect-handler context must travel when moving release from cancellation stack onto an owned worker. Distinct attachment listener identities are required when detach can overlap replacement attachment.