Handoff: Finish the UrlResolver remote gates, failure classification and Clock retries

Written 2026-10-04T15:39:52.810087+00:00. RE-VERIFY: This is a write-time snapshot. Use gh pr view <full URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, git ls-remote, bounded raw buildtest run/result/attempt reads, and actual Maven POM/JAR resolution before acting. A fresh query wins. Original handoff date: 2026-10-04.

Mission

Original request: “Finish the root-cause and fix assignment for testPruneVerificationRequiresUrlRpcProtocol, testFindPeersPollingInterruptionPreservesSignal, and testActiveQueryUnhandledWarningRetriesNextAddress,” encountered during unrelated harness work for https://github.com/CodexCoder21Organization/UrlResolver/pull/1140. This invocation was asked to finish polling gates/classification, recover the strict warning oracle, publish the specifically authorized Noise snapshot30 and open the consumer pin PR, then implement Clock retries if time remained. Remaining work is the remote gates and classification, supervisor final review/merge, and the Clock implementation. Supervisor owns every merge, enqueue, deploy and completion decision. The authorized publication is complete.

Gate: BLOCKED_INFRA. The accepted full and required runs have failed during executor provisioning, before tests can execute. The warning required run remains provisioning with zero execution. Two reviewed candidates are in draft PRs; none is declared ready to merge.

What was found and done

1. The four supervisor rulings remain: prefer the polling fixture PR over its duplicate; preserve the documented stderr record on every established root close; carry the merged Noise owner fix into the consumer; reject breaker seeding that bypasses the three prune rounds. The supervisor closed the rejected prune PR. Protected PRs and relay admission round 4 were not changed.
2. Polling source was verified and rebased on main 1053ace97e38d3a3ee85294b07a3f8465adf6fd0 with no change. Its historical baseline failed 31/32 and its corrected fixture passed 32/32 plus ten final selected runs. This invocation fetched the complete f4d53e0c canonical result set: 1887 passed / 1 failed / 1888. The failure is stressTestBootstrapServiceResyncDoesNotGiveUpUnderSaturation, missing peers in all three attempts. Its mechanism is unknown.
3. The complete attempt journal has 1894 distinct (project, test, attempt) entries: 1889 passed / 5 failed. Resync failed attempts 1, 2 and 3; large ordinary RPC traffic failed attempt 1 and passed 2 and 3; fresh join withdrawal failed attempt 1 and passed 2 and 3. The changed polling test passed its single attempt. Canonical totals hide the latter two initial failures. Wrong-response and late-old-dial were single passes in this run.
4. Seven earlier failures have qualified main failure proof: six fresh failures in 5b4fa251 and the archived warning failure. Wrong-response, large RPC, late-old-dial, and the newly exposed resync/fresh-withdrawal cases are not established as pre-existing by this invocation. Polling resync ee4fc066, main resync 170315e3, and main large RPC 770f4be4 all failed before actual execution. Required polling run 1f23c644 also failed during provisioning. These are not same-test main proofs.
5. The warning-only whole-stderr expectation conflicts with the documented root-close records, independent of the Noise pipeline warning. The saved old strict oracle failed 2/2. Recovered candidate 3785f6dc was corrected to preserve full output equality, one resolver warning, and exact permitted close records. The first revised candidate passed 7/7 on snapshot27 but failed against published30 because it named a removed close wrapper. The final candidate 4d93ec6cdbd075c869770ffeb80895d8c46b5a62 pins30 in its test, uses a named local Runnable on the real event loop, and independently requires that exact caller plus all original metadata. It passes 7/7 on actual published30. Unknown, duplicate or wrong records are still rejected. Full df3b8c11 failed before execution: 0 passed / 0 failed / 1888 unexecuted. The final draft warning PR is open; its required run a3302e67 is provisioning with zero execution at the last raw read.
6. The Noise write-failure owner fix is merged on exact develop ff579d51d91fcb392c244d165a544bec62a0a67c. It sends the failure to the owning handler, completes the pending connect with the original cause, and closes the transport. Fresh exact-develop jar plus NoiseHandshakeWriteFailureTest passed 21 / 0 failures / 0 errors / 0 skipped. Historical full upstream counts remain qualified: base 992/7/2 existing skipped and final 1006/14/2 existing skipped; six failures have same-test base proof and eight do not. The targeted gate does not claim a green full upstream suite.
7. Explicit third-invocation authorization was used once to publish community.kotlin.libp2p:jvm-libp2p:1.3.0-codexcoder21-snapshot-30. All preconditions were recorded: clean exact develop, successful jar and Noise gate, absent metadata and POM 404 immediately before upload. An external Gradle init supplied version30 for project, Jar and MavenPublication without source edits. The generated Gradle POM has the correct coordinate and 27 dependency entries; every jar entry matches the exact-develop build. The publisher succeeded. Downloaded POM/JAR and Coursier resolution were verified; jar SHA256 is d93c07ae09165f190568c7e50bd805b18c76b7d95253c1e85927d5158489bfa3. Do not republish this immutable coordinate.
8. The consumer diagnostic was salvaged into a maintained real-query regression with an event-loop completion barrier and complete cleanup. The identical body fails 3/3 on snapshot27 at the expected complete Netty pipeline-tail warning and passes 7/7 with the downloaded published30 jar. Only the runtime libp2p jar differs. The module and README pin27 to30. Final source df96365a88deeb2559e494dafd22ebb4dc1b762d is pushed and its main-target draft PR is open. Exact-head full 7db6e0a7 failed provisioning, reporting 0 passed / 1889 infrastructure-marked failed / 1889 total, with zero actual attempt entries. Required run 52cb0097 also failed provisioning, zero actual attempt entries. These forced rows are not test failures.
9. Separate test-comprehensiveness and adversarial code reviews were completed for polling, warning and consumer candidates, with the warning integration correction re-reviewed. Findings are on the evidence branch. No timeout/count/assertion weakening, retry wrapper, skipped test, reflection, consuming production workaround, merge, enqueue, deploy or handoff completion was used.
10. Clock behavior remains reproduced and unimplemented. The prior branch has three public Clock scenarios run twice, six intended failures. Callback-close currently fails first at missing schedule registration, so it is not independent evidence of a separate close defect. The exact next implementation design is retained below; item 4 was conditional on completing earlier gates, which remote provisioning blocks.

Relevant PRs / refs

Published artifact: https://kotlin.directory/community/kotlin/libp2p/jvm-libp2p/1.3.0-codexcoder21-snapshot-30/ . Evidence: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/L6-urlresolver-third-2026-10-04/handoffs/artifacts/L6-urlresolver-third-2026-10-04 . Prior exact Clock/release designs and red logs: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/L6-urlresolver-follow-through-2026-10-04/handoffs/artifacts/L6-urlresolver-follow-through-2026-10-04 . Final evidence checkpoint: https://github.com/CodexCoder21Organization/PlanRepository/commit/56522b56c93a1dfb1cd634a7494c99b4f3d17654 . Re-query its head before resuming.

Next steps

1. Re-verify current PRs, main/develop, artifact contents and all accepted run IDs first. Account for a3302e67, 52cb0097, the earlier pin ID eea1b7ac, and prior warning full 6e99a017. Do not infer executor death from WUI projection or kill another lane’s run.
2. Ask the infrastructure owner to inspect createDropletAsync/getDropletCreationStatus and the provider inventory. Coordinator fields show no assigned/acknowledged executor, not proof that the remote create handler never made a physical droplet. EOF/read 0 of 4 bytes and ambiguous RPC response loss are observed; their mechanism is unknown. No authorized local full-suite workaround exists on this six-lane host.
3. Once provisioning is usable, reproduce resync on the polling head and main, and large RPC on main, through targeted runs. Preserve the complete attempts, including fresh withdrawal, and classify wrong-response/late-old-dial only on actual same-test evidence. A passing main run alone cannot prove a flake absent. Do not modify protected relay work; pass its evidence to the supervisor.
4. Finish exact-head full and required gates for the two drafts and the polling PR. Do not re-request terminal unchanged-head failures or invent a commit just to dispatch. The one pin suite re-request was for zero check runs on a wip checkpoint push before PR creation; it recovered dispatch. Follow the supervisor’s legitimate recovery/head-change decision. Rebase immediately before any update and run the required full gate on the exact updated head.
5. Only after all required checks are green, ask the supervisor for final review and merge. Recommend closing the duplicate polling PR and its branch after the preferred polling PR lands; no closure occurred here. Keep both new drafts and the red reproduction branches. The rejected prune PR stays closed.
6. Implement the Clock contract from the following design after the earlier gates. No production deployment is part of this lane. The Noise publication is done and requires no repeat.

Exact Clock implementation brief

Use a nullable, zero-safe ordinary retry start/deadline, separate from withdrawal retention age. Use the same absolute injected Clock milliseconds in drain pre-checks, remaining-window calculations, each message, post-send checks, send helpers, and first-contact admission before and after waiting for the permit. Preserve the 5000 ms budget, existing retry counts, first transport bound, and all three genuine prune failure rounds, including URL-RPC verification rejection, stream closure, retained pooled parent, and stalled controller cleanup.

Replace only ordinary real-time backoff with a package-level suspension scheduled through Clock.schedule at an absolute deadline. Relinquish the cancellation token exactly once even when the callback fires immediately, cancellation precedes token publication, or the callback re-enters. Closing or retiring the lane is terminal: advancing the Clock or completing an old operation cannot dial again or republish the lane. Preserve an already successful prefix, release the unsent suffix, and reach zero queued/in-flight messages before reporting idle.

The public tests must cover ManualClock starting at zero; first dial consuming 5000 ms and then idle with one dial; a frozen backoff registering at +200 ms, still one dial after +199, then a second dial after +1 that consumes the remaining budget; callback-triggered close before release with one cancellation and no later dial; retry permit admission expiring before release with no NetworkProvider.dial; late completion after close/retirement; successful prefix and unsent suffix cleanup. Verify the existing three Clock scenarios fail first (the prior six failures are evidence, not fresh execution); add the remaining lifecycle rows and force the exact ordering through public APIs. Write the resource invariant table first, retain all timeouts/counts, run the exact-head remote full suite, and perform separate coverage and code reviews before pushing the shared-contract change.

Operational knowledge

Use PATH=/code/ws/bin:$PATH in every shell. The system coursier binary has the wrong architecture; repo scripts use an ignored clone-local jars/coursier launcher that executes java -jar /code/ws/bin/coursier.jar. Use fresh clones in /code/ws/L6 only. Never run the full local suite or a large local fat jar on this shared 6 GiB host.

The WUI projection can omit accepted run IDs. ReadRawResults.java and ReadPaged.java are preserved for bounded raw coordinator reads. Full result requests can hit OUTBOX_FULL; paginate and advance by actual returned count, not requested limit. Some service stack traces are truncated and point to test-events.jsonl. Complete canonical and attempt counts were checked for unique keys. A read-only client using the published30 libp2p jar retrieved records after an older client health read timed out; sequential observations do not establish the timeout’s cause.

All launched watchers/clients had bounded deadlines and are now stopped; a final process audit found zero owned background Java/timeout processes. Remote jobs were not cancelled. Hardware fabric credentials are absent, so no authenticated server inventory check was possible and no SSH fallback or server restart was attempted. Structured REST PATCH updates PR descriptions when gh pr edit fails on the deprecated projectCards GraphQL field. The report-challenge workflow auto-merges, which this lane forbids; challenge details are preserved for the supervisor to record separately. Claim reports were uploaded; only this handoff was updated.