Handoff: Finish the remaining UrlResolver test failures and full-suite verification

Written 2026-10-18 13:19UTC. RE-VERIFY: Every state below is a write-time snapshot. Re-check the PR with gh pr view 1316 --repo CodexCoder21Organization/UrlResolver --json state,isDraft,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, remote branch SHAs with git ls-remote, and the build run's raw attempt journal before acting. Fresh state wins.

Design decisions in force

2026-10-18 owner direction, unchanged:

One PR against UrlResolver main that makes these tests pass deterministically on a machine shaped like the remote one, by fixing their actual cause.

For each test: state the mechanism in writing with evidence. PUBLIC_IP: decide whether the defect is in the test (it must control its own address environment through a public seam, e.g. explicit configuration, never by depending on the host) or in product address selection (a provider reachable on loopback should not be advertised/selected by its public IP) — check how PublicAddressDiscovery and setExplicitPublicIp are meant to be used and what production needs; fix at the owning layer.

Reproduce each deterministically first (PUBLIC_IP set; the remote's resource shape — the runner record shows a 2000-millicore budget, so pin to 2 CPUs with taskset/-XX:ActiveProcessorCount; genuine concurrency only, never artificial load). Push the failing state (or a reproducer) before fixing.

Fix; show each passes first attempt repeatedly (10 consecutive first attempts per test, attempt counts from the runner records — kompile-cli auto-retries up to 3 times, so read attempt records) both with and without PUBLIC_IP set; revert control red.

If a test cannot be fixed within the bar, say so with evidence; do not hold the others.

Run the targeted family plus the full suite once (remote once; if cut off by the buildtest provisioning deadline https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-10-18-1033-buildtest-full-suite-runs-for-urlresolver-fail-with.md run locally in a quiet checkout). Never run git in a checkout while a build/test runs there.

PR description opens with the why above. List every edit to an existing test with before/after and why; no assertion may be weakened.

Fresh gh clone of UrlResolver main into your -C directory. No merge, enqueue, deploy, publish, production access, timeout increases, iteration reductions. Two other jobs edit UrlResolver on other branches in /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/deleg/r18 and /tmp/claude-1000/-code/2f854167-89e3-433d-bb44-9afc59221d1d/scratchpad/deleg/r22 — do not touch them. 90 minutes wall clock; bounded waits; stop every process you start.

The prohibition above supersedes the original outage effort's “fix it, merge and deploy” wording for this branch. Do not merge or deploy this PR without a new instruction.

Mission summary

Two production fixes for the 2026-10-18 TinySite outage, https://github.com/CodexCoder21Organization/UrlResolver/pull/1314 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1315, could not obtain clean remote checks because existing tests failed on unchanged main. This session was assigned one test-cause fix PR with a 90-minute bound. It delivered a reviewed address fix and preserved the failures it could not resolve; it does not claim the complete brief or a clean full suite.

Read the evidence branch's investigation/r23/r23-findings.md and r23-repeats-summary.json for raw counts. XML files include retry histories and full stack traces/console output. The previous r14/r17 findings are preserved there too, so the original evidence does not depend on this disposable filesystem.

What was found and done

1. Fresh gh repo clone of main046396a677e14cf950dc7c3d102cbc2c187b75a1 under the r23 workspace. Read README, Testing Architecture, Engineering Philosophy, prior r14/r17 findings and the linked provisioning-deadline challenge. No root repository AGENTS.md existed. Protocol resolves to0.0.1421 despite older individual test pins.
2. Proven address mechanism: configurePublicAnnounceAddresses(host) read PUBLIC_IP on every resolver host startup, then wrote that host's TCP ports into process-global AnnounceAddressConfig. The next host replaced other hosts' effective addresses and overrode caller explicit IP/full address configuration. PublicAddressDiscovery.setExplicitPublicIp is the public caller seam and normally derives ports from each passed host; full AnnounceAddressConfig lists represent intentional caller overrides. Loopback fixtures must explicitly select loopback, while production wildcard listeners must still advertise PUBLIC_IP when no caller override exists.
3. Three initial real-host regression tests failed0/9 attempts on old product under PUBLIC_IP. The permanent owned-child regression also failed0/6 attempts on old product with parent PUBLIC_IP unset; every attempt reported the second host's port for the first host. Failing states were pushed before the product change. Red commits: https://github.com/CodexCoder21Organization/UrlResolver/commit/3c2900723 and https://github.com/CodexCoder21Organization/UrlResolver/commit/7db2c205e . The latter control was exercised by reverting product implementation locally, then restoring the fix.
4. Product fix91c05787ca5e7e20441bd5beab9c3d9d0cebf586 removes startup global mutation and routes all10 resolver address paths through private package-level effectiveResolverAnnounceAddresses. PUBLIC_IP fallback derives the current host's ports; explicit caller full addresses and explicit IP keep precedence. No dependency changes, no production deploy/publish. README describes public behavior.
5. Four existing test fixtures now save explicit IP, set127.0.0.1, and restore in finally: restart, moves, registry-order and synchronous PEX. All original assertions, time limits and iterations are retained. Whitespace-normalized before/after audits are in evidence; the PR inventories every edit. New owned-child support is compiled only for its test, never into the main artifact. Independent read-only test/code review found no remaining scoped blocker after closing coverage gaps.
6. Restart residual cause is established at the fixture level: it claims ManualClock freezes periodic work, but runPeerExchangeSchedule uses real coroutine delay(1000) then2000 on an independent dispatcher. Console evidence shows the query client reconnecting before the separate readiness probe completes. Thus its cold-client assertion fails before the behavior under test. The current constructor documents clock for time measurements, so changing all maintenance scheduling was not guessed. Internal stop/gate test hooks exist around5801 but were not used because this brief requires public-API tests and prohibits making internals public merely for tests. A supported way to control background lifecycle or a correctly designed public scenario remains needed. Removing early joinNetwork was checked and rejected as an unproven shortcut: queryServices calls ensureNetworkJoined, so joining would occur after injecting the incomplete registry and could repair the state being tested.
7. Moves residual failure: after replacing the registry with the moved endpoint and closing the original, first query sometimes tries the closed configured loopback endpoint and emits an exact-warning assertion failure. Wrong-port registry-order failure without PUBLIC_IP was already observed in prior main evidence. The address fix does not prove those races resolved. Round15 later reproduced registry-order directly: expected loopback35079, actual loopback34329 for the same peer; first attempt failed, second passed. Outbound roots to configured34329 and registry35079 opened25ms apart. Raw console/stack are retained; exact competing caller is not established. This is the same wrong-loopback-port class as prior main evidence. Exact registry/lifecycle event ordering is still unknown; do not dismiss it as scheduling slowness or loosen warning/address assertions.
8. Large-upload recurrence: on two CPUs with PUBLIC_IP unset, round8 failed attempts1/2 then passed3; round9 with PUBLIC_IP set failed1 then passed2. The same-route256-byte control succeeded, the4MiB upload reported ambiguous outcome and EOF/read0of4. Full stacks/console are retained. Current test already caps a genuine socket link to60KiB/s; no artificial CPU load was used. Old main remote failure is established by r17; exact transport closure mechanism remains unknown. No transport/test change was guessed or timeout raised. Read-only source trace: UrlResolver doSendRequest encodes the full frame around31083 and writes encodedRequest to controller.output around31119; NettyOutputStream wraps the full input in one ByteBuf around9865 and uses5000ms backpressure delivery admission. A separate registered completeFrameWriter uses the segmented writer, but this consumer path does not call it. This supports the existing test KDoc candidate; failed consoles did not prove the actual sibling/root retirement initiator. Prove that ordering and root reuse deterministically before changing the write path. If source is upstream, reproduce consumer then upstream and fix the owning library under the global upstream workflow; publishing remains prohibited for this task unless reauthorized.
9. Background-relay round4 first attempt failed cleanup with SocketException: Broken pipe in its socket pump; retry passed. Eight byte-equivalent copies on unchanged main (only names differ) all passed first attempt. This leaves classification UNKNOWN, not branch-only or fixed. Copies were removed after completion; manifest/hash/selector list and raw results are retained. A deterministic real-socket close-order reproducer is needed before changing error handling; do not blanket-ignore SocketException.
10. Response-read and saturation resync tests passed sampled first attempts but their previously observed remote mechanisms remain unproven. Synchronous PEX baseline first failed setup (bootstrap service missing within5s), unlike prior remote average694ms failure. Explicit loopback fixture setup addresses environment dependence only; sampled passes are not proof of every prior timing cause.
11. Full suite: manual remote a97cfbdf (2075cases) progressed through BUILDING to TESTING. Its external1500s client bound interrupted/canceled the run, exit124/sleep interrupted; journal retained97attempts74PASS1FAIL22CANCELED, while canonical pages reset to PENDING/zero completed. Failed journal attempt: testHealthySameTierRelayWinsAfterDelayedSameTierHandshake,46.586s; no stack remained in canonical pages. All5 result pages and full attempt journal are preserved. This is not the linked provisioning-deadline failure signature and not a full verdict. Local fallback in separate exact-head checkout exited137 during support compilation, no test verdict; shell printed Killed. Shared cgroup limit10GiB, memory.events had oom_kill4 when first inspected, no before snapshot to assign a particular kill. No product conclusion follows from that observation. PR's automatic remote full run4ca2fb7d is a separate run; final status below.

PR remote snapshot13:06UTC:1196PASS11FAIL,6RUNNING862PENDING of2075; all5pages and full failure output retained. All new3 address regressions, moves, registry order, syncPEX and public-IP guard were canonical PASSED then, without remote first-attempt proof (attempt endpoint empty). Named failures: restart same readiness assertion; large same4MiB ambiguous/EOF; response-read DIFFERENT exact-count assertion expected2responses/actual1; resync missing8of52services within75s. These last signatures remain unclassified vs main. Seven other failures are listed with full stacks in failures-1307.json; do not silently expand this one-PR brief or assume the address patch caused them.

Final validation snapshot

At13:12UTC, twenty completed sample slots produced176canonical verdicts175PASS1FAIL; first164PASS12FAIL; raw175PASS18FAIL/193attempts. New3regressions each10/10firstPASS with PUBLIC_IPset and10/10unset, as did syncPEX and public-IP guard. Registry order9/10set10/10unset, so below bar. All samples use two CPUs and exact feature head. Interrupted samples are retained and excluded. No extra samples were added after failures.

PR required remote check endedFAILURE. Run4ca2fb7d terminalFAILED1196PASS11FAIL/2075 with868unfinished. Error: Provisioning deadline exceeded for runId=4ca2fb7d: elapsedMs=842056 limitMs=600000 firstAdmissionAt=1792326662986 currentAttempt=13 ... lastFailure=none. This matches the known provisioning class. Final attempt journal inProgress=false/0records; local XML is the first-attempt authority. PR remainsOPEN/draft/BLOCKED/unmerged at91c05787ca5e7e20441bd5beab9c3d9d0cebf586; bld-build and informational kompile-remote-build stillIN_PROGRESS at final query. No full suite completed/clean, no unknown failure claimed fixed.

The scoped address fix is complete and reviewable; the broader named-test/full-suite acceptance is incomplete. Full-suite failure records are evidence, not a reason to rerun CI for debugging.

Relevant PRs and refs

No code was deployed, published, merged or enqueued by this session. Its command/action record was checked; no production host access occurred. No production state was queried, because that is outside this brief. Any existing outage deploy state from other sessions must be reverified by its owner and is not claimed here.

Next steps

1. Re-verify PR/branch/check state first. Fresh clone under your own workspace; read README and instructions. Do not use r18/r22 checkouts.
2. Read retained failure XML and per-test denominators before running anything. Do not count canonical green after a retry as a first-attempt pass. Rebase only when no build/test is active in that checkout.
3. Preserve this single PR for the verified address fix. Confirm its exact final sample table and remote CI outcome. Do not merge/enqueue/deploy/publish under current limits.
4. Resolve restart fixture's mistaken clock assumption with a public supported lifecycle contract and deterministic test; independently trace moves/registry lifecycle ordering. Do not use internal hooks/reflection merely to make the test pass.
5. Trace the upload transport's actual close initiator while its frame is in flight; force that exact condition deterministically, obtain a red consumer/upstream reproducer, then fix its owner. Existing cap is genuine traffic and must stay; no timeout/iteration changes.
6. Classify relay cleanup and prior response-read/resync/PEX signatures against unchanged main; preserve unknowns honestly. Eight relay passes and current stress passes do not refute established failures.
7. Run targeted family and a completed full suite from exact final head in an isolated quiet checkout. The two full attempts here are incomplete. The journal/result-reset discrepancy is documented locally on the evidence branch, not filed via report-challenge because that CLI automatically enqueues/merges and this task forbids both.
8. Re-review tests/code after changes, update the existing PR's complete before/after test inventory, and require10 consecutive first attempts with PUBLIC_IP set and unset for each claimed fixed test. Push every milestone and preserve red/revert controls before proceeding.

Reusable operational knowledge

Use taskset -c0,1 plus JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 (with a space after -c in actual commands). Explicit environment example PUBLIC_IP=203.0.113.7; unset with env -u PUBLIC_IP. Existing public configuration setters must be restored in finally. For replay scripts, change the hard-coded repo path to your fresh workspace and use fresh output paths; never overwrite retained evidence or reuse a previous result file as a new verdict. The child-environment regression controls its own ProcessBuilder env so it works in an ordinary parent process.

r23-verify.py is the original fixed20-round controller. r23-verify-remaining.py resumes after9completed12-selector rounds, excluding the4cases whose first-attempt criterion already failed; it completed round10with8selectors. r23-verify-address-family.py then resumed after10completed rounds with6address/configuration selectors; both interrupted in-progress continuations are preserved and excluded. Response-read/resync local denominator is10 (5per env); the4earlier-failing cases have9samples (5set/4unset). No failed result was removed or rerun until green. Every child has process-group cleanup and a hard bound. Each round fetches/rebases only between completed invocations and refuses any changed HEAD. Final denominator table is authoritative.

Buildtest read-only endpoints: /api/test-attempts?id=<runId> preserves raw attempts; /api/test-results?id=<runId>&page=1&limit=500 returns a paged object (without page args it can return a list); collect5pages for2075cases. /api/test-progress?id=<runId> summary can disagree after cancellation; retain the journal and all canonical rows, do not substitute a zero summary. Remote wrapper interruption cancels the submitted run. The build-watchman page default100rows is not the full-suite total. Watch PR with explicit --check 'kotlin.build (remote)', because its default required gate may be bld-build. Do not use --to-merged under this task.

The final gh pr edit --body-file failed on deprecated Projects classic GraphQL projectCards; patching only the body with structured REST JSON via gh api --method PATCH repos/CodexCoder21Organization/UrlResolver/pulls/1316 --input worked. No new PR or code push was needed.

Fresh depth1 clone may lack origin/main and common ancestry; fetch the explicit main ref and unshallow before attempting rebase. Never rebase a depth1 feature root onto main. No exception to the no-Git-during-tests rule.

Evidence deliberately excludes jars/build outputs/dependency caches/credentials. Full retry XML, failing console output, relevant compiler logs, manifests and final controller scripts are retained. All owned test/build/watch workers were stopped and reaped before finalization. A /proc census found no java/test.bash/timeout/git process left in any r23 checkout. Primary, full-validation, local-full and main-classification checkouts were clean, with no stash/local-only commit; the evidence checkout contains only the pushed snapshots. The manual remote run is CANCELED, PR remote run terminalFAILED; no remote rerun/delete is outstanding. PR automatic checks may continue under GitHub ownership; restart your own bounded watcher if needed.

Final workflow note13:19UTC: The first central create failed before submission during bytecode fetch. The second returned ambiguous createHandoff/read0of4, so it was not repeated. A read-only list found exactly one matching record and get verified the full body matched. This update refreshes the evidence branch table to the final verified remote head and preserves the successful handoff state. Full CLI stderr/verified record snapshots and final status are on that evidence branch; no new create is needed.