Handoff: get the UrlResolver address-fix PR green by identifying the child deadline mechanism

Written 2026-10-19 10:40 UTC. RE-VERIFY: this is a write-time snapshot. Query https://github.com/CodexCoder21Organization/UrlResolver/pull/1316 with gh pr view <full URL> --json state,isDraft,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt; inspect its linked required run and re-check every branch with git ls-remote. Current observations do not become future facts.

Design decisions in force

The user stated these requirements on 2026-10-19; none was relaxed:

Disk rule: run df -h / before any clone or build; stop if under 10 GB free; delete your clone and its ~/.aibuildcaches entries when you finish.

1. Large upload: 2/3 first-attempt passes on the PR versus 3/3 on main. Determine whether the PR causes this. Run matched first attempts on PR head 19b86bb7 and main under identical conditions, amplified (many concurrent runs, -XX:+UseSerialGC, smaller heap) until each side has enough runs for a confident comparison (at least 20 per side). If the PR is worse, find the mechanism, write a failing test, and fix it. If the rates match, record that the failure already exists on main and hand it to a separate fix with its own issue.

2. The remote selection passed 8/17 with unexplained child-process timeouts. CI executors have 2 vCPUs. Reproduce locally with the same selection pinned to 2 CPUs (taskset), and run main the same way. Name the mechanism (what the child process is waiting on when it times out: take thread dumps at the moment of the timeout) before changing anything. Fix the cause; never raise timeouts or cut iterations.

3. Restart test 1/3 on both sides: already on main; file an issue with the evidence if none exists, don't fix it here.

• Never raise timeouts, reduce iterations, weaken or delete assertions, or skip tests. Changing an existing test's expectation needs a written proof and is reported to me rather than pushed.
• Fetch and rebase before pushing; push with --force-with-lease. Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. Keep the PR description's opening why-paragraph; update it via REST PATCH (gh api -X PATCH repos/CodexCoder21Organization/UrlResolver/pulls/1316 -F body=@file) and read it back.
• Do NOT merge, enqueue, publish or deploy. Any wait loop has a hard deadline and you kill it before finishing.
• Write progress to out/r72-findings.md as OBSERVED/INFERENCE lines; checkpoint every 15 minutes; write the mechanism down before writing any fix. About 3 hours.

The original acceptance criteria and exact wording live in the user's brief. These requirements do not authorize weaker tests or a resource knob in place of a cause. No downstream workaround for an upstream defect, no reflection, no mocks, and public-API end-to-end regressions remain required by the global instructions. Read current global/project instructions before continuing.

Mission

The user asked to get https://github.com/CodexCoder21Organization/UrlResolver/pull/1316 green by resolving the prior lane's upload asymmetry and unexplained child-process timeouts. It supports the TinySite outage request, “figure out what went wrong, reproduce it in a test (TDD), fix it, merge and deploy,” but the current brief expressly prohibits merge, queue, publish and deploy. The prior lane proved the address fix and strengthened ownership/cleanup tests; it did not resolve these two classes. Its evidence is https://github.com/CodexCoder21Organization/PlanRepository/pull/11065.

The upload question and restart bookkeeping are answered. The child deadline question and green CI remain incomplete. No product fix was written or pushed by this follow-up. No current test assertion or timeout was altered. Passing constrained runs are not proof that the earlier child deadline disappeared.

Findings and actions

1. The runtime pair is exact. PR source is https://github.com/CodexCoder21Organization/UrlResolver/commit/19b86bb71cde66617de9da3d4a26d994d744069b and main is https://github.com/CodexCoder21Organization/UrlResolver/commit/046396a677e14cf950dc7c3d102cbc2c187b75a1. Main's added test-support build retained all 1330 production classes byte for byte, with no additions or removals. Its copied address tests intentionally fail their assertions and can stop earlier, so their lifetime is not equal lifecycle work.

2. The upload asymmetry is answered as baseline. Forty fresh JVM attempts ran in ten paired waves, with two PR and two main uploads concurrently, alternating launch order, 96 MiB/SerialGC, identical compiled test/dependency hashes, and no retries. Both arms omitted the native SHA agent. PR passed 11/20; main passed 9/20. The original ambiguous-request/read-zero-bytes signature occurred eight times on PR and eleven on main; PR also had one distinct relay-health reset. Exact paired McNemar p=0.6875 gives no evidence of a PR regression; it does not prove equivalence. The unchanged upload test explicitly sets loopback IP and the new address helper delegates to the same method on that path. An independent review checked all 40 attempts and added four fresh main attempts: three passed, one had the same failure. Filed and verified https://github.com/CodexCoder21Organization/UrlResolver/issues/1323. Upload work belongs there, as requested.

3. Restart already has an issue. Added the prior matched 1/3 first-pass and raw 2-pass/4-fail evidence per runtime to https://github.com/CodexCoder21Organization/UrlResolver/issues/1319#issuecomment-6262576430. No restart change was made.

4. The native local runner did not reproduce child deadlines. Three matched exact 17-test selections on physical CPUs 0 and 1 had four native test JVMs observed concurrently. PR first passed 48/51 and canonically passed 51/51, with raw 51 pass / 4 fail. Main first passed 37/51 and canonically passed 41/51, with raw 41 pass / 36 fail. Nine main canonical failures are the three intended address controls in three invocations. Among 42 first attempts of old tests, PR passed 39 and main passed 37. All nine PR address-child first attempts passed. No child 20-second deadline occurred in any arm. Canonical results include native automatic retries. The same selection under one actual CPU gave PR first 10/17, canonical 13/17; main first 8/17, canonical 13/17, again without a child deadline. The existing synchronous service-discovery timing assertion first failed on both revisions (PR 842 ms / main 807 ms against 500 ms), proving the baseline symptom in that constrained profile, not its internal cause.

5. Direct unchanged child scenarios reproduced deadlines only under stronger conditions. Each row below is 60 fresh attempts, cycling the three unchanged compiled child mains 20 times each. The original 20-second child verdict, 30-second outer verdict, assertions and cleanup checks remain. No retries were used. Main assertion failures are expected controls, not passes. Outer-timeout outcomes can hide an earlier child-timeout marker; count the distinct markers.

The first recorder buffered stdout until cleanup and lost some SIGQUIT output when the outer controller stopped it. The corrected recorder flushes each line and records Linux scheduler counters at the first timeout marker. The first batch is retained separately. An independent read-only review recounted the first nine batches from raw logs/events, checked arguments and affinities, and inspected full dumps and scheduler records; root reviewed its findings. The last one-CPU pair was recounted separately. All 11 batches and 660 one-shot scenarios are retained.

6. CPU queueing is proven in the amplified local profile; the earlier remote cause is not. Representative corrected 16-child captures show main threads using about 1.4 seconds of CPU and spending about 21 seconds runnable in the Linux run queue at roughly 24 seconds elapsed. Full SIGQUIT dumps show different phases: class loading, Netty startup, and host-stop futures, rather than a common blocked lock. The complete dumps are in child-output.log; several jcmd files contain only a PID and are not complete dumps. Some captures reach DestroyJavaVM, but their remaining global executor and returned main are less than one second old; those samples do not establish a long-lived executor leak. No OOM-kill counter increase was observed during these child studies. The C1-only pair demonstrates a CPU-cost effect in this profile, not a proved remote fix or a hardware-independent regression. No flag or lifecycle fix was shipped.

7. The final local comparison uses the actual published remote runner. Retained full-run logs identify BuildTestRunner 0.0.124, whose published POM pins manager 0.0.205; the local native CLI 0.0.124 pins manager 0.0.207. A separately declared pair runs the same 17 tests through published runner 0.0.124 on actual CPUs 0 and 1, with a four-test cap and 768 MiB / two-reported-CPU parent/build-script profile. Each whole invocation has a 600-second bound, including preparation. It submits no remote build or reporting service. Its test JVMs use CompileThresholdScaling=5, while the earlier native launcher used TieredStopAtLevel=1. Both leave the unchanged address app-child at default compiler settings. The PR sample completed first/canonical 13/17, with one attempt each and no child deadline. All three address children passed. Its four failures were upload relay health, slow bootstrap, ping adoption and the synchronous service-discovery timing assertion (795 ms against 500 ms). The full-address wrapper reported 20.258 seconds including parent/startup overhead, with process-tree CPU 3.640 seconds and combined thread run-queue wait 32.733 seconds; this is not a child 20-second failure. The matched main sample is active at this snapshot. See the saved plan, commands, event logs and outcome files for exact limits and final state.

8. A scheduler annotation or blanket JVM policy is not a demonstrated repair. Pinned local manager 0.0.207 lacks workerSlots handling; current core and published 0.0.209 support it. But https://github.com/CodexCoder21Organization/kompile-core/pull/390 explicitly distinguishes remote BuildTestRunner, where the annotation reserves nothing since https://github.com/CodexCoder21Organization/BuildTestRunner/pull/92. The prepared public-API local scheduler-contract probe was compiled but NEVER EXECUTED: no old-fail/new-pass result exists, and it cannot establish the remote path. The saved historical JVM-policy handoff records that scaling 5 was already adopted and every measured historical policy still had failures. Read it before generalizing from one C1 experiment.

9. The original remote evidence has a real limit. The raw log for https://buildtest.kotlin.build/run?id=6e657f49 expired at 09:11:04 UTC under runs-pool retention. The prior evidence retains all result pages and two-CPU/four-worker metadata, but no live child dump. Its runner version remains unknown; the later full-run version does not establish it. The missing state cannot be reconstructed as a fact.

10. CI is still red. Required full run https://buildtest.kotlin.build/run?id=8d419c5b has all 2075 unique results: 2073 pass and two fail. Failures are the upload relay-health Connection reset and synchronous service discovery averaging 573 ms against unchanged 500 ms. All three address children canonically passed; first-attempt history was not established from result JSON. The separate informational kompile route failed provisioning before tests: run 5a51301f reached its 600000 ms deadline with 501 unfinished creation records against a 500-record limit and no created droplets. Its 2075 synthetic failures are not executed-test results. No CI rerun was requested.

Relevant refs and partial work

All useful text/source/logs are on the evidence branch below. Jars, classes, dependency caches, and unrelated dashboard HTML are deliberately omitted; their manifests and source/build recipes are retained. Absolute scratch paths in logs are provenance. They must be regenerated for a fresh task, not assumed runnable.

Deployed/published but not merged: none introduced by this session. Its action history contains no artifact publication, jar upload, production restart/deploy, PR merge or queue request; only research runs, evidence-branch pushes, the upload issue, restart issue comment, and local diagnostic work. The description update is pending readback at this snapshot. Existing deployments from the wider outage effort were not queried or changed; no claim is made about their current code.

Next steps

1. Re-verify current PR/head/main/checks and tracked work first. Read current global instructions, repo README/AGENTS, Testing Architecture and Philosophy; read investigations/r72/r72-evidence-README.md and latest findings. The original 20/30s limits and exact assertions remain fixed requirements.
2. Recover the saved scripts from the evidence branch and regenerate only the omitted binary classpaths through the documented source/build path. Keep all fresh checkouts under workspace; check disk before every clone/build; record versions/hashes. Do not rerun expired CI unchanged or turn canonical results into fresh attempts.
3. Establish why the actual four-test remote child deadline differs from the passing matched local profile. Capture child dumps and process-tree CPU/run-queue state at the original false 20-second verdict. The amplified 16-child observations are leads, not a substitute. Any baseline upload work remains separate in its issue; restart remains separate.
4. Once the mechanism is named end to end, write it down and add a deterministic public-API failing regression that discriminates on the contract rather than the machine. If it belongs upstream, reproduce/fix there, then verify downstream. Never replace that work with a timeout increase, test relaxation or speculative JVM/scheduler knob. Existing-expectation changes require written proof reported to the user, not pushed.
5. Independently review test coverage and implementation; fetch/rebase, run required tests, OPEN-check before force-with-lease; append the required commit trailer; preserve the exact opening why paragraph when REST-updating the description and read it back. Obtain green CI from a supported code change. Leave the PR unmerged and undeployed under this brief.

Operational details

The exact 17-test selector and native result parser are under prior-tools/; commands/manifests capture all argv, child affinities and environment clearing. r72-evidence-README.md explains regeneration and every evidence family. Root's transient diagnostic patches never reached the product PR and contain local absolute paths: use them only as diagnostic references.

The system /usr/local/bin/coursier has the wrong architecture here; ~/bin/cs works. rg is unavailable, so use scoped grep/find. The shared cgroup has 10 GiB memory; foreign jobs can affect its OOM counters, so no individual test was blamed from the aggregate counter alone. Disk remained above the requested 10 GB floor.

build-watchman0.0.21 ended IDLE_TIMEOUT after required CI had completed; its legacy page exposed100 records while explicit pagination found2075. See watchman-challenge.md. The report-challenge CLI auto-merges, prohibited by this brief; reviewable challenge notes were saved without that mutation. HandoffCLI0.0.8 health succeeded in this pass; central creation does not require a PR, merge or enqueue. The older git README's automatic-merge advice is superseded by the current skill and user prohibition.