Handoff: Decide the UrlResolver fixture scope, then finish the response-read flake fix

Written 2026-10-22 04:23 UTC.

RE-VERIFY: This is a write-time snapshot. Check git ls-remote for the branch and main, and gh pr list --repo CodexCoder21Organization/UrlResolver --head fix/stresstestactivequeryreportsresponsereadfailure-flake --state all --json url,state before acting. GitHub polling must be no more frequent than once every three minutes. The branch has intentionally failing diagnostic tests, no product fix and no PR.

Design decisions in force

The user's 2026-10-22 instructions include:

• "Fix a flaky UrlResolver test at its root cause, following the fix-flakey-test skill (read ~/.claude/skills/fix-flakey-test/SKILL.md first and follow it)."
• "work on a new branch fix/stresstestactivequeryreportsresponsereadfailure-flake."
• "Fix the product code (or, if the existing test itself is wrong, STOP and write the reason in findings as a question — do not edit an existing test on main)."
• "no raised timeouts, no lowered iterations, no weakened assertions, no edits to tests that exist on origin/main, no mocks, no reflection, no sleep-based synchronization."
• "run the original test and its neighbours one JVM at a time with scripts/test.bash --local --test <file> (never the full local suite; the host is shared and under heavy CPU pressure, so never use elapsed time as evidence)."
• "Do not enqueue or merge — the supervisor owns merging and may enqueue while you work; leave queue entries and watchers you did not start alone. No production actions."
• PR description must begin with the reason, link the observed CI run, and end with "🤖 Generated with Claude Code". Commits end with "Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com".
• Any eventual PR required check is watched with /home/u/bin/cs launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlResolver --pr <n> in plain mode. List every first-attempt failure from its attempt records. Do not rerun CI to diagnose or hide a failure.

Mission and current decision

Investigate foundation.url.resolver.stressTestActiveQueryReportsResponseReadFailure, which failed first attempt in https://github.com/CodexCoder21Organization/UrlResolver/actions/runs/39704423232 on https://github.com/CodexCoder21Organization/UrlResolver/pull/1357 (head 185c0d24), then passed its retry. The assertion at original test line 239 expected two received requests but observed one. The reviewer ruled out that PR's startup changes on the executed path.

Question pending with the user: May the existing test fixture be revised to establish successful stream setup before injecting its single read fault, or must the test remain unchanged while tracing the opening deadline further? An asynchronous question was submitted and the same question is in the findings. No answer has been received. Do not infer approval from elapsed time.

The reported count failure has been captured in an unchanged-workload diagnostic copy. It contains an already-reported opening timeout before the injected read fault. The product walks targets, not successful received requests. This raises a fixture-precondition question rather than justifying an extra product retry. The lower-layer cause of the natural opening expiry is not isolated or fixed. Do not silently convert the task into a retry-budget change or claim the entire flake fixed.

Findings and work completed

1. Read the requested fix-flakey-test skill, repository README (no root AGENTS.md exists), the original test, the reviewer notes, live TESTING.md, and relevant product paths. Wrote the invariant table and mechanism before product changes; no product changes followed.
2. Downloaded the original test-attempt artifact with gh. Its only first-attempt failures were the requested test and foundation.url.resolver.testObservablePushInvalidationReplaysAfterProviderClose; both passed attempt two. The original count assertion omits its collected warnings, so the original CI artifact does not identify the preceding query-attempt phases.
3. In UrlResolver.kt, activelyQueryForService selects streamTargets = listOf(null) + addresses for a reusable parent, otherwise only addresses (at validated base 9f282dfc67, around line 18558). The reused stream's opening wait is 2000 ms; reading is 3000 ms. A failed opening is reported and consumes that target; the loop moves to the next target. It does not promise two successfully received requests. The existing adjacent test testActiveQueryRetriesAfterStreamOpenFailureAndReportsIt.kts explicitly pins a controlled opening timeout and continuation to the next address.
4. An initial diagnostic copy preserved the original 64 scenarios / 16 workers and added connection snapshots and all warnings to its count assertion. The unchanged original passed once locally; a separate clean diagnostic passed once. In a ten-command diagnostic batch, first attempts 1–8 passed, 9 and 10 failed earlier at ping warm-up, and run 10 attempt two also failed warm-up. The CLI retried automatically and eventually displayed green. All attempts were audited, not treated as ten clean passes.
5. The diagnostic then logged every scenario's failure before rethrowing. The test's in-order future joins can otherwise report an earlier warm-up failure while hiding a later completed count failure. Direct invocation of the already-built test body without automatic retries captured the requested count failure in run 3 / iteration 7. Its parent remained physically open in both snapshots. Exact warnings: opening on the reused target -> java.util.concurrent.TimeoutException; reading on the configured address -> ExecutionException / EOF, 36 of 73 bytes. That last target received the first actual ServiceQuery, so the responder injected its first partial body there. No second complete reply was generated or lost.
6. An early real-child-reset experiment reached count 1 in 3/3 attempts, but it added a different setup fault. It was explicitly rejected as proof of the original cause. A subsequent deadline probe copied ping warm-up and reached the target in only 3/10 trials; three failed ping warm-up and four reached the direct launcher's cap. These results remain recorded; no rate claim is based on selecting the successes.
7. The final probe stressTestActiveQueryReadFaultAfterExpiredWarmAttempt.kts uses a completed real responder-to-client RPC negotiation as its setup barrier. It drains gossip, freezes unrelated periodic work on an injected ManualClock, and disables relay work only in the new probe. A public raw-Host stream visitor withholds negotiation input on exactly one armed child and releases each Netty message. The existing query-open policy expires that child. The next child negotiates normally and receives the first partial-body fault. Before the final count assertion, the probe asserts the exact two phases, the opening exception type/full detail, and the full partial-read EOF detail.
8. That final body reached expected 2 / actual 1 in 10/10 fresh-node invocations. Calls were serial in one JVM because they own process-wide address configuration; every call creates and closes fresh real nodes/workers. The direct launcher uses the same 128 MiB heap and 120-second cap per invocation, calls the public test function directly without reflection or automatic retry, and stops on every terminal outcome. The temporary .kt body was verified byte-equal to the .kts after removing only Kompile/runner annotations and their imports. These are red evidence tests, not a final regression suite to merge.
9. Inspected exact upstream sources without changes: UrlProtocol 0956eaf75 publishes pinned 0.0.1429; jvm-libp2p 0f8adf04 is snapshot-32. Read their READMEs, FORK.md, NettyStreamController, NetworkImpl, HostImpl and connection construction. Duplicate physical-close records alone do not prove loss: NetworkImpl closes surplus settled dials. No upstream fix, publish or dependency bump was attempted.
10. Independent interpretation/test passes confirmed the probe's exact guards prevent a different opening/read failure from reaching the count verdict; buffers and resources are owned/closed. They also confirmed the limitation: the count-consumption mechanism is proven, while the natural opening expiry's lower-layer cause and the original CI's omitted warning sequence remain unproven. Existing main tests, product sources, scripts, build.kts and README are unchanged.

Relevant refs and partial work

Verified at 04:22 UTC: local HEAD equals the remote branch head below; root worktree and stashes are clean. Main is 9f282dfc67af6e05aa9e2fbb358fdeb4ff67231f. gh pr list for this branch returned no PR. Both inspected upstream clones are clean detached checkouts with no stashes or modifications.

Durable evidence at that head:

• Findings and full chronological reasoning
• Natural same-assertion failure with all warnings
• Deterministic 10/10 attempt records
• Full deterministic failures
• Probe source

No production deploy, restart, artifact publication, CI rerun, enqueue or merge command was run. The only Handoff-service action before this document was its read-only health command, which returned OK. No code is deployed or published from this branch.

Next steps

1. Re-verify branch/main/PR state first. Read the durable findings and exact warning evidence, plus current testing guidance. Treat the deliberately red test as a diagnostic, not something to merge unchanged.
2. Resolve the pending fixture question with the user. Existing-main tests must remain untouched until the instruction changes. If the user keeps the fixture unchanged, trace why the reused open exceeds its policy; do not add a retry or extend a timeout. If a lower-layer defect is identified, reproduce and fix it at that owning layer rather than masking it here.
3. If fixture changes are authorized, design a deterministic successful-setup precondition while preserving the original assertions, scenario counts and timeout values. The current red probe demonstrates why that precondition matters; it does not itself supply the finished fixture correction. Remove/rework the intentionally inconsistent negative probe before a final PR.
4. For any actual fix, demonstrate the red/green flip reliably; run the original test and its neighbours individually with the user's exact scripts/test.bash --local --test <file> command, one JVM at a time. Never run the full local suite. Review tests and product changes independently.
5. Fetch/rebase, test the rebased head, create the requested PR with provenance/footer, and watch its required check with build-watchman 0.0.21. Audit every first-attempt failure from that run. Do not enqueue or merge.

Operational notes

• The Kompile CLI 0.0.124 hardcodes three local test attempts and exposes no retry-control flag. Read the attempt XML, not just its green summary. No retries were added to tests.
• Kompile recursively scans build scripts under the workspace. An upstream inspection clone nested under the resolver root was detected; moving it after discovery caused one preparation-only FileNotFoundException. No test body ran in that launch. Inspection clones were moved outside the resolver root. Do not treat that preparation error as a flake or rerun CI for it.
• The direct diagnostic launchers are preserved as Java source under investigations/evidence. Compiled jars/classes, dependency caches, temporary classpaths and empty/failed thread-dump files were intentionally excluded as build outputs or unusable diagnostics. The useful natural and controlled failure logs and records are remote.
• Original local findings path: /tmp/claude-1000/-code/25ce0ba7-ba4c-44e2-8aa3-c58875bcce27/scratchpad/out/FL9-findings.md; the remote copy above is durable.
• All owned test batches and operational waits exited before stopping. No existing supervisor watcher or queue entry was touched. Recheck process state if resuming on the same host.