Handoff: UrlResolver active-query TTL flake blocked at reproduction gate

Written 2026-07-14 UTC.

RE-VERIFY: This is a write-time snapshot. Before acting, fetch CodexCoder21Organization/UrlResolver, compare origin/main with commit 133fab3e14f8c3d310020f5dfcc6b4dc8f388c86, and reopen the three linked BuildTest runs. A fresh repository/build query always wins over this document.

Mission summary

Fix the flaky main test testActiveQueryDiscoveredServiceIsEvictableByTtlCleanup, which failed identically in three CI runs when a direct on-demand lookup returned no service. The required workflow was reproduce-first: reach at least a 50% failure rate over at least ten attempts under an isolated, CPU-contended profile; determine from that reproducer whether the test races provider readiness or the product active-query path fails under scheduler pressure; then fix only the owning layer, preserve discovery plus TTL eviction, verify the requested stress/unconstrained/sibling gates, create a PR, and watch it through the pre-authorized merge.

The effort stopped before any fix because the mandatory reproduction threshold could not be reached. Guessing at either a test readiness gate or a product change would violate the task's explicit rules.

What was found and done (the chain)

1. A fresh UrlResolver checkout was created at /code/fix-activequery/workspace/UrlResolver on local branch fix/active-query-ttl-eviction-under-load, based on origin/main. The repository README and the authoritative testing architecture were read before test work.
2. The three CI records were rechecked at handoff time. Each still contains the identical assertion text: Active query for 'active.query.ttl.service' should return the service from the provider, but got: [].
3. The target test uses real UrlProtocol2 provider/client nodes. It registers a provider-local service, manually seeds the provider into the client's peer registry with advertisedServices=emptyList(), calls client.queryServices(serviceId) once, then verifies the active-query cache entry is later removed by TTL cleanup. No timeout, deadline, budget, assertion, mock, or reflection was changed.
4. The first isolated cold runs exposed tooling rather than product behavior. Setting only HOME did not isolate Coursier because Java continued to derive cache/config paths from user.home; after adding COURSIER_CACHE and -Duser.home, the isolated home also lacked /home/leslie/.config/coursier/mirror.properties, causing direct unauthenticated requests to kotlin.directory and long metadata stalls. This was recorded as a challenge: exporting an isolated home does not isolate Coursier.
5. A temporary end-to-end harness repeated the unchanged scenario with fresh real provider/client nodes. It was removed afterward; the UrlResolver checkout is clean. Measured results:
   • two cores, one equal-priority spinner per core, ten sequential scenarios: 10/10 pass;
   • one core, eight equal-priority spinners, ten concurrent independent scenarios: 10/10 pass;
   • one core, 48 equal-priority spinners, ten concurrent independent scenarios: 10/10 pass;
   • one core, 192 equal-priority spinners, ten concurrent independent scenarios: 10/10 pass.
6. The amplified batches used a temporary filesystem handshake at test entry. Workspace compilation ran unconstrained; when the test created ready, an external controller pinned every JVM thread, started the requested spinners, and created release. This makes the test-load boundary causal and preserves the runner's unchanged 30-second limit.
7. Non-measured/tooling executions were kept separate: one isolated unconstrained warmup pass; one unconstrained ten-scenario pass when an early stack-sampling amplifier missed test entry; interrupted cache-hydration attempts before any test execution; and one gate-only 30-second timeout caused by trying to use child stdin (no scenario began).
8. The verified measured baseline is therefore 0 failures in 40 scenarios (0%), not the required >=50%. This evidence does not distinguish the two live hypotheses: a provider/announcement readiness race in the test versus a product active-query failure/short-circuit under pressure. There is no verified root cause yet.
9. No UrlResolver source/test change remains, no commit was made, the branch was not pushed, no UrlResolver PR exists, and no build-watchman was armed.

Relevant PRs / refs

| Reference | Write-time state | | --- | --- | | BuildTest run c02de0a6 | Re-verified 2026-07-14: identical active-query empty-result assertion present. | | BuildTest run ae34b898 | Re-verified 2026-07-14: identical active-query empty-result assertion present. | | BuildTest run 11dbc261 | Re-verified 2026-07-14: identical active-query empty-result assertion present. | | CodexCoder21Organization/UrlResolver at 133fab3e | Local branch and origin/main matched this commit; worktree clean; no PR for fix/active-query-ttl-eviction-under-load. | | Coursier isolation challenge | Merged record of the tooling issue encountered before reproduction. | | /code/fix-activequery/PROGRESS.md | Local milestone log with exact measured/tooling separation and STUCK status. |

Next steps

1. Re-verify current state first: fetch/rebase fresh UrlResolver origin/main, reopen all three BuildTest pages, and confirm whether a newer failure contains additional diagnostics or a newer commit changed the target/product path.
2. Do not edit the test or product until a different reproducer reaches >=50% over >=10 attempts. CPU starvation up to one core plus 192 spinners was insufficient.
3. Prefer an experiment that observes the active-query stages through public diagnostics: provider protocol readiness, stream negotiation, query delivery, provider response generation, response read, and the currently swallowed exception path. The aim is an end-to-end mechanism, not a timeout/fallback/retry mitigation.
4. Reproduce the exact CI runner topology/runtime if available. Compare its architecture, JVM, cgroup quota, and concurrent build load with this host; CPU spinners alone did not reproduce the CI state.
5. Once a reliable failure exists, decide with evidence:
   • if the query is issued before a causally observable provider-ready/announcement event, gate readiness in the test while preserving one strict active query and later TTL eviction;
   • if a ready provider receives/answers the query but the client returns empty or short-circuits under scheduler pressure, fix the product path and keep the test strict.
6. Re-run the original required verification: at least 10/10 under the reproducing profile, 3/3 unconstrained, and the nearest active-query/TTL siblings. Review test comprehensiveness and perform adversarial review.
7. Only after those gates pass: rebase immediately before the build/PR, run the complete unit suite, commit/push, create the evidence-backed UrlResolver PR, and arm exactly one build-watchman 0.0.12 --to-merged as originally authorized.

Reusable / operational knowledge

• The isolated environment used here was:

  export HOME=/code/fix-activequery/home
  export COURSIER_CACHE=/code/fix-activequery/home/.cache/coursier
  export JAVA_TOOL_OPTIONS='-Duser.home=/code/fix-activequery/home'
  

  The isolated home must also contain the Coursier mirror configuration from /home/leslie/.config/coursier/; do not print its contents or clear shared caches.
• scripts/test.bash --test <name> still resolves/compiles the workspace's roughly 1,102 test scripts before executing the selected test. On this shared host that cost 10–18 minutes per test-file edit even with warm artifacts. A temporary multi-attempt harness was used so measured scenarios, rather than compilation, dominated the experiment.
• The test runner captures stdout and does not show per-attempt prints when the selected test passes. Overall harness success was therefore used to establish all-attempt pass batches; any collected assertion caused the wrapper to fail.
• The runner enforces an unchanged 30-second selected-test limit. A causal filesystem gate can arm contention immediately at test entry without consuming that limit materially; a stdin barrier cannot be released because the build runner does not expose child stdin.
• The temporary harness and gate directory were removed. Verify with git status --short before starting; it was empty at handoff time.