Handoff: Restore attempt evidence, deliver the shared-cache fix, and finish UrlResolver first-attempt repairs

RE-VERIFY — 2026-10-04T20:08Z snapshot. This is a write-time snapshot. Before acting, use handoff-cli get/claims, gh pr view <full URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, git ls-remote, current Maven metadata/POMs, and BuildTest run/result/attempt endpoints. The WUI run projection currently reports outOfDate=true and refreshFailing=true, so it is not authoritative when it contradicts terminal run evidence. A green final result with automatic retries is not proof of clean first attempts.

Mission

Original request: explain why UrlResolver required remote runs after approximately 20:04 UTC on 2026-09-30 failed 19–40 tests, then later stopped with zero tests, and fix the causes at their owning layer. The original evidence compared unchanged UrlResolver commit c2564bca across clean run c748ca69 and failing run 3070c51e. This lane triaged the existing work; all merges, queue actions, releases, deployments, and handoff completion are reserved to the supervisor.

What was found and done

1. Restart dispatch source work is already on main. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1177 merged as https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/8142487a4245a1487c78fab84cbbba094dda965d. Current main verified at 13ae9f5c6d78f2b0bb57e7160629f716fdb2b012. Recovery previously treated a retained shard without a durable queue as a static owner even if no runner ever executed, and missing shards became static if no queue survived. Current ownership excludes only persisted work held by actual runner execution evidence, while retained no-runner shards and missing shards consume the rebuilt dynamic queue. Before-start, after-start, and ownership-matrix tests exercise real SSH, persisted records, event spools and public terminal results. The merged PR records red-first proof and a 27-test passing neighborhood. Those counts are historical; the resumed lane did not complete a fresh test execution. Do not repeat this fix.

2. The shared-cache root correction is merged upstream, but is not delivered by the default runner. https://github.com/CodexCoder21Organization/kompile-executionenvironment/pull/74 merged as https://github.com/CodexCoder21Organization/kompile-executionenvironment/commit/a326e79811baa1125900112c4c791a1eb42cbaae. Its production evidence is four MISS/four Stored records for one key: concurrent children each rebuilt UrlResolver because the parent did not claim an in-flight cache key. Real child-JVM tests demonstrated four computations before and one after; tests cover failed/dead owners, dead waiters, independent keys, cycles, nested children and uncacheable values. Memory admission also landed in https://github.com/CodexCoder21Organization/BuildTestRunner/pull/140 and preparation heap sizing in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1179. These are separate changes, not a measured license to call an overloaded host the root cause.

3. Delivery still requires a reviewed runner release and coordinator identity update. BuildDriver.kt default and PIPELINE tuples pin published runner 0.0.122 / manager 0.0.198. Fresh Maven metadata still names runner 0.0.122 as latest/release, and runner/manager POMs resolve executionenvironment 0.0.38. Prior direct JAR inspection found the cache coordination in 0.0.44, not 0.0.38/0.0.39. Existing https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145 adopts CLI 0.0.121, manager 0.0.205 and executionenvironment 0.0.44. It publishes no runner. Its required remote check for run 0cd86dc8 passed 313/313 final results at cf8a4058d6d14787cec6ece0607b07f6d8d1e874, but explicitly warns that seven tests passed only after retry. /api/test-attempts?id=0cd86dc8 returned READ_FAILED with no rows and stated that the coordinator did not return a complete response. Historical focused verification was 7/7; the prior full fallback had 312 first-attempt passes and one CPU-accounting failure followed by a second-attempt pass. Do not call this a clean suite. A coordinator-only deployment cannot deliver the dependency fix.

4. The original fifty-test repair corpus is incomplete, and existing work must be salvaged. Six complete old static runs plus a partial seventh yielded 120 distinct failed names, retained on the prior evidence branch. That is a different set from the original unnamed fifty first-attempt failures in nominally clean runs. A mention scan of 53 UrlResolver PRs found 30 of the 120 names; it is an index, not proof of fixes. Historic raw logs now return 404. Historic c748ca69 attempt endpoint first returned LOADING, then READ with zero rows; it cannot currently recover those names. Newer attempt endpoints return LOADING or READ_FAILED. The named examples have useful existing work: clock-driven deadline proofs in https://github.com/CodexCoder21Organization/UrlResolver/pull/1171, polling readiness in https://github.com/CodexCoder21Organization/UrlResolver/pull/1162 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1183, and close-during-preparation ownership in https://github.com/CodexCoder21Organization/UrlResolver/pull/1184. Preserve their tests. The combined https://github.com/CodexCoder21Organization/UrlResolver/pull/1167 overlaps those component fixes. No PR was closed or modified by this lane.

5. Current failures are a mixture of service failures and executed assertions. https://github.com/CodexCoder21Organization/UrlResolver/pull/1183 required run 1f23c644 stopped during createDropletAsync provisioning: 1888 discovered, zero executed, zero failed. https://github.com/CodexCoder21Organization/UrlResolver/pull/1186 required run 067e8783 recorded 1887 passes and four failures out of 1891. Its check says terminalization found already-terminal persisted evidence after 196980ms held in TESTING, and attempt-history enrichment failed after 30 seconds. The stale run-list still calls it BUILDING with zero results. Result pages 1/2 expose three failures: stressTestActiveQueryMalformedResponseAlwaysReachesParsing (zero of sixteen streams opened), testActiveQueryReportsRequestWriteFailureAndRetries, and testActiveQueryReportsResponseReadFailure. The endpoint reports only 1567 rows with complete=false, and pages 3/4 return cached-view-not-ready errors. The fourth name remains unverified. Therefore neither a universal test defect nor a universal service-only explanation survives evidence. The separate historical Mode C run d8ed7de1 remains uninvestigated; do not present it as resolved.

6. Verification and reports are blocked by unavailable detail and a constrained local host. Current run projection flags include outOfDate=true, refreshFailing=true, runMembershipReconciled=false and upToDateAsOfEpochMs=1791101091862. Health reports projection reset required and getTestResultsPaginated relay-response failure. No local Fabric certificate directory or SSH identity files are present, so direct host journal recovery is unavailable from this lane. Host memory repeatedly exceeded the brief's 4.5 GiB launch threshold; no local build/full suite was run. The previous invocation's remote command failed while materializing its 259 MiB checkout into a client ByteArrayOutputStream before a run ID existed. The resumed lane prepared a source-only export, then stopped two client startup invocations (wrong working directory, then above-threshold memory) before health/submission; neither yielded a run ID or verdict. No further tests were submitted. This is a verification limitation, not a diagnosis that the test code is too demanding. The RUNNING report upload timed out after 120 seconds with a closed-channel warning; authoritative claim recheck/update results are recorded in the evidence branch. No workaround was added to product code.

Relevant PRs / refs

Merged source:

Evidence branches (do not merge):

No code was deployed or artifact published by either invocation of this lane. The live coordinator artifact was not established here; no assertion is made that the merged restart source is deployed. The current published runner and current main release tuple are verified separately above. Historical Mode C is unresolved.

Next steps

1. Re-verify all live states and claims first. This lane's original claim/get worked, but later report upload timed out and a capped claims read returned an empty list. It reclaims only after that no-owner response. Do not assume board state from the stale WUI.
2. Restore readable attempt histories and complete result pages, or obtain the durable coordinator journals/logs through an operator with Fabric credentials. For runner run 0cd86dc8, identify the seven first-attempt failures; for bundle run 067e8783, recover all four failures and full attempt records. Existing evidence explicitly refutes both a universal service-only failure and a universal assertion-only failure. Do not request blind CI reruns.
3. Once a test host is available within the memory budget, run the merged before-start, after-start and ownership matrix regressions. Use a source-only git archive outside .git and downloaded launcher directories for remote packaging; fetch/rebase first. At most two remote submissions were permitted; none reached a new run in this lane, but the earlier packaging attempt and interrupted client starts are recorded honestly. Do not increase heap, timeouts, or weaken tests to compensate.
4. Supervisor: review the existing runner adoption PR and its seven first-attempt failures. After verified repair and final review, decide its merge and authorize a runner release separately. Record the served runner artifact hash, complete dependency closure digest, manager/buildscript versions and both coordinator release tuples in a new main-targeting BuildTestEmbedded PR. No guessed version/hash may replace artifact evidence. A separately authorized rollout and real preparation measurement are still needed to establish delivery.
5. Supervisor: select existing UrlResolver component/bundle PRs for salvage. Keep the stronger deterministic deadline tests and host-preparation-close proof. Polling PRs overlap; compare content before choosing, not merely final check colors. Existing silent-stream identity proof in https://github.com/CodexCoder21Organization/UrlResolver/pull/1178 is worth keeping. Recover the original fifty first-attempt identities from durable evidence/backups; if they no longer exist, make an explicit scope decision about using the recovered 120 static failures and current first-attempt failures. Do not equate those corpora silently.
6. Related BTE cache PRs in the table have distinct pending work and were not proved obsolete. Keep them with their existing owners. No PR or branch closure is recommended on current evidence. After every required repair, perform independent test/code review and full remote landing gates from exact heads. The supervisor owns every merge, publication, deploy, and eventual handoff completion.

Operational knowledge

• Read PlanRepository handoffs README, especially Triaging, Format and Partial Work; PHILOSOPHY Root Cause/Upstream Fixes/Refuted Hypothesis; TESTING Public API, Crash Recovery, Flaky Policy and Never Calibrate; CODE_REVIEW Claims That a Host Is Overloaded/Unmeasured Performance/Weakening Tests. Source has no root AGENTS.md. All source/test review here is distinguished from historical or unexecuted gates in findings.
• Correct WUI routes are /api/test-results?id=<run>&page=N and /api/test-attempts?id=<run>. /api/test-progress summarizes final rows, not first attempts; /api/runs does not filter by id. HTTP 200 with LOADING/READ_FAILED proves no successful history read. Even READ with an empty historic journal cannot establish original first-attempt reliability.
• Use PATH=/code/ws/bin:$PATH. The working Coursier entry is cs or java -jar /code/ws/bin/coursier.jar. Current BTE scripts already bootstrap a pinned verified JVM Coursier. Bound CLI JVM heap on this shared host; preserve the same test settings. Read cgroup memory immediately before launch and use a conditional guard, not merely a printed value.
• Plain build-watchman 0.0.21 watches required checks and exited GREEN for runner adoption; it does not authorize enqueue. Tracked timeouts only; no handwritten condition loops. Do not treat 503s alone as worker death, and do not delete/rerequest a run without the owning operator's verified ground truth.
• The Handoff report was recorded to disk but timed out in the service. The refreshed body and update receipt/error are retained on the evidence branch. Local Fabric certificates and SSH keys are absent; do not guess credentials or manage production through alternate unverified paths.
• Report-challenge auto-merges its record, forbidden in this lane. Supervisor should record: workspace packaging materializes the whole checkout; attempt-history RPCs fail/return incomplete views; handoff claim/report/read state changed within the claim heartbeat period; and shared disk exhaustion killed the previous lane. These are concrete tool/service issues, not explanations that a host is inherently overloaded.

Write-time terminal state: BLOCKED_INFRA. Both review passes and all retained evidence are in investigations/L39-resumed on the re-verification evidence branch. The supervisor must decide merges, release, rollout, and eventual completion. Current source-only proof and historical counts must not be reported as a fresh clean suite.