← Handoffs

Repository · handoffs

Handoff: Restore attempt evidence, deliver the shared-cache fix, and finish UrlResolver first-attempt repairs

View on GitHub ↗

id: hf-2026-09-30-buildtest-restart-recovery-drops-pre-testing-runs-to-static-dispatch-shared-cache-prep-over-deadline-urlresolver-remote-red url: url://handoff/handoffs/hf-2026-09-30-buildtest-restart-recovery-drops-pre-testing-runs-to-static-dispatch-shared-cache-prep-over-deadline-urlresolver-remote-red title: Restore attempt evidence, deliver the shared-cache fix, and finish UrlResolver first-attempt repairs summary: Restore readable test-attempt evidence, finish the reviewed runner release and coordinator pins, and complete the remaining UrlResolver first-attempt repairs. Restart and cache source fixes are merged; runner adoption has a green required check with seven retry cases, while attempt histories and complete result pages remain unavailable. See the durable L39 evidence branches; no merge, publication or deploy was performed. created: 2026-09-30T21:49:13.770Z completed: null dependencies:

Handoff: Restore attempt evidence, deliver the shared-cache fix, and finish UrlResolver first-attempt repairs

RE-VERIFY — 2026-10-04T20:08Z snapshot. This is a write-time snapshot. Before acting, use handoff-cli get/claims, gh pr view <full URL> --json state,headRefOid,mergeStateStatus,statusCheckRollup,mergedAt, git ls-remote, current Maven metadata/POMs, and BuildTest run/result/attempt endpoints. The WUI run projection currently reports outOfDate=true and refreshFailing=true, so it is not authoritative when it contradicts terminal run evidence. A green final result with automatic retries is not proof of clean first attempts.

Mission

Original request: explain why UrlResolver required remote runs after approximately 20:04 UTC on 2026-09-30 failed 19–40 tests, then later stopped with zero tests, and fix the causes at their owning layer. The original evidence compared unchanged UrlResolver commit c2564bca across clean run c748ca69 and failing run 3070c51e. This lane triaged the existing work; all merges, queue actions, releases, deployments, and handoff completion are reserved to the supervisor.

What was found and done

  1. Restart dispatch source work is already on main. https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1177 merged as https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/8142487a4245a1487c78fab84cbbba094dda965d. Current main verified at 13ae9f5c6d78f2b0bb57e7160629f716fdb2b012. Recovery previously treated a retained shard without a durable queue as a static owner even if no runner ever executed, and missing shards became static if no queue survived. Current ownership excludes only persisted work held by actual runner execution evidence, while retained no-runner shards and missing shards consume the rebuilt dynamic queue. Before-start, after-start, and ownership-matrix tests exercise real SSH, persisted records, event spools and public terminal results. The merged PR records red-first proof and a 27-test passing neighborhood. Those counts are historical; the resumed lane did not complete a fresh test execution. Do not repeat this fix.

  2. The shared-cache root correction is merged upstream, but is not delivered by the default runner. https://github.com/CodexCoder21Organization/kompile-executionenvironment/pull/74 merged as https://github.com/CodexCoder21Organization/kompile-executionenvironment/commit/a326e79811baa1125900112c4c791a1eb42cbaae. Its production evidence is four MISS/four Stored records for one key: concurrent children each rebuilt UrlResolver because the parent did not claim an in-flight cache key. Real child-JVM tests demonstrated four computations before and one after; tests cover failed/dead owners, dead waiters, independent keys, cycles, nested children and uncacheable values. Memory admission also landed in https://github.com/CodexCoder21Organization/BuildTestRunner/pull/140 and preparation heap sizing in https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1179. These are separate changes, not a measured license to call an overloaded host the root cause.

  3. Delivery still requires a reviewed runner release and coordinator identity update. BuildDriver.kt default and PIPELINE tuples pin published runner 0.0.122 / manager 0.0.198. Fresh Maven metadata still names runner 0.0.122 as latest/release, and runner/manager POMs resolve executionenvironment 0.0.38. Prior direct JAR inspection found the cache coordination in 0.0.44, not 0.0.38/0.0.39. Existing https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145 adopts CLI 0.0.121, manager 0.0.205 and executionenvironment 0.0.44. It publishes no runner. Its required remote check for run 0cd86dc8 passed 313/313 final results at cf8a4058d6d14787cec6ece0607b07f6d8d1e874, but explicitly warns that seven tests passed only after retry. /api/test-attempts?id=0cd86dc8 returned READ_FAILED with no rows and stated that the coordinator did not return a complete response. Historical focused verification was 7/7; the prior full fallback had 312 first-attempt passes and one CPU-accounting failure followed by a second-attempt pass. Do not call this a clean suite. A coordinator-only deployment cannot deliver the dependency fix.

  4. The original fifty-test repair corpus is incomplete, and existing work must be salvaged. Six complete old static runs plus a partial seventh yielded 120 distinct failed names, retained on the prior evidence branch. That is a different set from the original unnamed fifty first-attempt failures in nominally clean runs. A mention scan of 53 UrlResolver PRs found 30 of the 120 names; it is an index, not proof of fixes. Historic raw logs now return 404. Historic c748ca69 attempt endpoint first returned LOADING, then READ with zero rows; it cannot currently recover those names. Newer attempt endpoints return LOADING or READ_FAILED. The named examples have useful existing work: clock-driven deadline proofs in https://github.com/CodexCoder21Organization/UrlResolver/pull/1171, polling readiness in https://github.com/CodexCoder21Organization/UrlResolver/pull/1162 and https://github.com/CodexCoder21Organization/UrlResolver/pull/1183, and close-during-preparation ownership in https://github.com/CodexCoder21Organization/UrlResolver/pull/1184. Preserve their tests. The combined https://github.com/CodexCoder21Organization/UrlResolver/pull/1167 overlaps those component fixes. No PR was closed or modified by this lane.

  5. Current failures are a mixture of service failures and executed assertions. https://github.com/CodexCoder21Organization/UrlResolver/pull/1183 required run 1f23c644 stopped during createDropletAsync provisioning: 1888 discovered, zero executed, zero failed. https://github.com/CodexCoder21Organization/UrlResolver/pull/1186 required run 067e8783 recorded 1887 passes and four failures out of 1891. Its check says terminalization found already-terminal persisted evidence after 196980ms held in TESTING, and attempt-history enrichment failed after 30 seconds. The stale run-list still calls it BUILDING with zero results. Result pages 1/2 expose three failures: stressTestActiveQueryMalformedResponseAlwaysReachesParsing (zero of sixteen streams opened), testActiveQueryReportsRequestWriteFailureAndRetries, and testActiveQueryReportsResponseReadFailure. The endpoint reports only 1567 rows with complete=false, and pages 3/4 return cached-view-not-ready errors. The fourth name remains unverified. Therefore neither a universal test defect nor a universal service-only explanation survives evidence. The separate historical Mode C run d8ed7de1 remains uninvestigated; do not present it as resolved.

  6. Verification and reports are blocked by unavailable detail and a constrained local host. Current run projection flags include outOfDate=true, refreshFailing=true, runMembershipReconciled=false and upToDateAsOfEpochMs=1791101091862. Health reports projection reset required and getTestResultsPaginated relay-response failure. No local Fabric certificate directory or SSH identity files are present, so direct host journal recovery is unavailable from this lane. Host memory repeatedly exceeded the brief's 4.5 GiB launch threshold; no local build/full suite was run. The previous invocation's remote command failed while materializing its 259 MiB checkout into a client ByteArrayOutputStream before a run ID existed. The resumed lane prepared a source-only export, then stopped two client startup invocations (wrong working directory, then above-threshold memory) before health/submission; neither yielded a run ID or verdict. No further tests were submitted. This is a verification limitation, not a diagnosis that the test code is too demanding. The RUNNING report upload timed out after 120 seconds with a closed-channel warning; authoritative claim recheck/update results are recorded in the evidence branch. No workaround was added to product code.

Relevant PRs / refs

Merged source:

Repo Branch Head SHA PR State / checks Recommendation
BuildTestEmbedded resume-keeps-dynamic-dispatch c8fe4fe5ee406063acec3db1d00813708f205faf https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1177 MERGED; UNKNOWN; bld-test-shard (1 of 4): SUCCESS, bld-test-shard (2 of 4): SUCCESS, bld-test-shard (3 of 4): SUCCESS, bld-test-shard (4 of 4): SUCCESS, bld-build: SUCCESS Retain merged proof; no action
BuildTestEmbedded prepheap/preparation-runner-heap 26cb27079283fa78db471f7405322805e766144c https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1179 MERGED; UNKNOWN; bld-test-shard (1 of 4): SUCCESS, bld-test-shard (2 of 4): SUCCESS, bld-test-shard (3 of 4): SUCCESS, bld-test-shard (4 of 4): SUCCESS, bld-build: SUCCESS Retain merged proof; no action
kompile-executionenvironment keesf-cache-single-flight-fix 0c46ccc8a215769202e922672621539ac9be4418 https://github.com/CodexCoder21Organization/kompile-executionenvironment/pull/74 MERGED; UNKNOWN; bld-all-tests: SUCCESS, kotlin.build (kompile-remote-build): SUCCESS, kotlin.build (remote): SUCCESS Retain merged proof; no action
BuildTestRunner btrgate/build-only-memory-gate 74aa8e7391ca5511e8f764b808ec3b5e25d5cce9 https://github.com/CodexCoder21Organization/BuildTestRunner/pull/140 MERGED; UNKNOWN; kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Retain merged proof; no action
BuildTestRunner release/pub1-manager-cli-pins cf8a4058d6d14787cec6ece0607b07f6d8d1e874 https://github.com/CodexCoder21Organization/BuildTestRunner/pull/145 OPEN; UNSTABLE; kotlin.build (kompile-remote-build): IN_PROGRESS, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver wip/sweep3-w4-r83-idle-close-injected-clock 028f528587b6d1fb8024c2012c310f0a29448ab1 https://github.com/CodexCoder21Organization/UrlResolver/pull/1120 OPEN; UNSTABLE; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver wip/K59-findpeers-join-signal 53867434b99f5aa23e9bca32951f12c40a40f6a4 https://github.com/CodexCoder21Organization/UrlResolver/pull/1162 OPEN; BLOCKED; bld-build: FAILURE, kotlin.build (kompile-remote-build): IN_PROGRESS, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver flakefix/batch-1120-1156-2026-10-02 1b4778881068e7785130d8a85d083a1e89f67354 https://github.com/CodexCoder21Organization/UrlResolver/pull/1167 OPEN; UNSTABLE; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver fix/deterministic-persistent-rpc-timeout-close 4c6c33f5a527a1b0e95468afb2c26d481bb55704 https://github.com/CodexCoder21Organization/UrlResolver/pull/1171 OPEN; UNSTABLE; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver fix/quiet-parameter-request-redial-proof bc9157c3260c534fec7d260a2e5d341833b38771 https://github.com/CodexCoder21Organization/UrlResolver/pull/1178 OPEN; UNSTABLE; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver fix/polling-interruption-fixture-2026-10-04 68f268041dd9121d22babb7735e3ebb89c0d6f64 https://github.com/CodexCoder21Organization/UrlResolver/pull/1183 OPEN; BLOCKED; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): FAILURE Keep; unresolved work, do not close
UrlResolver fix/close-during-host-preparation 451bf2e3b4626518787c88a22a83696384786d33 https://github.com/CodexCoder21Organization/UrlResolver/pull/1184 OPEN; UNSTABLE; bld-build: SUCCESS, kotlin.build (kompile-remote-build): FAILURE, kotlin.build (remote): SUCCESS Keep; unresolved work, do not close
UrlResolver fix/urlresolver-intermittent-test-bundle-2026-10-04 70c7908e2fdffa97dd9c686061c1a0ae2b7c7aee https://github.com/CodexCoder21Organization/UrlResolver/pull/1186 OPEN; BLOCKED; bld-build: SUCCESS, kotlin.build (remote): FAILURE, kotlin.build (kompile-remote-build): NEUTRAL Keep; unresolved work, do not close
BuildTestEmbedded fix/shared-preparation-uses-runner-path-20261003 9e072b9b19fa324b4cd62160fb182662dff0d596 https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1226 OPEN; BEHIND; bld-test-shard (1 of 4): FAILURE, bld-test-shard (2 of 4): SUCCESS, bld-test-shard (3 of 4): SUCCESS, bld-test-shard (4 of 4): SUCCESS, bld-build: FAILURE Keep with current owner; related cache work, not proved obsolete
BuildTestEmbedded wip/u2c-restart-2026-09-29b 2158b7172c8bd265769ab0b0eb5369bcbb5de95d https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1150 OPEN; DIRTY; no checks Keep with current owner; related cache work, not proved obsolete
BuildTestEmbedded wip/u6-2026-09-29b 381d4d98d0d8f265367b1d30452d30b51f527672 https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1147 OPEN; DIRTY; bld-test-shard (1 of 4): SUCCESS, bld-test-shard (2 of 4): SUCCESS, bld-test-shard (3 of 4): SUCCESS, bld-test-shard (4 of 4): SUCCESS, bld-build: SUCCESS Keep with current owner; related cache work, not proved obsolete
BuildTestEmbedded wip/shared-preparation-cold-rule-budget-W89 809d49f37650d56ad8aab34cc71f6764f69b6997 https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1115 OPEN; DIRTY; bld-test-shard (1 of 4): SUCCESS, bld-test-shard (2 of 4): SUCCESS, bld-test-shard (3 of 4): SUCCESS, bld-test-shard (4 of 4): SUCCESS, bld-build: SUCCESS Keep with current owner; related cache work, not proved obsolete

Evidence branches (do not merge):

Repo Branch Remote head PR What is on it State
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L39-restart-cache-evidence 3bb604209fbbb272d56aa6266ce01e26c1635d2b none Prior failure inventory, PR mapping, diffs and packaging/provisioning stacks Verified remote; retained
BuildTestEmbedded https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/L39-restart-cache-reverify 9012bdc89273f1bcc81390b2521cf58810d548ff none Resumed live state, service payloads, reviews and findings in investigations/L39-resumed Evidence only; no product changes, retain

No code was deployed or artifact published by either invocation of this lane. The live coordinator artifact was not established here; no assertion is made that the merged restart source is deployed. The current published runner and current main release tuple are verified separately above. Historical Mode C is unresolved.

Next steps

  1. Re-verify all live states and claims first. This lane's original claim/get worked, but later report upload timed out and a capped claims read returned an empty list. It reclaims only after that no-owner response. Do not assume board state from the stale WUI.
  2. Restore readable attempt histories and complete result pages, or obtain the durable coordinator journals/logs through an operator with Fabric credentials. For runner run 0cd86dc8, identify the seven first-attempt failures; for bundle run 067e8783, recover all four failures and full attempt records. Existing evidence explicitly refutes both a universal service-only failure and a universal assertion-only failure. Do not request blind CI reruns.
  3. Once a test host is available within the memory budget, run the merged before-start, after-start and ownership matrix regressions. Use a source-only git archive outside .git and downloaded launcher directories for remote packaging; fetch/rebase first. At most two remote submissions were permitted; none reached a new run in this lane, but the earlier packaging attempt and interrupted client starts are recorded honestly. Do not increase heap, timeouts, or weaken tests to compensate.
  4. Supervisor: review the existing runner adoption PR and its seven first-attempt failures. After verified repair and final review, decide its merge and authorize a runner release separately. Record the served runner artifact hash, complete dependency closure digest, manager/buildscript versions and both coordinator release tuples in a new main-targeting BuildTestEmbedded PR. No guessed version/hash may replace artifact evidence. A separately authorized rollout and real preparation measurement are still needed to establish delivery.
  5. Supervisor: select existing UrlResolver component/bundle PRs for salvage. Keep the stronger deterministic deadline tests and host-preparation-close proof. Polling PRs overlap; compare content before choosing, not merely final check colors. Existing silent-stream identity proof in https://github.com/CodexCoder21Organization/UrlResolver/pull/1178 is worth keeping. Recover the original fifty first-attempt identities from durable evidence/backups; if they no longer exist, make an explicit scope decision about using the recovered 120 static failures and current first-attempt failures. Do not equate those corpora silently.
  6. Related BTE cache PRs in the table have distinct pending work and were not proved obsolete. Keep them with their existing owners. No PR or branch closure is recommended on current evidence. After every required repair, perform independent test/code review and full remote landing gates from exact heads. The supervisor owns every merge, publication, deploy, and eventual handoff completion.

Operational knowledge

  • Read PlanRepository handoffs README, especially Triaging, Format and Partial Work; PHILOSOPHY Root Cause/Upstream Fixes/Refuted Hypothesis; TESTING Public API, Crash Recovery, Flaky Policy and Never Calibrate; CODE_REVIEW Claims That a Host Is Overloaded/Unmeasured Performance/Weakening Tests. Source has no root AGENTS.md. All source/test review here is distinguished from historical or unexecuted gates in findings.
  • Correct WUI routes are /api/test-results?id=<run>&page=N and /api/test-attempts?id=<run>. /api/test-progress summarizes final rows, not first attempts; /api/runs does not filter by id. HTTP 200 with LOADING/READ_FAILED proves no successful history read. Even READ with an empty historic journal cannot establish original first-attempt reliability.
  • Use PATH=/code/ws/bin:$PATH. The working Coursier entry is cs or java -jar /code/ws/bin/coursier.jar. Current BTE scripts already bootstrap a pinned verified JVM Coursier. Bound CLI JVM heap on this shared host; preserve the same test settings. Read cgroup memory immediately before launch and use a conditional guard, not merely a printed value.
  • Plain build-watchman 0.0.21 watches required checks and exited GREEN for runner adoption; it does not authorize enqueue. Tracked timeouts only; no handwritten condition loops. Do not treat 503s alone as worker death, and do not delete/rerequest a run without the owning operator's verified ground truth.
  • The Handoff report was recorded to disk but timed out in the service. The refreshed body and update receipt/error are retained on the evidence branch. Local Fabric certificates and SSH keys are absent; do not guess credentials or manage production through alternate unverified paths.
  • Report-challenge auto-merges its record, forbidden in this lane. Supervisor should record: workspace packaging materializes the whole checkout; attempt-history RPCs fail/return incomplete views; handoff claim/report/read state changed within the claim heartbeat period; and shared disk exhaustion killed the previous lane. These are concrete tool/service issues, not explanations that a host is inherently overloaded.

Write-time terminal state: BLOCKED_INFRA. Both review passes and all retained evidence are in investigations/L39-resumed on the re-verification evidence branch. The supervisor must decide merges, release, rollout, and eventual completion. Current source-only proof and historical counts must not be reported as a fresh clean suite.