← Handoffs

Repository · handoffs

Handoff: Obtain deployed BuildTest versions and aligned projection/connection evidence

View on GitHub ↗

id: hf-2026-10-03-investigate-the-stale-buildtest-projection-after-deploy-4a url: url://handoff/handoffs/hf-2026-10-03-investigate-the-stale-buildtest-projection-after-deploy-4a title: Obtain deployed BuildTest versions and aligned projection/connection evidence summary: Obtain operator-supplied current coordinator/WUI image identities, persisted projection metadata, and aligned provider/client close logs before choosing a source fix. Original source and server-pin PRs are merged; classifier gate passed 1/1, but public membership remains incomplete and a completed direct run is still projected PENDING. Evidence and exact request are preserved on wip/L8-projection-triage-20261004. created: 2026-10-03T01:33:34.744Z completed: null blocked-reason: Need operator-supplied current images, projection metadata and aligned connection-close logs; this host has no production credentials. dependencies:

Handoff: Obtain deployed BuildTest versions and aligned projection/connection evidence

Written: 2026-10-04T11:55:37.397024+00:00

RE-VERIFY: Every state below is a snapshot. Check live claims before updating; use gh pr view <url> --json state,headRefOid,statusCheckRollup,mergedAt, git ls-remote, and public /api/health and /api/runs?limit=500. Re-check actual production images through the operator before proposing a deploy or attributing a deployed defect. Fresh queries win.

Mission

The original request was to hotfix-deploy the coordinator read-back fix at https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1202 as "deploy 4a". Deploy 4a means the coordinator deployment at 2026-10-03T00:48:51.789960Z of BuildTestServerService source 6d58976f4238e5770a45996245f73c9e7cb9cccb, embedding version 0.0.69274744 built from validated source 6a955c852db4bebb677e82397e1e4f705f9dcec9. Its ten-minute startup check passed; the forty-minute acceptance failed. The follow-up asked why its WUI projection remained behind.

Current lane outcome is NEEDS_USER: original source work is delivered, but public projection is still behind and its physical-close/reset condition is not identified. The supervisor owns merge, queue, production, publish and completion decisions. This host has no production credentials; no production operation, artifact publish, source merge, PR closure, or handoff completion was performed.

Findings and work (chain)

  1. Both original PRs are MERGED with green recorded checks. By content, BuildTestEmbedded main 1de45a68db4c34d27e12494b2df00877d012e527 contains bounded public snapshot rows, compact utilization reads, prefix-before-value pending scans, and scenario-specific whole-directory repair. All fourteen regression files added by the source PR remain present. Source merge commit: https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/59db2da309fdb76715445fd574308cc929047a06. Its pending-merge instructions are stale.
  2. Legacy baseline paging, which deploy4a deliberately excluded, is now on main through https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 and https://github.com/CodexCoder21Organization/BuildTestEmbedded/commit/068b5607bc15a53346d3ee6ad0bebd935aeba061. Whole-directory removal is distinct from a missing child/source file in a directory that remains. Do not merge a stale full source branch or interpret every missing file as deletion.
  3. Public samples at 11:06–11:48 UTC show snapshot commits and totalCount advancing. The final sample has snapshotAsOf=1791114484478, totalCount=10644, but fully caught-up=1791101091862 (2026-10-04T08:04:51.862Z), outOfDate=true and runMembershipReconciled=false. Health samples retain resetRequired=true. The historical October 2 frozen-head timestamp is stale, but complete recovery is not established.
  4. Earlier samples report an ambiguous getTestResultsPaginated RPC after its write began, an EOF before the four-byte response prefix, and a reconnect rejected on a terminal root. Later samples clear the transport error/gap. The terminal-registration message originates in UrlProtocol admission control and its public-path terminal-refusal test intentionally emits it; it does not name the preceding close event. The stream-close symptom existed before deploy4a. Old heap/GC observations are not adopted as a cause.
  5. Initial wider pages had no run start after 11:04:18; the final page advances to 11:34:47.944. It now contains this lane's remote run d6ef289a as PENDING although the direct remote client completed and returned one selected test passed. This is delayed/incomplete projection, not a totally frozen writer. It may overlap the separately owned completed d0cb2b45 result-view lane; no duplicate result-count change was made. Supervisor should correlate the two evidence sets before allocating more source work.
  6. Rebased WUI main targeted gate projectionFeedSandboxTransportMessagesAreTransportFailuresTest passed: 1 selected /0 failed, 976 filtered by selector; case duration6532ms, selected buildMaven175011ms, client exit0. This validates classification through real Jetty/coordinator and sandbox rendering, not the cause of the production close. No product code changed. Nonselected artifact build rows report FAIL/0ms in the returned log; no such builds were requested, and no full-suite/full-build claim is made.
  7. The bounded plain watcher continued to see PENDING and hit its540-second process deadline (shell124; RESULT INTERRUPTED/130). It did not certify the direct client's result. The client and watcher ended; no lane-owned background jobs remain. Private temporary local-client heap inspection used only to recover the initially unprinted runId was deleted, never uploaded.
  8. Separate parent test-comprehensiveness and adversarial reviews are preserved in the evidence findings. The existing recovery/continuation/missing-child/baseline-memory PRs below remain separately owned; their current red checks are not hidden by reruns. There is no new fix PR because no unique deployed mechanism or valid fail-first reproducer has been established.

Relevant PRs and refs

PR State Head SHA Observed checks Disposition
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1202 MERGED 684fdf66dab7c87ce25b1cf9d448aa9cfbb701a9 bld-test-shard (1 of 4) SUCCESS; bld-test-shard (2 of 4) SUCCESS; bld-test-shard (3 of 4) SUCCESS; bld-test-shard (4 of 4) SUCCESS; bld-build SUCCESS Merged source; retain provenance
https://github.com/CodexCoder21Organization/BuildTestServerService/pull/387 MERGED 6d58976f4238e5770a45996245f73c9e7cb9cccb bld-build SUCCESS Merged source; retain provenance
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1208 MERGED b6fb5bce51f70be70131055e84184cf9199ea6d8 bld-test-shard (1 of 4) SUCCESS; bld-test-shard (2 of 4) SUCCESS; bld-test-shard (3 of 4) SUCCESS; bld-test-shard (4 of 4) SUCCESS; bld-build SUCCESS Merged source; retain provenance
https://github.com/CodexCoder21Organization/BuildTestServerService/pull/389 OPEN 2caebf6921301630b28c48c9c48d7907f9d12e9a bld-build SUCCESS Recommend supervisor close stale pin; preserve history
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1198 OPEN 3a7293e1d73b8a9be56a48d91ec06e8aa84c3042 bld-test-shard (1 of 4) FAILURE; bld-test-shard (2 of 4) FAILURE; bld-test-shard (3 of 4) FAILURE; bld-test-shard (4 of 4) FAILURE; bld-build FAILURE Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1223 OPEN 64f45f52590ee5877ba1e1eb7e8b9de3bcbd9b1d bld-test-shard (1 of 4) SUCCESS; bld-test-shard (2 of 4) SUCCESS; bld-test-shard (3 of 4) SUCCESS; bld-test-shard (4 of 4) FAILURE; bld-build FAILURE Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestWui/pull/338 OPEN 2b0cefb467578eb416dd7ea8a7123075852b198b bld-build FAILURE Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestWui/pull/359 OPEN 5166dd501a553f098da26614b1829bdc41f77595 bld-build SUCCESS Keep with existing owner; no takeover
Remote branch Head SHA Disposition
https://github.com/CodexCoder21Organization/BuildTestServerService/tree/release/pin-embedded-0.0.69274745-deploy4b 2caebf6921301630b28c48c9c48d7907f9d12e9a Recommend supervisor close stale pin; preserve history
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/projection-repair-missing-child-sources-2026-10-02 3a7293e1d73b8a9be56a48d91ec06e8aa84c3042 Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/fix/bounded-baseline-snapshots-20261003 64f45f52590ee5877ba1e1eb7e8b9de3bcbd9b1d Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestWui/tree/wip/L3-convergence-reproducer 2b0cefb467578eb416dd7ea8a7123075852b198b Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestWui/tree/fix/L123-projection-reset-resolver-recovery 5166dd501a553f098da26614b1829bdc41f77595 Keep with existing owner; no takeover
https://github.com/CodexCoder21Organization/BuildTestEmbedded/tree/wip/deploy4a-validated-source-2026-10-03 6a955c852db4bebb677e82397e1e4f705f9dcec9 Keep exact published-source/fixture evidence; substantive mechanisms and fourteen added tests are on main.
https://github.com/CodexCoder21Organization/BuildTestServerService/tree/release/pin-embedded-0.0.69274744-deploy4a 6d58976f4238e5770a45996245f73c9e7cb9cccb Keep exact historical deployed pin.
https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/deploy4a-evidence-2026-10-03 9d6ab2bcbd0c9ddbf3494a5ded60e8164e9e8ff6 Keep historical deployment/acceptance evidence.
https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/L8-projection-triage-20261004 b7fa05523cf126507b16dd831d31552e0c770ee0 Open evidence checkpoint; no PR. Raw public samples, source comparison, separate reviews, passing classifier log, operator request. This is the only branch pushed by L8.

Evidence: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/L8-projection-triage-20261004/handoffs/artifacts/2026-10-04-L8-projection-triage

Decision and next steps

  1. Re-verify current state first, including claims, source/main refs, PRs and public view. Do not assume historical image/pins are current.
  2. Ask the operator to supply an aligned read-only bundle: current coordinator and WUI route/container/PID/JAR hash and actual packaged versions; a consistent SQLite metadata snapshot including projection_feed_metadata, projection_checkpoint, projection_metadata's projection_recent_head_state and projection_run_membership; and provider/coordinator plus WUI logs around the same failed RPC, including root close/admission records if the actual protocol version emits them. Preserve matching process exits, routing/reaping records, GC times and thread stacks if relevant. Exact cold-reader request and options: https://github.com/CodexCoder21Organization/PlanRepository/blob/wip/L8-projection-triage-20261004/handoffs/artifacts/2026-10-04-L8-projection-triage/operator-request.md
  3. Recommended option: retain the handoff, obtain that bundle, compare deployed code with main, and force the actual discovered condition in a public-API self-contained reproducer before a fix. No restart, deploy, publish or merge is requested by the evidence bundle. Alternative requires an explicit supervisor scope decision: narrow this record to delivered original source work and transfer the still-live incident to an existing owner; do not label the incident resolved from merge state alone.
  4. Coordinate the d6ef289a completed-client/PENDING-view evidence with the lane owning hf-2026-10-04-reconcile-the-completed-d0cb2b45-test-result-view. If it is the same mechanism, keep one source owner and preserve this evidence rather than duplicating the fix. The supervisor decides any other handoff changes.
  5. Supervisor should review closing the old server .745 pin PR/branch: main already pins .748 and both constituent source fixes merged. Preserve its exact already-deployed provenance until actual production images are established. Keep the original published-source/pin/evidence branches as historical records and the separately owned open PRs with their owners. No closure or cherry-pick was performed here.
  6. Production acceptance/completion remains for the supervisor/operator after any authorized fix/rollout. This lane provides no artifact to deploy and no new production authority.

Historical artifact identity

Historical image /root/ContainerNursery-uploads/jars/buildtest-server-6d58976f-20261002-deploy4a.jar,155465901bytes,SHA256 e4ea0beb0dbd9533e133cba986f2f6a6d416cb802c267aeec639cfb6a29d1a7b. Embedded0.0.69274744 JAR SHA256 6e7e8e120b52624799a756990de015e4e84788f92a63c7ab34476aacec93d8d1, POM SHA256 c029e48f9d7252d091b912f135d8be1536bc07e22d77c4e157b3bb649cd201df. Historical acceptance recorded39/39 targeted server tests, RUNNING in405s, and a failed forty-minute quiet/freshness check. Full original evidence: https://github.com/CodexCoder21Organization/PlanRepository/tree/wip/deploy4a-evidence-2026-10-03/handoffs/artifacts/2026-10-03-deploy4a . These identify the original deployment, not the current live process.

Operational knowledge

All shells prepend /code/ws/bin to PATH. Working cs and wrappers are there; /usr/local/bin/coursier has the wrong architecture. Repositories have pinned launchers; do not reuse other lanes' checkouts. Use remote tests/builds under bounded tracked commands on this shared host. Never use --to-merged in this lane.

Snapshot-write time is not fully caught-up time; an intermediate baseline commit can advance one without the other. A connected wake signal alone cannot certify ordered freshness. Registration refusal on a terminal root is correct behavior; the preceding retirement/physical-close event must be established. A PENDING projection observed by watchman can disagree with a completed direct client; preserve both and do not blindly rerun the passing test. Default WUI SQLite path is buildtest-wui-data/projections.sqlite, overridden by BUILDTEST_WUI_PROJECTION_DB; obtain a consistent read-only snapshot rather than copying a changing database file carelessly.