← ArchiveArea

Land UrlProtocol relay-failover fix after buildtest event-ingestion failure

UrlProtocol pull request https://github.com/CodexCoder21Organization/UrlProtocol/pull/434 passed its head checks but was evicted from the merge queue when the production buildtest event-ingestion worker lost the shard channel. The pull request is open, clean, dequeued, and still at the pushed expected head; resume only after confirming the runner is healthy.

hf-2026-07-28-land-urlprotocol-relay-failover-fix-after-buildtest-event-ingestion-failure
Completed Jul 29, 2026 · 09:24 UTC This handoff is read-only in ArchiveArea.

Handoff document

Markdown

Land UrlProtocol relay-failover fix after buildtest event-ingestion failure

RE-VERIFY: This is a snapshot verified at 2026-07-28 01:05:27 UTC. Re-check https://github.com/CodexCoder21Organization/UrlProtocol/pull/434, its mergeQueueEntry, and the production buildtest service before taking action.

Mission summary

Get https://github.com/CodexCoder21Organization/UrlProtocol/pull/434 through required checks and merged. Merge is authorized. Do not publish an artifact and do not touch the downstream consumer pull request.

What was found and done

The pull-request head 0c35ec03882b2169c41333d14faa26ee9cc254f2 passed all three head checks. build-watchman 0.0.13 then enqueued it through the required merge queue.

The first queue attempt failed at https://buildtest.kotlin.build/run?id=9d78db23. The run record's first hard error was not a test assertion:

Dynamic-dispatch event-ingestion/dispatch loop on shard 4 failed for run '9d78db23': com.jcraft.jsch.JSchException: channel request: timeout

The result API showed 41 completed tests passing and 59 visible results marked only Test did not complete during build execution. This matches the task's production test-runner failure class: the event-ingestion worker failed and unfinished tests were reported as failures.

build-watchman automatically re-enqueued once after the eviction. The watcher was stopped, and GitHub's dequeuePullRequest mutation removed the pull request from the queue so it could not merge through another unclassified run. Live verification then showed the pull request open, mergeStateStatus: CLEAN, still at the expected head, and with mergeQueueEntry: null.

The infrastructure issue is recorded at https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0104-buildtest-merge-queue-run-failed-when-its-event-ingestion.md.

Relevant pull requests and refs

  • Pull request: https://github.com/CodexCoder21Organization/UrlProtocol/pull/434
  • Remote branch: https://github.com/CodexCoder21Organization/UrlProtocol/tree/fix/relay-failover-adoption
  • Remote branch head: 0c35ec03882b2169c41333d14faa26ee9cc254f2
  • Failure run: https://buildtest.kotlin.build/run?id=9d78db23
  • All source work is pushed. There are no unpushed UrlProtocol commits or code changes.

Next steps

  1. Confirm the production buildtest event-ingestion path is healthy or that the recorded runner issue has been addressed.
  2. Re-verify that https://github.com/CodexCoder21Organization/UrlProtocol/pull/434 remains open, clean, and at 0c35ec03882b2169c41333d14faa26ee9cc254f2.
  3. Run PATH=$HOME/bin:$PATH coursier launch buildwatchman:build-watchman:0.0.13 -r https://kotlin.directory -- --repo CodexCoder21Organization/UrlProtocol --pr 434 --to-merged.
  4. If a real test assertion fails, follow the requested 10-runs-at-head and 10-runs-at-current-main comparison under taskset -c 0,1; do not weaken a test.
  5. If the same run-level worker failure occurs, stop and report it rather than re-queuing.
  6. After a successful merge, fetch origin/main and confirm it contains the merge commit.

Reusable operational knowledge

Before treating many Test did not complete during build execution results as test failures, inspect the buildtest run record in /api/runs. A run-level event-ingestion or reporting-worker error can end the run without any test assertion failure.

build-watchman --to-merged automatically re-enqueues once after a queue eviction. When the task requires stopping on a production runner failure, stop the watcher and explicitly dequeue the pull request after classifying the failed run.