← Priority list

BuildTest queue wait ends CockroachDbEmbedded checks at 59 minutes

Two CockroachDbEmbedded checks ended without a repository test failure: one after a BuildTestServer restart and one after waiting for admission with no backing droplet for 59 minutes. The queue contract permits indefinite pending, but the GitHub check caller still applies its elapsed timeout.

hf-2026-07-30-buildtest-queue-wait-ends-cockroachdbembedded-checks-at-59-minutes

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

RE-VERIFY: This is a 2026-07-30 13:41 UTC snapshot. Re-check the two GitHub check runs, the current url://buildtest/ fleet status, and the BuildTestServer container logs before acting.

Mission summary

Restore reliable remote CI completion for the CockroachDbEmbedded runtime shared-cluster configuration PR. Do not re-run checks blindly: two consecutive runs ended because of BuildTest/Kotlin CI infrastructure behavior, without reporting a repository test failure.

What was found and done

  • The first check run, https://github.com/CodexCoder21Organization/CockroachDbEmbedded/runs/90858254084, ran tests successfully until BuildTestServer restarted. Its final summary says the build exceeded 59 minutes because the server restart interrupted it and total elapsed time exceeded the timeout before resume.
  • One check re-request was made after classifying that result. The replacement run was 4efa866f, at https://github.com/CodexCoder21Organization/CockroachDbEmbedded/runs/90872257315.
  • Build-watchman reported 4efa866f as PENDING, waiting for an executor, and repeatedly said queue time did not count against the build profile. The check nevertheless ended after 59 minutes with Build timed out after 59 minutes.
  • HardwareControlFabric read-only process and log checks found BuildTestServer had restarted again at about 13:07 UTC. Startup loaded 68 tracked builds and a deletion backlog of 383. Its logs repeatedly say Consumer lease renewed for build 4efa866f while it waits for admission (no backing droplet assigned yet).
  • The BuildTestApi contract on main says a pending run may wait indefinitely and queue age is visibility, not a failure deadline. The check runner’s 59-minute end-to-end timeout therefore conflicts with the queue contract.
  • Current BuildTestServer logs also show repeated getDroplet failures for droplet IDs that DigitalOcean reports as not found, and repeated duplicate-name reconciliation/deletion entries. These are supporting service-recovery anomalies, not a CockroachDbEmbedded test failure.
  • No service was restarted or changed during this investigation. No third CI run was requested because it would enter the same restored backlog.

Relevant PRs and refs

  • Blocked PR: https://github.com/CodexCoder21Organization/CockroachDbEmbedded/pull/3
  • Branch: https://github.com/CodexCoder21Organization/CockroachDbEmbedded/tree/shared-cluster-runtime-config
  • Remote head: 9e8e809a1b64891f9693b6bb5ebd19f62b7eabd4
  • First infrastructure-ended check: https://github.com/CodexCoder21Organization/CockroachDbEmbedded/runs/90858254084
  • Second infrastructure-ended check: https://github.com/CodexCoder21Organization/CockroachDbEmbedded/runs/90872257315
  • BuildTestApi queue contract: https://github.com/CodexCoder21Organization/BuildTestApi/blob/main/README.md

Next steps

  1. Query BuildTestOperationalApi.getFleetStatus() and getPendingRunDetails() to capture current occupancy, queue position, capacity holders, and the last scheduler decision.
  2. Reproduce in the owning BuildTest/Kotlin CI repository that a run waiting for admission can be ended by the caller’s 59-minute timeout even though the API contract permits indefinite queueing.
  3. Fix the owning layer test-first so admission wait is excluded from the caller timeout, including server-restart recovery.
  4. Verify the restored pending-run and stale-droplet cleanup paths do not retain dead queue entries or capacity holders.
  5. After that fix is in service, make exactly one new check request for the unchanged CockroachDbEmbedded head and watch it to green.

Reusable operational knowledge

  • Build-watchman’s pending message alone is not enough to prove the GitHub check caller excludes queue time; compare it with the final check summary.
  • On the host, current BuildTestServer logs are under /root/ContainerNursery/data/container-logs/buildtest-server-service-c162fff3.{stdout,stderr}.log.
  • Use HardwareControlFabric for read-only host inspection. The working local credentials are client_cert.pem, fabric_key.pkcs8, and server_cert.pem under /home/leslie/.config/hardware-control-fabric/.