buildtest quota-pressure admission degrades large suites to unsharded single-droplet runs instead

Reported (UTC): 2026-07-16 14:14

buildtest quota-pressure admission degrades large suites to unsharded single-droplet runs instead of waiting for shard capacity

What was being attempted: SLO monitoring of production buildtest (198.199.106.165, url://buildtest/) during the 2026-07-15/16 CI-flakes-and-parallelism workstream, under droplet-quota pressure (BUILDTEST_DROPLET_LIMIT=75, bursts of 5+ concurrent 10-shard runs).

What went wrong: When fleet admission cannot grant a full shard allocation, large run-all suites (>250 tests, which should shard across up to 10 droplets with 4-way per-droplet parallelism) are admitted DEGRADED to a single unsharded droplet instead of briefly waiting for capacity. A 1,100-test UrlResolver suite on one 2-vCPU droplet takes 60-90+ minutes versus ~10-15 minutes sharded ��� silently blowing the 10-minute time-to-first-test SLO and the 40-way concurrency goal precisely during the busiest periods. Admission-wait behavior ('Waiting for droplet capacity; retrying admission in 300s') exists and works for full waits, but the degrade-to-1-shard path bypasses it.

Impact: Worst-case CI latency (10x) hits exactly when the fleet is busiest; monitoring shows R1 SLO violations that are admission policy, not wedges; operators waste ground-truth checks on runs that were doomed to be slow at admission time.

Workaround used: None (runs complete correctly, just slowly). Monitoring treats these as known-cause violations after host verification.

Suggested durable fix: In BuildTestEmbedded's admission/shard-planning (BuildTestEmbeddedService/planTestShards + admission), for suites above the sharding threshold prefer bounded waiting for at least a partial multi-shard allocation (e.g., wait up to N minutes for >=K shards, then admit with however many are grantable) over immediate single-droplet degradation; record the degradation decision in run.json and the build.log so the WUI/monitoring can distinguish 'admitted degraded under quota' from normal runs. Hermetic e2e per the existing dropletLimit admission test family (see BuildTestEmbedded #294 for related edge-condition coverage).

---

Production verification — 2026-07-17

Status: STILL EXISTS. Current production remains degraded rather than retired: buildtest.kotlin.build returned HTTP 200 but took 8.89 seconds, while the production host showed load average 60.59, CPU pressure near 90%, and the BuildTest JVM at about 1.8 GiB RSS.

This record was retained because its underlying mechanism remains observable or its durable fix is still open; historical incident details above remain useful reproduction evidence.