← Priority list

Org-wide flake elimination: five fixes landed, three open pull requests with a known dependency chain

Five fixes merged. Three open: a quadratic peer-eviction regression in UrlProtocol (fix complete, local suite 1212/1212, but its performance test must be rewritten to be hardware-independent before landing), a ContainerNursery ordering fix where a live container became replaceable before its stop began (verified 15/15 locally, CI not yet run on the latest head), and a UrlResolver registration-ordering fix that is gated on the peer-eviction fix being published. An earlier claim that the merge queue was blocked org-wide by a build-runner defect has been retracted and must not be acted on.

hf-2026-07-28-org-wide-flake-elimination-four-verified-fixes-ready-merge-queue-blocked-by-the-shared-test-runner

Edit handoff

Title Summary Markdown body

# Org-wide flake elimination — five fixes landed, three open with a known dependency chain

**Written 2026-07-28 ~08:10 UTC. RE-VERIFY EVERYTHING BELOW BEFORE ACTING.** Every state fact here is a write-time snapshot. Re-check each pull request with `gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt` before acting on it.

## Mission

Find repositories with flaky test failures using the history at https://buildtest.kotlin.build/, confirm each suspected flake by running three concurrent remote builds against the repository's unmodified main branch, fix whatever is confirmed, and re-verify with three more concurrent builds. Goal: eliminate flakes across all repositories.

## Read this first — a correction to how one test must be written

`testPeerRegistryOverCapacityScoreBearingDistinctAddressesCompletesWithinBound` on https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 currently asserts a **wall-clock bound** (200 over-capacity insertions within 8,000 ms). It passes locally with the fix and fails in CI. A previous session proposed investigating the bound on the build hardware. **That is the wrong fix and must not be done.**

The operator's guidance, which supersedes it:

> Tests should allow generous timeouts such that you do not need to know which hardware you are running on. Tests protecting against things like exponential growth can be structured such that they will pass when the problem is fixed (no quadratic growth) but will always fail consistently regardless of hardware when quadratic growth occurs, by making the quadratic number grow incomprehensibly large.

So rewrite the test to separate the two behaviours by orders of magnitude rather than by a tight threshold. Options, in preference order:

1. **Scale the workload until the quadratic version is hopeless.** Pick a registry size and insertion count where the fixed implementation finishes in well under a second and the quadratic one would take minutes or hours, then set a very generous timeout (tens of seconds). Any hardware then separates the two cleanly.
2. **Assert a growth ratio, not an absolute time.** Run at size `n` and at `4n`, and assert the elapsed-time ratio stays below a threshold that linear-ish growth satisfies comfortably and quadratic growth cannot. Ratios cancel out machine speed.
3. **Count operations instead of timing them,** if the public API can expose a comparison or visit count. Fully deterministic and hardware-independent; preferred if reachable through public interfaces only.

Do not raise the existing threshold, and do not relax the assertion. The test exists to fail if the quadratic behaviour returns; a bound tuned to particular hardware cannot do that job.

## Current state — three open pull requests, one dependency chain

### 1. UrlProtocol — peer-registry eviction is quadratic

https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 "Speed up peer-registry capacity eviction", branch `fix/peer-registry-eviction-ranking`, head `bb5b6ee2f05ff25e1074d4e521e3ef2c1eb74bf9`.

**The defect:** adding a peer to a registry already at its 1,000-peer capacity computes the complete greedy ranking of all peers merely to take its last element as the eviction victim. That is quadratic. Version 0.0.409 instead selected a single minimum-scored victim in one pass; the regression arrived somewhere before 0.0.417. A build thread dump showed every writer thread stuck in that ranking routine: https://buildtest.kotlin.build/run?id=cae37dc4.

Source: `src/foundation/url/protocol/PeerRegistry.kt` — the call around `:401-408`, the greedy rank at `:513-624`, and `:627-645`.

**The fix (four commits, pushed):** the fail-first performance test, an incremental eviction ranking, cache-invalidation documentation, and an exact trie path for the tied-IPv4 case.

**Evidence:** unfixed, 2 of 200 insertions completed inside the budget; fixed, 200 of 200. A companion test `testPeerRegistryDistinctIpv4CapacityEvictsGreedyLastPeer` inserts 1,001 tied-score peers with distinct addresses, independently evaluates the *prior* complete greedy rule, and asserts the optimized registry evicts that exact peer — so the speed-up provably does not change which peer is chosen. Two review passes confirmed every original ranking rule survives: first-peer rule, score calculation, identical-address shortcut, diversity floor, IPv4 `/16` and IPv6 `/32` block rules, insertion-sequence tie-break, peer-id final tie-break.

**Local suite: 1,212 of 1,212 passing.** CI: 1,211 passed, 1 failed — only its own timing test, per the correction above.

**Next:** rewrite that test per the guidance, then land. This pull request gates the UrlResolver one below.

### 2. ContainerNursery — a live container became replaceable before its stop began

https://github.com/CodexCoder21Organization/ContainerNursery/pull/558, branch `fix/intermittent-zombie-url-tests`, head `79959955da9e46c7c4d5af4e5a10987b9dc7212f` (pushed at 08:08 UTC; **CI has not run on this head yet**).

This pull request has a long history worth knowing:

- It originally introduced a mechanism retaining containers whose death cannot be confirmed. That **broke eleven existing tests** — 0 clean invocations out of 10 at the branch head versus 5 of 5 on main. Three deterministic failures identified what changed: a defensive second force-termination attempt was bypassed; a specific route-failure diagnostic was replaced by a generic one; and containers *confirmed* dead were being retained. All three were judged unintended and the change was narrowed rather than the tests rewritten.
- A source analysis then found a **separate, more serious defect** introduced by the same change: `killContainerWithResult` removed a live container's active route entry *before* its force-stop began, so a concurrent creator could start a replacement while the predecessor was still alive. Full write-up: `/code/workspace/codex-lanes/cn558-mechanism-report.md`. The same hole existed in `killContainerIfCurrent`, reached from the URL facade's generation-scoped recovery — a production path no test covered.
- The fix makes an unconfirmed cleanup block a replacement by default on drain-first routes, with opt-outs only where the predecessor was already observed dead, the container was never started, or the route uses documented concurrent restart. It also reversed https://github.com/CodexCoder21Organization/ContainerNursery/commit/e5063c75d1c18bc1d4bcafe1e7e94f629263deb2, which had removed that protection from definite-start-failure cleanup — unsound because `start()` can throw after spawning a process.
- CI on the previous head then failed three tests; the just-pushed head resolves them: one was a **real defect in the fix** (a shutdown path returned without awaiting the cleanup it owned — fixed at source), and two tests legitimately permit overlapping containers and now configure that explicitly, with assertions, bounds, timeouts and iteration counts unchanged.

**The narrower force-stop-only form of the blocking rule was explicitly considered and rejected**: the deterministic reproducer `testGetOrCreateContainer_blockingCleanupKillReleasesRoute` does not pass under it.

**Latest local verification** (confirmed fresh build, one batched invocation, 15 of 15): the three previously-failing tests 3/3, the deterministic reproducer 1/1, and the full eleven-script regression set 11/11 — the last of these proving the correction did not swing back to the over-broad behaviour.

**Next:** run CI on head `79959955` and land.

### 3. UrlResolver — registration blocked on relay acknowledgement

https://github.com/CodexCoder21Organization/UrlResolver/pull/855 "Stop blocking global service registration on the relay acknowledgement", branch `fix/register-global-service-relay-ack-latency`, head `42c572e28fa6c97cf5e7c0f635f112e4e5c4e708`. Pinned to `foundation.url:protocol:0.0.417`.

Removing the wait exposed a real ordering defect, since fixed: the initial announcement and the acknowledgement-triggered circuit-address reannouncement stamped their timestamps in independent coroutines with no ordering edge, so a fast acknowledgement plus a delayed initial coroutine let the stale circuit-less announcement stamp later and overwrite the good state. Source: `src/foundation/url/resolver/UrlResolver.kt:8629-8637`, with the two launches at `:8667-8675` and `:9789-9799`.

The fix reserves the initial announcement's timestamp synchronously before an acknowledgement can arrive, passing it into the still-asynchronous broadcast. Reviews confirmed: registration performs no network wait (the change's whole purpose), the reserved announcement takes immutable copies, withdrawals still flow through the real broadcaster, and unregistering retains a later timestamp.

**Its CI cannot go green yet.** Latest run: 1,204 passed, 4 failed of 1,208.

- `stressTestOnPeerRegisteredWithRelayObservationBeforeRegistry` — **caused by the quadratic peer-eviction defect above**, not by this change. Will clear when 0.0.417's successor is published and pinned.
- `stressTestNonBlockingAnnouncementReachesReachablePeersUnderLoad` and `testNonBlockingAnnouncementStillReachesReachablePeers` — **unexplained.** Both bypass the changed registration path entirely and prove only late-or-lost delivery at a five-second observation deadline. The analysis ruled out random fanout exclusion in the two-node fixture and any observation-before-registry breach, and stated it could not choose among queue rejection, delayed execution, or exhausted delivery without a stage-level trace. Do not modify these tests on current evidence.
- `testDiscoverPeersShouldEnsureNetworkJoined` — appeared in the latest run only; not yet examined.

**Next:** after the peer-registry fix is published, bump this pull request's dependency pin, then re-assess the remaining failures.

## Relevant pull requests, branches and refs

**Open:**
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 — head `bb5b6ee2`
- https://github.com/CodexCoder21Organization/ContainerNursery/pull/558 — head `79959955`
- https://github.com/CodexCoder21Organization/UrlResolver/pull/855 — head `42c572e2`

**Merged this session:**
- https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/430 — 2026-07-28 04:48:26 UTC. Admission wake-up and key-generation fixes; verified at 63 fresh executions, 0 failures.
- https://github.com/CodexCoder21Organization/UrlProtocol/pull/434 — 2026-07-28 03:03:25 UTC. Relay-failover registration leak; published as `foundation.url:protocol:0.0.417`.
- https://github.com/CodexCoder21Organization/ContainerNursery/pull/557 — 2026-07-28 01:36:57 UTC.
- https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/423 — 2026-07-27 13:14:15 UTC.
- https://github.com/CodexCoder21Organization/BuildTestCli/pull/21 — 2026-07-26 04:24:02 UTC.

**Branch pushed for preservation:** https://github.com/CodexCoder21Organization/BuildTestCli/tree/wip/cancel-verb-handoff-2026-07-28 — remote head `01450e0446e31a892caa0e3f8f5d7c4423624364`. Adds a `cancel <runId>` verb calling `cancelBuildRun`, written because a run was wedged in `TESTING` and `deleteBuildRun` refused it with no released CLI able to cancel. Compiles, worked in practice, no tests, unreviewed. Worth contributing properly.

**Published artifact:** `foundation.url:protocol:0.0.417` — verified byte-for-byte against the local build and confirmed by inspection to contain the relay fix. **It also carries the quadratic peer-eviction regression**, so its successor must include the fix from https://github.com/CodexCoder21Organization/UrlProtocol/pull/439.

**Challenge records filed:**
- https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0104-buildtest-merge-queue-run-failed-when-its-event-ingestion.md
- https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-2221-hardwarecontrolfabric-process-skill-points-to-certificate.md
- https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-2308-containernursery-cli-cannot-inspect-logs-for-active-url.md
- https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0322-hardwarecontrolfabric-skill-default-certificate-paths-are.md

**Working notes** are in `/code/workspace/codex-lanes/*.md` on the session machine — briefs, per-lane reports, verification logs. These are local-only and will not survive the machine; the substance of each is summarized above. The multi-megabyte console logs were deliberately not pushed.

## A retracted claim — do not act on it

An earlier report in this session claimed the merge queue was blocked org-wide by a defect in the build-test runner, and requested authorization to deploy version 0.0.75 of it. **That was withdrawn.** The evidence had conflated two different failures (a JSON parse error on console records containing a zero byte, and an SSH channel timeout), and the deployed-jar timestamp was repeated from a subagent without independent verification. Three runs did fail in the service's dispatch loop across three unrelated repositories between 00:40 and 00:47 UTC — genuinely infrastructure, not the changes under test — but the window closed at 00:47 and roughly fifteen clean runs followed. No deploy is needed and none was performed.

## Next steps, in order

1. **Rewrite the peer-registry performance test** per the guidance at the top of this document — separate fixed from quadratic by orders of magnitude, not by a tuned threshold — then get https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 green and merged.
2. **Publish the successor to `foundation.url:protocol:0.0.417`** containing that fix. Confirm the new coordinate is unused before building, that the build genuinely compiled (a build rule completing in milliseconds is a stale cache hit), that the published checksum matches the local jar, and that the published binary contains the change.
3. **Land https://github.com/CodexCoder21Organization/ContainerNursery/pull/558** — run CI on head `79959955` and merge.
4. **Bump the dependency pin on https://github.com/CodexCoder21Organization/UrlResolver/pull/855** to the new version, then re-assess its remaining failures. Expect the peer-observation stress test to clear; the two announcement tests and `testDiscoverPeersShouldEnsureNetworkJoined` still need a stage-level trace before anyone touches them.
5. **Resume the survey.** Not yet verified with three concurrent builds: kompile, kompile-cli, AiCliSupervisorManagerWui. Known-suspected and unfixed: kompile-core timing margins (2 slow tests), BuildTestServerService upload-teardown, FileRelationalFilesystemExplorerWui (3 tests), AiCliSupervisorManagerEmbedded wall-clock race in its usage-hang test, BuildTestWui screenshot upload budget, and UrlResolver `testResilienceHarnessFirstLaunchUsesAtomicEphemeralBind` (evidenced as pre-existing on main).

## Reusable knowledge

- **Do not write hardware-sensitive performance assertions.** Structure a complexity test so the bad implementation is astronomically slower than the good one, then use a generous timeout — or assert a growth ratio across two problem sizes, or count operations. Investigating "what bound does this machine need" is the wrong branch of that decision tree.
- **Measuring how often a test fails is not the same as reproducing it.** Two lanes here spent about eighty minutes each on branch-versus-mainline rate comparisons and produced roughly one usable sample between them. Amplifying to a deterministic reproducer, or reading the source for the mechanism, returned located root causes in thirteen and eighteen minutes — including a defect in a different repository from the one under investigation.
- **Deliver before the final broad sweep.** Four lanes finished their real work, started an expensive full-suite verification, and hit their time limit before pushing — leaving finished work needing rescue. Push and open the pull request first; run the sweep as confirmation.
- **Build the artifact once and batch test selections.** These repositories recompile a large script catalog on every test invocation — several minutes each. Use the documented colon-separated multi-test specification, not repeated `--test` options, which triggers one catalog pass per option.
- **Attach an equivalence obligation to every optimisation.** Requiring proof that the same eviction victim is still chosen killed a wrong approach (a cache that cannot survive an insertion) and produced a test that independently recomputes the original algorithm's answer.
- **When a change moves a behavioural boundary, expect the tests on the other side to fail, and suspect the change first.** This happened twice in one repository tonight; the first time the change was wrong, not the eleven tests.
- **Verifying that a published artifact contains your fix does not verify the artifact.** `0.0.417` was confirmed freshly built, checksum-matched, and containing the intended methods — and still carried an unrelated quadratic regression.
- **A run reporting many failures but naming no failing assertion is an infrastructure failure until proven otherwise.** Look for the first hard error; results marked "did not complete" with zero duration and wildly swinging counts are signatures of the reporting path breaking.

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Org-wide flake elimination — five fixes landed, three open with a known dependency chain

Written 2026-07-28 ~08:10 UTC. RE-VERIFY EVERYTHING BELOW BEFORE ACTING. Every state fact here is a write-time snapshot. Re-check each pull request with gh pr view <n> --repo <owner>/<repo> --json state,mergeStateStatus,statusCheckRollup,mergedAt before acting on it.

Mission

Read this first — a correction to how one test must be written

testPeerRegistryOverCapacityScoreBearingDistinctAddressesCompletesWithinBound on https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 currently asserts a wall-clock bound (200 over-capacity insertions within 8,000 ms). It passes locally with the fix and fails in CI. A previous session proposed investigating the bound on the build hardware. That is the wrong fix and must not be done.

The operator's guidance, which supersedes it:

Tests should allow generous timeouts such that you do not need to know which hardware you are running on. Tests protecting against things like exponential growth can be structured such that they will pass when the problem is fixed (no quadratic growth) but will always fail consistently regardless of hardware when quadratic growth occurs, by making the quadratic number grow incomprehensibly large.

So rewrite the test to separate the two behaviours by orders of magnitude rather than by a tight threshold. Options, in preference order:

Scale the workload until the quadratic version is hopeless. Pick a registry size and insertion count where the fixed implementation finishes in well under a second and the quadratic one would take minutes or hours, then set a very generous timeout (tens of seconds). Any hardware then separates the two cleanly.
Assert a growth ratio, not an absolute time. Run at size n and at 4n, and assert the elapsed-time ratio stays below a threshold that linear-ish growth satisfies comfortably and quadratic growth cannot. Ratios cancel out machine speed.
Count operations instead of timing them, if the public API can expose a comparison or visit count. Fully deterministic and hardware-independent; preferred if reachable through public interfaces only.

Do not raise the existing threshold, and do not relax the assertion. The test exists to fail if the quadratic behaviour returns; a bound tuned to particular hardware cannot do that job.

Current state — three open pull requests, one dependency chain

1. UrlProtocol — peer-registry eviction is quadratic

https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 "Speed up peer-registry capacity eviction", branch fix/peer-registry-eviction-ranking, head bb5b6ee2f05ff25e1074d4e521e3ef2c1eb74bf9.

The defect: adding a peer to a registry already at its 1,000-peer capacity computes the complete greedy ranking of all peers merely to take its last element as the eviction victim. That is quadratic. Version 0.0.409 instead selected a single minimum-scored victim in one pass; the regression arrived somewhere before 0.0.417. A build thread dump showed every writer thread stuck in that ranking routine: https://buildtest.kotlin.build/run?id=cae37dc4.

Source: src/foundation/url/protocol/PeerRegistry.kt — the call around :401-408, the greedy rank at :513-624, and :627-645.

The fix (four commits, pushed): the fail-first performance test, an incremental eviction ranking, cache-invalidation documentation, and an exact trie path for the tied-IPv4 case.

Evidence: unfixed, 2 of 200 insertions completed inside the budget; fixed, 200 of 200. A companion test testPeerRegistryDistinctIpv4CapacityEvictsGreedyLastPeer inserts 1,001 tied-score peers with distinct addresses, independently evaluates the prior complete greedy rule, and asserts the optimized registry evicts that exact peer — so the speed-up provably does not change which peer is chosen. Two review passes confirmed every original ranking rule survives: first-peer rule, score calculation, identical-address shortcut, diversity floor, IPv4 /16 and IPv6 /32 block rules, insertion-sequence tie-break, peer-id final tie-break.

Local suite: 1,212 of 1,212 passing. CI: 1,211 passed, 1 failed — only its own timing test, per the correction above.

Next: rewrite that test per the guidance, then land. This pull request gates the UrlResolver one below.

2. ContainerNursery — a live container became replaceable before its stop began

https://github.com/CodexCoder21Organization/ContainerNursery/pull/558, branch fix/intermittent-zombie-url-tests, head 79959955da9e46c7c4d5af4e5a10987b9dc7212f (pushed at 08:08 UTC; CI has not run on this head yet).

This pull request has a long history worth knowing:

It originally introduced a mechanism retaining containers whose death cannot be confirmed. That broke eleven existing tests — 0 clean invocations out of 10 at the branch head versus 5 of 5 on main. Three deterministic failures identified what changed: a defensive second force-termination attempt was bypassed; a specific route-failure diagnostic was replaced by a generic one; and containers confirmed dead were being retained. All three were judged unintended and the change was narrowed rather than the tests rewritten.
A source analysis then found a separate, more serious defect introduced by the same change: killContainerWithResult removed a live container's active route entry before its force-stop began, so a concurrent creator could start a replacement while the predecessor was still alive. Full write-up: /code/workspace/codex-lanes/cn558-mechanism-report.md. The same hole existed in killContainerIfCurrent, reached from the URL facade's generation-scoped recovery — a production path no test covered.
The fix makes an unconfirmed cleanup block a replacement by default on drain-first routes, with opt-outs only where the predecessor was already observed dead, the container was never started, or the route uses documented concurrent restart. It also reversed https://github.com/CodexCoder21Organization/ContainerNursery/commit/e5063c75d1c18bc1d4bcafe1e7e94f629263deb2, which had removed that protection from definite-start-failure cleanup — unsound because start() can throw after spawning a process.
CI on the previous head then failed three tests; the just-pushed head resolves them: one was a real defect in the fix (a shutdown path returned without awaiting the cleanup it owned — fixed at source), and two tests legitimately permit overlapping containers and now configure that explicitly, with assertions, bounds, timeouts and iteration counts unchanged.

The narrower force-stop-only form of the blocking rule was explicitly considered and rejected: the deterministic reproducer testGetOrCreateContainer_blockingCleanupKillReleasesRoute does not pass under it.

Latest local verification (confirmed fresh build, one batched invocation, 15 of 15): the three previously-failing tests 3/3, the deterministic reproducer 1/1, and the full eleven-script regression set 11/11 — the last of these proving the correction did not swing back to the over-broad behaviour.

Next: run CI on head 79959955 and land.

3. UrlResolver — registration blocked on relay acknowledgement

https://github.com/CodexCoder21Organization/UrlResolver/pull/855 "Stop blocking global service registration on the relay acknowledgement", branch fix/register-global-service-relay-ack-latency, head 42c572e28fa6c97cf5e7c0f635f112e4e5c4e708. Pinned to foundation.url:protocol:0.0.417.

Removing the wait exposed a real ordering defect, since fixed: the initial announcement and the acknowledgement-triggered circuit-address reannouncement stamped their timestamps in independent coroutines with no ordering edge, so a fast acknowledgement plus a delayed initial coroutine let the stale circuit-less announcement stamp later and overwrite the good state. Source: src/foundation/url/resolver/UrlResolver.kt:8629-8637, with the two launches at :8667-8675 and :9789-9799.

Its CI cannot go green yet. Latest run: 1,204 passed, 4 failed of 1,208.

stressTestOnPeerRegisteredWithRelayObservationBeforeRegistry — caused by the quadratic peer-eviction defect above, not by this change. Will clear when 0.0.417's successor is published and pinned.
stressTestNonBlockingAnnouncementReachesReachablePeersUnderLoad and testNonBlockingAnnouncementStillReachesReachablePeers — unexplained. Both bypass the changed registration path entirely and prove only late-or-lost delivery at a five-second observation deadline. The analysis ruled out random fanout exclusion in the two-node fixture and any observation-before-registry breach, and stated it could not choose among queue rejection, delayed execution, or exhausted delivery without a stage-level trace. Do not modify these tests on current evidence.
testDiscoverPeersShouldEnsureNetworkJoined — appeared in the latest run only; not yet examined.

Next: after the peer-registry fix is published, bump this pull request's dependency pin, then re-assess the remaining failures.

Relevant pull requests, branches and refs

Open:

https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 — head bb5b6ee2
https://github.com/CodexCoder21Organization/ContainerNursery/pull/558 — head 79959955
https://github.com/CodexCoder21Organization/UrlResolver/pull/855 — head 42c572e2

Merged this session:

https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/430 — 2026-07-28 04:48:26 UTC. Admission wake-up and key-generation fixes; verified at 63 fresh executions, 0 failures.
https://github.com/CodexCoder21Organization/UrlProtocol/pull/434 — 2026-07-28 03:03:25 UTC. Relay-failover registration leak; published as foundation.url:protocol:0.0.417.
https://github.com/CodexCoder21Organization/ContainerNursery/pull/557 — 2026-07-28 01:36:57 UTC.
https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/423 — 2026-07-27 13:14:15 UTC.
https://github.com/CodexCoder21Organization/BuildTestCli/pull/21 — 2026-07-26 04:24:02 UTC.

Branch pushed for preservation: https://github.com/CodexCoder21Organization/BuildTestCli/tree/wip/cancel-verb-handoff-2026-07-28 — remote head 01450e0446e31a892caa0e3f8f5d7c4423624364. Adds a cancel <runId> verb calling cancelBuildRun, written because a run was wedged in TESTING and deleteBuildRun refused it with no released CLI able to cancel. Compiles, worked in practice, no tests, unreviewed. Worth contributing properly.

Published artifact: foundation.url:protocol:0.0.417 — verified byte-for-byte against the local build and confirmed by inspection to contain the relay fix. It also carries the quadratic peer-eviction regression, so its successor must include the fix from https://github.com/CodexCoder21Organization/UrlProtocol/pull/439.

Challenge records filed:

https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0104-buildtest-merge-queue-run-failed-when-its-event-ingestion.md
https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-2221-hardwarecontrolfabric-process-skill-points-to-certificate.md
https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-27-2308-containernursery-cli-cannot-inspect-logs-for-active-url.md
https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-28-0322-hardwarecontrolfabric-skill-default-certificate-paths-are.md

Working notes are in /code/workspace/codex-lanes/*.md on the session machine — briefs, per-lane reports, verification logs. These are local-only and will not survive the machine; the substance of each is summarized above. The multi-megabyte console logs were deliberately not pushed.

A retracted claim — do not act on it

An earlier report in this session claimed the merge queue was blocked org-wide by a defect in the build-test runner, and requested authorization to deploy version 0.0.75 of it. That was withdrawn. The evidence had conflated two different failures (a JSON parse error on console records containing a zero byte, and an SSH channel timeout), and the deployed-jar timestamp was repeated from a subagent without independent verification. Three runs did fail in the service's dispatch loop across three unrelated repositories between 00:40 and 00:47 UTC — genuinely infrastructure, not the changes under test — but the window closed at 00:47 and roughly fifteen clean runs followed. No deploy is needed and none was performed.

Next steps, in order

Rewrite the peer-registry performance test per the guidance at the top of this document — separate fixed from quadratic by orders of magnitude, not by a tuned threshold — then get https://github.com/CodexCoder21Organization/UrlProtocol/pull/439 green and merged.
Publish the successor to foundation.url:protocol:0.0.417 containing that fix. Confirm the new coordinate is unused before building, that the build genuinely compiled (a build rule completing in milliseconds is a stale cache hit), that the published checksum matches the local jar, and that the published binary contains the change.
Land https://github.com/CodexCoder21Organization/ContainerNursery/pull/558 — run CI on head 79959955 and merge.
Bump the dependency pin on https://github.com/CodexCoder21Organization/UrlResolver/pull/855 to the new version, then re-assess its remaining failures. Expect the peer-observation stress test to clear; the two announcement tests and testDiscoverPeersShouldEnsureNetworkJoined still need a stage-level trace before anyone touches them.
Resume the survey. Not yet verified with three concurrent builds: kompile, kompile-cli, AiCliSupervisorManagerWui. Known-suspected and unfixed: kompile-core timing margins (2 slow tests), BuildTestServerService upload-teardown, FileRelationalFilesystemExplorerWui (3 tests), AiCliSupervisorManagerEmbedded wall-clock race in its usage-hang test, BuildTestWui screenshot upload budget, and UrlResolver testResilienceHarnessFirstLaunchUsesAtomicEphemeralBind (evidenced as pre-existing on main).

Reusable knowledge

Do not write hardware-sensitive performance assertions. Structure a complexity test so the bad implementation is astronomically slower than the good one, then use a generous timeout — or assert a growth ratio across two problem sizes, or count operations. Investigating "what bound does this machine need" is the wrong branch of that decision tree.
Measuring how often a test fails is not the same as reproducing it. Two lanes here spent about eighty minutes each on branch-versus-mainline rate comparisons and produced roughly one usable sample between them. Amplifying to a deterministic reproducer, or reading the source for the mechanism, returned located root causes in thirteen and eighteen minutes — including a defect in a different repository from the one under investigation.
Deliver before the final broad sweep. Four lanes finished their real work, started an expensive full-suite verification, and hit their time limit before pushing — leaving finished work needing rescue. Push and open the pull request first; run the sweep as confirmation.
Build the artifact once and batch test selections. These repositories recompile a large script catalog on every test invocation — several minutes each. Use the documented colon-separated multi-test specification, not repeated --test options, which triggers one catalog pass per option.
Attach an equivalence obligation to every optimisation. Requiring proof that the same eviction victim is still chosen killed a wrong approach (a cache that cannot survive an insertion) and produced a test that independently recomputes the original algorithm's answer.
When a change moves a behavioural boundary, expect the tests on the other side to fail, and suspect the change first. This happened twice in one repository tonight; the first time the change was wrong, not the eleven tests.
Verifying that a published artifact contains your fix does not verify the artifact. 0.0.417 was confirmed freshly built, checksum-matched, and containing the intended methods — and still carried an unrelated quadratic regression.
A run reporting many failures but naming no failing assertion is an infrastructure failure until proven otherwise. Look for the first hard error; results marked "did not complete" with zero duration and wildly swinging counts are signatures of the reporting path breaking.