Handoff: Verify the PeerRegistry listener full-suite gate

Written: 2026-10-07 10:03 UTC.

RE-VERIFY: This is a write-time snapshot. Read the existing run's API pages at https://buildtest.kotlin.build/api/test-results?id=8eaa3889&page=1 through page 24, 100 rows per page; query the branch's current PR with gh pr list --repo CodexCoder21Organization/UrlProtocol --head fix/peer-registry-change-listeners --state all.

Mission summary

Finish verification of the upstream PeerRegistry listener API, then review its draft PR. This worker is ending within its 90-minute phase time box. The full remote suite is incomplete and result pages repeatedly return HTTP 503. The worker opens the draft PR as its last phase action after this snapshot; discover its URL from the branch query above. Do not merge, enqueue, publish, deploy, or change UrlResolver under this phase.

What was found and done

The resolver finding is https://github.com/CodexCoder21Organization/UrlResolver/pull/1205 and its deviation is on https://github.com/CodexCoder21Organization/UrlResolver/tree/work/fx1205r-round12. Its pinned and latest published foundation.url:protocol artifacts expose no listener. The upstream implementation now reports mutation kind and peer after published state, with idempotent identity registration/removal, warning effects after sibling delivery, and deferral across enclosing registry/lifetime operations so nested peer-property, persistence and clock calls cannot notify under a registry lock.

Twelve public-API tests passed ten consecutive final-head runs: 120/120 cases, zero failures. Eight initial cases failed against the missing API; publication ordering and three nested lock probes failed against earlier implementations. Testing and Philosophy documents were read: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md . No mocks, reflection, sleeps, retries, timeout increases, or stress reductions were used.

Relevant PRs / refs

Source checkpoint: https://github.com/CodexCoder21Organization/UrlProtocol/commit/07d5896d4ad4589a9d31754e975adbe2d8cc2a72 . All code is pushed; working tree, stashes and unpushed-commit checks are empty. UrlResolver was read-only and has no worker changes. Nothing was deployed or published; no production or Maven-publish command was run. Local generated test logs/XML are excluded from source commits under the lane's artifact rule; measured results and failure mechanisms are recorded below, and all reproducer code is on the remote branch.

Next steps

1. Re-verify current state first. Follow existing raw run 8eaa3889 without deleting or re-running it. A read-only watcher can use /home/u/bin/cs launch buildwatchman:build-watchman:0.0.18 -r https://kotlin.directory -- --run 8eaa3889, under a finite deadline. First-page watcher counts are not whole-suite counts.
2. Page the authoritative API through totalCount (2349, 24 pages, at most 100 rows/page). Last read retrieved 11 pages/1049 rows: 156 passed, 4 running, 889 pending; 13 pages returned HTTP 503. Do not infer a code verdict from absent counts, client timeout, or WUI cached-view errors.
3. Review the branch's draft PR after full-suite verification. Supervisor owns every merge decision. No publication or downstream change is authorized in this phase.

Reusable knowledge and measured findings

OBSERVED: Started upstream PeerRegistry listener phase. No consumer changes, publishing, deployment, merge, queue operations, or subagents are allowed. INFER: Plan: (1) locate upstream and read consumer/docs; (2) record invariants and write failing public API tests; (3) implement minimal listener API and README; (4) prove new tests 10/10 and full suite once; (5) review, rebase, checkpoint, open PR and stop. Step 1 is in progress. OBSERVED: Upstream is https://github.com/CodexCoder21Organization/UrlProtocol, artifact foundation.url:protocol. No repo AGENTS.md exists. Read required Testing and Philosophy documents, upstream README peer registry/persistence/lifetime sections, resolver round-twelve candidate refresh helper and service snapshot reader. Prior consumer deviation is https://github.com/CodexCoder21Organization/UrlResolver/tree/work/fx1205r-round12 (also in the PR body round 11); exact deviation is recorded in work/fx1205r-round12. INFER: The mechanism is that PeerRegistry mutates membership and publishes snapshots without any change callback, and the evidence is its addValidatedPeer/removePeerMembership paths plus the consumer candidate wait subscribing only to resolver-owned signals. Invariant table: listener subscription is caller-owned, identity-idempotent and removed explicitly; mutation captures eligible subscriptions and published affected peer; dispatch runs after all mutation/publication/dial/lifetime locks release; throwing listeners report specific algebraic warnings after sibling delivery; new listeners see later mutations only; expiry and each eviction carry their reason; refresh inside runIfActive defers delivery until the outer action releases its lock. Step 1 complete, step 2 tests written and baseline compilation is next. OBSERVED: Eight fail-first public contract tests are checkpointed at https://github.com/CodexCoder21Organization/UrlProtocol/commit/3959342cd5bdc2cf176c2f5921a04866a3f9ea76 on https://github.com/CodexCoder21Organization/UrlProtocol/tree/fix/peer-registry-change-listeners. Remote baseline connects to buildtest but has not yet returned a run ID; local targeted baseline is compiling the upstream artifact. Implementation has been prepared outside the repository but is not applied until baseline fails. No tests or bounds weakened. Full upstream suite contains 2308 existing scripts, making the full-suite gate potentially significant within the hard time box. OBSERVED: Local baseline deterministically fails compiling the public listener mutation test: unresolved PeerRegistryMutationKind, addChangeListener, and PeerRegistryChangeListener; verdict 0/1 successful, 1 failed. Full compiler stack is saved at /tmp/fl-listener-baseline-local.log. Remaining seven public scenarios are now compiling against unchanged upstream source to record each fail-first verdict. OBSERVED: All eight baseline scenarios fail (0/8 successful, 8 compilation failures) solely because the requested public listener API is absent. Logs /tmp/fl-listener-baseline-local.log and /tmp/fl-listener-baseline-all.log preserve complete compiler stacks. Listener implementation applied only after these failures. Step 2 complete; step 3 in progress. INFER: Review of the first implementation found a subscription gap if listeners were captured before public snapshot publication: a subscriber could read the old snapshot yet miss the eventual signal. Listener eligibility is now captured at snapshot publication under the capacity lock, and registration uses that same lock. Pending closed-batch events receive their eligible subscriptions with the published batch; callbacks remain outside the locks. This also prevents listeners added by an eviction callback from receiving an earlier insertion event. Targeted fixed run is compiling; no fixed verdict credited yet. OBSERVED: First fixed candidate ran 8/8 PASS, but final gate does not credit that earlier source. Final implementation now has publication-time listener snapshots, subscription/publication coordination, deferred active-action delivery, exact mutation reasons, and warning delivery after healthy siblings. The ninth deterministic subscription-before-snapshot probe has been added and its published-artifact API-absence baseline is still compiling. Source changes preserve stable peer IDs for warning formatting, so an affected peer getter cannot prevent sibling listeners being called. No consumer, build-tool, artifact-version, or production edits. INFER: Independent test review found the original eight scenarios cover every stated mutation path and registration/removal rule; the ninth closes the subscription/read ordering required by the consumer. Independent code review found the pre-publication subscription gap and corrected it before the final verification gate. Report-challenge skill was read, but its automatic enqueue/merge contradicts this lane prohibition, so no challenge CLI is invoked; remote client silence is recorded locally until a specific service verdict is available. OBSERVED: Second candidate passes 9/9 scenarios, including the deterministic subscription window. A trial baseline that merely changed WithArtifact to protocol:0.0.603 also passed because the runner supplied the current root module; that result is not credited as a baseline. The ninth probe is now running against a restored first implementation that captured listeners before publication, to obtain an assertion-level red proof. Its original eight missing-API baselines remain valid. Temporary baseline test file has been removed. Final source is saved separately while this deliberate reversal runs. OBSERVED: Final source/tests committed locally at d7f0b8c3f89f39b34a5d71a6de13c628d9fa02b0 after latest-main rebase. One full remote suite began at 09:02 UTC with a 40-minute external deadline; no other full suite was started by this lane. The nine-scenario repeat gate is running ten times, stopping immediately on any failure, with exact XML per run. Final candidate callbacks avoid retaining deferred work when no eligible listeners exist. Step 3 complete; step 4 verification in progress. Source commit is not pushed until the shared-contract full-suite gate is observed. OBSERVED: The additional nested-mutation lock probe fails reliably at a public assertion (expected 3 completed callbacks, actual 2) on candidate d7f0b8c3. A peer property makes real nested addPeer calls; one nested listener is invoked while the outer compute holds the parent mutation stripe, so its genuine worker recordDialSuccess(parent) cannot complete until the callback has returned. Its failure is already reported as the specific warning; reporting alone does not close the lock invariant. Full stack /tmp/fl-listener-nested-red.log and XML /tmp/fl-listener-nested-red.xml. The earlier nine-scenario repeat driver was stopped and its partial successes will not be counted as the final gate. A parallel public in-memory persistence probe is being checked before implementation. INFER: The mechanism is per-operation notification release without respecting enclosing registry operations, and the evidence is the deterministic nested public-mutation assertion plus the parent compute stripe held across the nested add. Fix requires one per-thread outer operation notification scope, covering add/removal/dial operations, lifecycle actions, persistence and quarantine reads that call the injected clock. The full suite already running belongs to the prior candidate and cannot establish final-source coverage after this required correction.

OBSERVED: Independent review found three deterministic lock re-entry failures in candidate d7f0b8: a caller peer address getter, an in-memory filesystem atomicMove callback, and an injected clock during isQuarantined each performed a genuine nested add; their change listeners could not complete a same-registry worker operation until the outer lock was released. The corrected quarantine probe failed 0/1 with expected callback count 1, actual 0; full stack is /tmp/fl-listener-quarantine-red-2.log. INFER: The mechanism is notifying a nested mutation before the enclosing registry/lifetime operation releases its locks, and the evidence is these three public-API probes failing consistently on the prior candidate. The replacement uses a per-thread enclosing-operation scope, removes it before draining callbacks, and preserves all queued deliveries and original failures. The 12-test targeted build is now running. The earlier remote full-suite candidate is TESTING; it does not verify this corrected tree.

OBSERVED: Corrected enclosing-operation scope passes all 12/12 public listener scenarios. Implementation and three additional red-first re-entry probes are committed locally as 07d5896d4ad4589a9d31754e975adbe2d8cc2a72. Ten-run gate started from that tree. Initial gate driver mistakenly expected JUnit testcase tags; the kompile report uses test tags, so the parser was corrected before relying on its count verdict. Test results themselves are read from their XML. OBSERVED: Stopped only our superseded full-suite client tree (PIDs 92864, 92856, 92855, 92853) at 09:31 UTC; no remote run, service, supervisor process or queue entry was stopped. That candidate's incomplete run has no verdict and is not credited. Started one final-head full remote suite with a 30-minute external deadline. API run index is stale (outOfDate=true, refreshFailing=true), so no inference from its missing new runs is credited. INFER: Independent test and code review now covers all stated invariants, subscription-before-snapshot ordering, nested peer-property/persistence/clock operations, exact warning text and sibling delivery, and preservation of operation failures. Required Testing and Philosophy documents were read. Step 3 complete; step 4 final repeat/full gates pending; step 5 PR pending.

OBSERVED: Final committed-head gate currently has 3/10 passes, 12/12 cases per pass, with no test failures. The initial gate XML parser correction is a harness-only correction; run one is independently confirmed by its 12 success=true test entries, and runs two onward use the correct parser. Final full remote suite progressed PROVISIONING → UPLOADING → BUILDING at 09:35 UTC. Head remains 07d5896d4ad4589a9d31754e975adbe2d8cc2a72. Branch working tree is clean and includes only README, registry source/API, and 12 test files; no build artifacts are tracked. INFER: Invariant closure: membership/address paths → Mutations, ImportAndEviction, Capacity, Expiry; post-publication visibility and subscriber read ordering → Mutations and SubscribeBeforeSnapshot; locks/re-entry → OutsideLocks, NestedMutationOutsideLocks, PersistenceOutsideLocks and QuarantineReadOutsideLocks; subscription identity/removal during delivery → Registration; thrown listener failures and typed warning text → Failure; genuine concurrent exact-count delivery → ConcurrentMutations. All rows have fail-first evidence in earlier recorded baselines.

OBSERVED: A read-only API schema lookup for an older completed UrlProtocol run at https://buildtest.kotlin.build/api/test-results?id=044d54c5&page=1 returned curl exit 22, HTTP 503. This is not our run and is not a verdict on our code. The final full-suite client still reports BUILDING; it has not returned an ID or counts. Repeat gate is 5/10, each 12/12. Meaningful results API friction is recorded here because the report-challenge CLI would automatically enqueue/merge, which this lane forbids.

OBSERVED: Final repeat gate complete: 10/10 runs, each 12/12, aggregate 120/120 PASS and zero failures. Independently checked every /tmp/fl-listener-final-{1..10}.xml success attribute and count. Exact head unchanged at 07d5896d4ad4589a9d31754e975adbe2d8cc2a72. Source is now checkpointed remotely at https://github.com/CodexCoder21Organization/UrlProtocol/commit/07d5896d4ad4589a9d31754e975adbe2d8cc2a72 on https://github.com/CodexCoder21Organization/UrlProtocol/tree/fix/peer-registry-change-listeners after latest-main rebase. No PR exists on this branch yet. OBSERVED: The full suite was started once from this exact head before the checkpoint push, but its verdict is still pending (BUILDING). Push is a WIP durability checkpoint under the lane checkpoint rule; it does not claim the full-suite gate has passed. No CI rerun, merge, enqueue, publishing, deployment or consumer change. Step 4 repeats complete; full counts pending; step 5 PR pending.

OBSERVED: Read-only run index now includes raw UrlProtocol run 8eaa3889, started 09:31:24 UTC, project foundation.url:protocol, 2349 tests. Prior raw run 9f9a3bd1 started 09:02:39 UTC and has 2346 tests; the exact +3 matches the three additional files added between our requests. INFER: 8eaa3889 is our final-head request; metadata has no commit label, so identity is correlated by start time, raw label, project and exact test inventory delta. OBSERVED: https://buildtest.kotlin.build/api/test-results?id=8eaa3889&page=1 returned HTTP 200 with totalCount=2349, totalPages=24, limit=100, complete=false. Attempt to page all 24 pages at limit=100 failed on HTTP 503; complete Python stack is /tmp/fl-full-api-counts.log. Watch-build 0.0.18 raw-run watcher is active as a tracked task with a 14-minute external bound, no --to-merged, and reports HTTP 503 because the WUI is still building the cached test-results view since process start. The full test client remains TESTING. No full-suite counts verdict is credited.

OBSERVED: Watchman stayed alive after the cache-read 503 and now reports TESTING progress at 12/100 rows on its first result page. That is not a repository suite count; the gate still requires all 24 pages and 2349 rows. After this observed API recovery, a fresh paginated read (no new test run) is checking the whole inventory and any failed rows. Full-suite client remains TESTING.

OBSERVED: At 09:55 UTC the full client still reports TESTING. Watchman has alternated first-page progress (12/100) with HTTP 503 cached-view failures. Both paginated full-inventory reads failed on HTTP 503; only pages 6 and 7 were recovered, with 200 rows total, not a whole-suite count (page 6: 2 PASSED/98 PENDING; page 7: 2 RUNNING/98 PENDING at their read time). These partial rows are not used as a gate verdict. No test rerun or parameter change was made. INFER: Remaining phase work is solely the full-suite gate and PR opening. If the bounded run cannot establish an authoritative complete 2349-row verdict before this lane time box, the review result must be BLOCKED with a draft PR, rather than claiming DONE from the completed 120-case repeat gate. All source, tests and README are recoverable from the pushed branch; the supervisor can continue the existing raw run 8eaa3889 without re-running or changing this worker's code.