Latest lane snapshot — 2026-10-06 05:05:19 UTC

RE-VERIFY: This snapshot was checked live on October 6. Recheck gh PR state, git ls-remote, ContainerNursery containers/list-jars, and public health/status before acting. The earlier body below is historical and its current-main SHA has moved.

OBSERVED: Final lane report, 2026-10-06 05:05:19 UTC. Request: drive hf-2026-09-30-finish-the-build-watchman-token-refresh-so-long-lived-watchers-stop-failing-with-401-bad-credentials to LANDABLE, OUTDATED, or BLOCKED, then stop. Outcome: BLOCKED. The one missing artifact is an approved production rollout record naming a post-fix kotlin-build-ci source commit and the deployed binary hash. Deployment and restarts are expressly prohibited in this lane. No Maven publication is required.

OBSERVED: Both code PRs are already MERGED and their original check runs are successful. https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/295 merged 2026-10-01T00:22:14Z as https://github.com/CodexCoder21Organization/kotlin-build-ci/commit/5ee546f20bfacae83e73400be4156c2a4d9ce66a; bld-build SUCCESS. https://github.com/CodexCoder21Organization/build-watchman/pull/14 merged 2026-09-30T21:59:47Z as https://github.com/CodexCoder21Organization/build-watchman/commit/0c891fe61b1cf8369b04e2c171d3b18b0846424c; kotlin.build (remote) SUCCESS and kotlin.build (kompile-remote-build) SUCCESS. No PR was created or updated by this lane, and there is no token-expiry PR left to land. These are live gh results, not newly executed tests.

OBSERVED: Fresh clones plus git ls-remote show consumer main https://github.com/CodexCoder21Organization/kotlin-build-ci/commit/e3f692ba148d2d46b74cb59dd395f85dd20e230d and upstream main https://github.com/CodexCoder21Organization/build-watchman/commit/d9b3777811779b24385e6e33f2471d3eb2629487. git merge-base --is-ancestor succeeds for both original merge commits against their current main. Consumer's original and checkpoint branches still point to 79c8a4bdedbb948911fd00a14e8bb47d3aa986a1. Upstream original and checkpoint branches still point to 54aca89ff5094f350f4315cffd4d50d869f7c3e4. The prior evidence branch still points to https://github.com/CodexCoder21Organization/kotlin-build-ci/tree/wip/L18-token-refresh-rollout-evidence at 6bd83034e9b8fa8ebf05c6d8890e172301f6a533. No branch was altered or deleted.

OBSERVED: The merged service creates each retained watcher with an installation-token provider. The library reads that provider for each GitHub request, allowing the existing 55-minute cache to replace the token before nominal one-hour expiry. Requests GitHub rejects continue to report a problem; the expiry fix does not add a retry or change a timeout. Consumer main retains the provider factory and its library 0.0.7 dependency.

INFER: The mechanism is a retained watcher continuing to send its original installation-token string after expiry, and the evidence is the old-to-merged provider change verified against current main in both repositories and their recorded successful checks. This lane did not newly reproduce the defect or measure production recurrence. The code work is already on main, but declaring the whole handoff OUTDATED would discard its still-unverified rollout and across-expiry production verification.

OBSERVED: Read-only ContainerNursery containers and list-jars succeed. Route https:githubci.kotlin.build:443 is RUNNING at host_port 42515 with image /root/ContainerNursery-uploads/jars/kotlin-build-ci-84213905-20260929.jar. The configured file is 68186826 bytes with last_modified 1790714704201, 2026-09-29 20:45:04 UTC. No listed service JAR is newer than the fix. Public https://githubci.kotlin.build/health and https://githubci.kotlin.build/api/status report 0.0.107. Main's HealthServlet.kt hard-codes that value, so the version cannot identify the deployed commit. No binary hash was exposed or read; filename and timestamp support a missing rollout without proving binary contents. A changed port since the prior report does not establish a new binary.

OBSERVED: Prior reports were read from the full handoff body and from the evidence branch's investigation/L18-token-refresh-rollout/README.md and review.md. The CLI reports list returned []. The prior lane already performed the source review and documented why rollout remained unverified. This lane ran zero tests, reverted no fixes, made no code changes, and did not repeat the LANDABLE review gate for already merged PRs. Fail-first and five local runs per new test are not claimed. Both clones are clean with no stash or local-only commits; the report itself is preserved in the existing Handoff service record. No build outputs were created.

OBSERVED: Delegated work: none; no other agents or codex processes were started. Landing parallelization sweep: there is no outstanding landing gate for this handoff; both PRs are merged. No merge, enqueue, dequeue, deployment, restart, Maven publication, CI rerun, timeout increase, or test weakening occurred. No excluded repository was changed.

OBSERVED: Plan complete for this lane: handoff read and claimed; live code, PR, branch, and rollout evidence checked; BLOCKED report uploaded. Next step for the supervisor/operator: arrange an explicitly approved deployment from an appropriate current main that contains the fix, preserve its source SHA and binary hash, then verify one retained unchanged-head watcher beyond 60 minutes before completing the handoff. The supervisor decides completion; this lane does not complete or archive it.

OBSERVED: Lesson: successful source checks and a merged commit do not establish which binary is serving production; identify deployment by source SHA and binary hash. Lesson: a hard-coded health version and a reused JAR filename cannot independently establish rollout identity. Lesson: a restart can replace cached watchers, so immediate quiet does not verify behavior across token expiry.

STATUS: BLOCKED Approved deployment record for a post-fix kotlin-build-ci binary is missing; both code PRs are already merged and verified on main.

Prior handoff body (historical snapshot retained in full)

Handoff: Request the merged token-refresh rollout and verify a kept watcher across expiry

RE-VERIFY — written 2026-10-04 14:48 UTC. This is a write-time snapshot. Recheck both PRs with gh pr view <full URL> --json state,headRefOid,mergedAt,statusCheckRollup; remote branches with git ls-remote; public health and /api/status; ContainerNursery containers --json and list-jars --json. The configured image filename may be reused. The health version is hard-coded, so preserve the built SHA and binary hash in any approved deployment record.

Mission and remaining request

The original mission is to finish build-watchman token refresh so long-lived watchers stop failing with 401 Bad credentials. Both library and consumer code are merged. Operator approval to deploy the merged consumer, followed by verification of a kept unchanged-head watcher beyond one hour, remains. This lane was assigned read-only rollout verification and may not merge, enqueue, deploy, publish, complete the handoff, or release its claim. Its terminal recommendation is NEEDS_USER, not completion.

What was found and done

1. Live https://github.com/CodexCoder21Organization/kotlin-build-ci/pull/295 is MERGED at 2026-10-01T00:22:14Z; head 79c8a4bdedbb948911fd00a14e8bb47d3aa986a1; squash merge and current main 5ee546f20bfacae83e73400be4156c2a4d9ce66a. bld-build SUCCESS. Source at 84213905 used one githubToken at watcher creation. Main now calls createPullRequestWatchman with GitHubApi::getInstallationToken. The factory supplies githubTokenProvider for each request, reading InstallationTokenCache with a 55-minute TTL. This removes the stale-token mechanism without retrying rejected requests. The original 234 error lines in three minutes on September 30 are historical PR-author evidence, not a newly measured count.
2. Live https://github.com/CodexCoder21Organization/build-watchman/pull/14 is MERGED at 2026-09-30T21:59:47Z, head 54aca89ff5094f350f4315cffd4d50d869f7c3e4; merge 0c891fe61b1cf8369b04e2c171d3b18b0846424c. Both kotlin.build checks SUCCESS. Current upstream main is d9b3777811779b24385e6e33f2471d3eb2629487. Consumer main pins buildwatchman:build-watchman-library:0.0.7 and declares kotlinbuild.ci:kotlin-build-ci:0.0.108. Published library 0.0.7 was previously verified available; no artifact was published by this lane.
3. Public https://githubci.kotlin.build/health returned HTTP 200 and 0.0.107 at 2026-10-04 14:37:36 UTC. Public /api/status also reports 0.0.107. Main's HealthServlet.kt still hard-codes 0.0.107 despite the 0.0.108 build coordinate. These versions cannot identify the live source. GitHub deployments API returned an empty list.
4. Fresh read-only ContainerNursery containers --json reports image /root/ContainerNursery-uploads/jars/kotlin-build-ci-84213905-20260929.jar, host_port 38107. list-jars --json reports that exact configured file at 68186826 bytes, last_modified 1790714704201, human time 2026-09-29 20:45:04 UTC. No newer kotlin-build-ci service JAR is listed. The configured file's timestamp is before the source fix written September 30 23:52 UTC and merged October 1 00:22 UTC. Filename plus pre-fix timestamp strongly supports a missing rollout. No public binary hash was available, so exact binary contents are not proved; do not describe this as a byte-level comparison.
5. The bounded container-logs --route-key https:githubci.kotlin.build:443 --lines 500 read exited 124 after 90 seconds with zero stdout/stderr bytes. This does not prove empty logs or stopped errors. Local scans covered 891 .log files including 16 watch logs: zero actual HTTP 401/Bad credentials error phrases. Eight post-merge GitHub check outputs and annotation lists across build-watchman, UrlResolver and FileMetadataCache contained no such error; all annotations were empty. Four were October 1 successful build-watchman checks (180/180 and 175/175), and two were current October 4 FileMetadataCache checks. Samples do not establish the production reconciler's state or cover one kept watcher beyond token expiry. CLI watchers usually use a different token source. Current recurrence remains unknown.
6. Dedicated test-comprehensiveness and adversarial source reviews are recorded in the lane findings and durable evidence branch. The consumer test runs real reconciler/cache/watcher wiring against an in-process HTTP GitHub stand-in, advances ManualClock 61 minutes, verifies the same watcher and every refreshed header, and counts one replacement mint. Upstream also covers rotation within a poll, provider exceptions and loud rejected requests without retry. The consumer PR author reports a fail-first control and 4/4 targeted passes; previous handoff reports 1029/1029 historically. This lane ran zero new tests and did not rerun CI. Source review supports the expiry mechanism, but source tests cannot establish production rollout.
7. Salvage: consumer provider/factory and four regression tests are on main. Entire consumer source branches in the table equal main by content. Upstream original/provider checkpoint branches equal the merged upstream tree. W61's four changed files carry the same provider contract and rotation scenario, now covered more fully by upstream main's shared authorization path and four tests. No unique code needs salvage for this mission. No PR was closed and no branch deleted. Separate early-rejection cache branches and their handoffs remain untouched.

Relevant PRs / refs

No token-related open PRs were found in either repository. No implementation work remains unpushed. Both fresh clones had no stash or local-only commits; the consumer's only new branch is the evidence checkpoint. No production deployment or Maven publication occurred. No deployed-but-unmerged code was identified from the public metadata; exact live binary identity remains unknown. All observations needed to resume are in this body and the linked evidence branch.

Exact next steps

1. Re-verify current main, public route/JAR metadata and any newer approved deployment first.
2. Operator decision: authorize building and deploying the merged service from main 5ee546f20bfacae83e73400be4156c2a4d9ce66a (artifact build coordinate 0.0.108, watcher library 0.0.7), or provide a trusted newer deployment record identifying the built source SHA and binary hash. Recommend deployment because the configured file still has a pre-fix modification time. A restart alone does not roll out the code. Do not publish a Maven artifact for this mission.
3. After deployment is explicitly requested, use a fresh approved checkout, fetch/rebase current main, record source SHA, build scripts/build.bash --remote kotlinbuild.ci.buildFatJar output/kotlin-build-ci-token-refresh.jar, and record sha256sum. Deploy using the container-nursery-deploy skill to existing route https:githubci.kotlin.build:443; preserve route settings and record the uploaded hash. upload-jar overwrites the configured file and restarts it, so the old filename will then cease to identify contents.
4. Confirm real HTTP health, preserve source/hash/upload/startup evidence, and obtain a bounded log read using the skill's CLI. Observe one existing cached watcher with unchanged PR head for more than 60 minutes. Verify successful GitHub reads before and after expiry and absence of that watcher's HTTP 401/Bad credentials PROBLEM lines. Immediate quiet after restart is not sufficient because restart replaces cached watchers. Do not manufacture production test traffic.
5. Record deployment identity, observed watcher, unchanged head, start/end times and outcomes in this handoff. Supervisor alone decides completion/archive and branch cleanup.

Operational knowledge

• Every shell on this host needs export PATH=/code/ws/bin:$PATH. cs is the working JVM Coursier wrapper. /usr/local/bin/coursier is the wrong architecture. Repositories calling coursier may need a clone-local executable shim that runs java -jar /code/ws/bin/coursier.jar "$@"; do not alter shared binaries.
• Read-only metadata commands: timeout 90 cs launch containernurserycli:container-nursery-cli:0.0.20 -r https://kotlin.directory -- containers --json --url https://api.nursery.wasmserver.com and the same launcher with list-jars --json. Select only the githubci row and avoid printing environment values. Single-container CLI lookup previously returned Endpoint not found: GET /containers/https%3Agithubci.kotlin.build%3A443; use the list.
• The 90-second log-read deadline is a command limit, not a changed production/test timeout. No log output on deadline cannot establish that errors stopped.
• github-repos search initially chose an expired ambient GITHUB_TOKEN while gh uses working GH_TOKEN. It succeeded when given gh's token internally without printing it. rg is unavailable on this host; plain grep/Python were used. The challenge recorder auto-merges a PR, conflicting with this lane's no-merge rule, so these facts are retained here instead.
• The repository README's GitHub-runner-only policy explains the absence of a kotlin.build (remote) check on the consumer PR. There was no red remote check to repair; none was invented for an already merged PR.
• The optional CLI release and separate rejected-token invalidation work are explicitly outside this rollout mission. Nominal expiry refresh is covered here; early rejection before the cache's TTL is a separate case. Do not discard those separate branches or edit their handoffs based on this snapshot.
• Read: PlanRepository handoffs/README.md, especially Triaging handoffs and Format; DocumentationRepository PHILOSOPHY.md, architecture/TESTING.md, and CODE_REVIEW.md; both repository READMEs. Neither clone has AGENTS.md. These require verifying hypotheses, preserving evidence, and avoiding mitigations around a source bug.