Handoff: Investigate two native first-attempt failures after the projection-close rebase

Written 2026-10-09 13:28 UTC.

RE-VERIFY: All state is a write-time snapshot. Before acting, read https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1297 with gh pr view fields state,headRefOid,mergeable,mergeStateStatus,statusCheckRollup,mergedAt and inspect the linked native Actions run. Fresh state wins.

Mission

The brief requests a semantic rebase of the projection-close PR onto main after the decoded-tail rewrite, preserving both invariants, re-gating, and leaving it for orchestrator review. The source rebase, targeted gates, authorized force-with-lease push and single body ledger are done. The bounded native CI watch is still pending near the lane's 75-minute ceiling. This handoff covers only the remaining required CI verdict; it grants no enqueue, merge, ready/draft change or deployment permission.

Verified work

1. Rebased 109 commits onto main 44842fcf0cd380c7e27d5a4381f0a7ac45cfc04e, the merged decoded-tail rewrite from https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/1343.
2. Exactly two conflicts: ProjectionChangeJournal.kt combines branch constructor failure cleanup with main decoded-tail bound validation; build.kts keeps both independent fixture rule sets. Main streamed rewriteRows, captured snapshot owners, 24-byte sequence rows, bounded tail default 256 and sealPrefix callbacks are preserved. Branch stop checks and close ordering are preserved. Range-diff and independent read-only review show no material finding.
3. Seven named regressions each passed 3/3 first attempts, 21/21 total, zero retries: deferred-baseline and released-baseline stop probes, baseline-row observer close, continuous maintenance original, and decoded-tail concurrent close/crash restart/cache write failure. Raw XML is saved remotely.
4. Contract partitions passed 105/105 and 116/116 first attempts, including all earlier 217 contracts plus four startup guards. Total 242/242 first passes, zero retries, exact selection. No source, test, iteration, timeout, pin or product edits beyond conflict resolution.
5. Existing PR was verified OPEN before authorized force-with-lease; pushed a858b77e5180ea4d3aa47a0f603fd73e0336e4b2. Appended exactly one ledger line via API PATCH, re-read byte-exact. Last live state 13:27 UTC OPEN/non-draft/MERGEABLE/BLOCKED. No enqueue/merge/deploy or draft flag change.
6. Native run is dispatched at exact new head. Four shards built successfully and remain running tests. Required bld-build depends on all four shards; its absence while they run is explained by workflow dependencies. No native assertion failure has been observed. No native pass credit is claimed before raw XML.

Remote refs and evidence

No code from this effort is deployed or merged. Source and selected raw XML, manifests, audit scripts, findings and review are on remote branches. Build outputs/caches are deliberately omitted. The lane will remove its fresh clone and stop its own bounded watcher before ending; no local path is required for pickup.

Next steps

1. Re-verify current PR head/state and native run first. If head changed, these counts apply only to a858b77e5180ea4d3aa47a0f603fd73e0336e4b2.
2. Continue report-only required-check watch with build-watchman 0.0.21: coursier launch buildwatchman:build-watchman:0.0.21 -r https://kotlin.directory -- --repo CodexCoder21Organization/BuildTestEmbedded --pr 1297 --interval-seconds 120 --idle-timeout-minutes 35, wrapped in a tracked hard deadline. Do not use --to-merged or mutate the PR.
3. On completion, download native artifacts, verify exact manifest coverage and first attempts vs retries using the evidence audit-native.py (artifact subdirectories bld-test-shard-N-test-results and bld-test-shard-N-manifest). Report full counts and any exact first failing assertion/stack trace. Native green alone does not prove first-pass cleanliness.
4. If red, diagnose conflict-resolution error first. Do not weaken tests/limits, change pins, repeatedly rerun CI, or broaden product edits under this brief; report the exact assertion to the orchestrator for scope.
5. Report whether required CI is green and first-pass clean to the orchestrator. No enqueue/merge/mark-ready/deploy permission is granted. Do not append another rebase ledger line; the authorized line already exists.

Operational notes

The watcher's missing-bld-build PROBLEM after 20 minutes is explained by needs: bld_test_shard, not evidence of lost dispatch. Shard successful medians reported by watcher are roughly 39/45/45/40 minutes; at 13:27 runtimes about 31 minutes. Installed kotlin-build-ci-test app queued suites also show zero runs on mature main commits; watcher identifies this as proven non-dispatch policy and does not rerequest. All gate builds ran on Actions, not the shared local box. Actual source launcher pin remains 0.0.112. Read the remote brief/findings/review and authoritative Testing/Philosophy links carried in the findings before any broader change.

Updated 13:31 UTC — first-pass cleanliness refuted

Native shard 1 is 1013/1013 first passes, zero retries and exact manifest coverage. Shard 4 is 1011/1013 first passes, two retry-only passes, zero final failures and exact manifest coverage. Shards 2/3 and required aggregate remain pending. The final verdict is NOT READY even if the aggregate later turns green. The two tests were outside the targeted 221-contract selection. Their files and ConstructorCleanup.kt are unchanged between the old and new PR heads; this rules out a direct conflict edit there, but does not establish the end-to-end mechanism. No guess or product change has been made.

Re-verify current CI first, then inspect the preserved full first-attempt logs/thread dumps and reproduce the exact behavior before proposing any fix. Broader edits require orchestrator scope; no test weakening, reruns to obtain green, queue action or merge. Remote evidence native/ contains raw XML/manifests and full failures.

admissionGrantPersistenceCannotOverwriteRequeue-first-stacktrace.txt

java.lang.RuntimeException: Test 'admissionGrantPersistenceCannotOverwriteRequeue' timed out after 90000ms, stuck at: buildtest.embedded.ConstructorCleanupKt.joinOwnedThread(ConstructorCleanup.kt:20)
	at kompile.WorkspaceKt.executeTestInSeparateJvm(Workspace.kt:3867)
	at kompile.WorkspaceKt.executeTestAttemptSafely(Workspace.kt:3952)
	at kompile.WorkspaceKt.access$executeTestAttemptSafely(Workspace.kt:1)
	at kompile.WorkspaceKt$prepareTests$executeAction$1.invoke$submitNext$lambda$4(Workspace.kt:5689)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

testStartupRecoveryContinuesPastFormatOneMarkerDurationHistory-first-stacktrace.txt

java.lang.AssertionError: The format-1 run must be republished from its event log exactly once: TestEventProjectionReplayCounts(started=2, bytesRead=816, completed=2, failed=0). Expected <1>, actual <2>.
	at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16)
	at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:652)
	at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11)
	at kotlin.test.Asserter$DefaultImpls.assertEquals(Assertions.kt:671)
	at kotlin.test.DefaultAsserter.assertEquals(DefaultAsserter.kt:11)
	at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:63)
	at kotlin.test.AssertionsKt.assertEquals(Unknown Source)
	at buildtest.embedded.TestStartupRecoveryContinuesPastFormatOneMarkerDurationHistoryKt.testStartupRecoveryContinuesPastFormatOneMarkerDurationHistory(testStartupRecoveryContinuesPastFormatOneMarkerDurationHistory.kt:109)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at kompile.TestRunner.executeTest(TestRunner.kt:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at community.kotlin.kompile.testrunner.bootstrap.BootstrapRunner.main(BootstrapRunner.java:416)