Handoff: Diagnose compiler timings and finish failed kompile-core review gates

Updated 2026-10-04 12:52 UTC. RE-VERIFY: This is a write-time snapshot. Before acting, run gh pr view https://github.com/CodexCoder21Organization/kompile-core/pull/361 --json state,isDraft,headRefOid,statusCheckRollup,mergedAt and fetch current main. Read the preserved findings and full failure traces; no timeout cause is established.

Mission

The owner asked that "shared caches must have fast, reliable locking internally so that serializing externally is never necessary; where that does not hold, we need concrete failure reproductions". F361d was assigned to close review of https://github.com/CodexCoder21Organization/kompile-core/pull/361: audit snapshot ordering, convert three witnesses, restore detailed errors, qualify two older tests, measure stamp work, update the draft PR and obtain green branch CI. Historical local gates passed, but the newer independent review refutes review completion; green CI is not achieved. The supervisor owns every merge, enqueue, deployment and handoff-completion decision. L2 stops at the explicit approximately2.5-hour CHECKPOINT limit. The original seven timing symptoms reproduce, but their named mechanism and a corrected-candidate gate remain unproved.

Required reading: https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/PHILOSOPHY.md and https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md, plus README and AGENTS if present. No repo AGENTS existed when checked. Clone fresh with gh repo clone into a new workspace directory; do not rely on this job's local disk.

What was found and done

1. RV361-A's stronger through-return premise was refuted. The three witnesses edit an earlier input after its own final check. Every hit/publication can still correspond to a common instant between the last pre-read stamp and the first final check. The call-site proof covers definition cache, legacy cache, index/result-index reads and publication, and found no ordering defect at the audited head. README and PR describe snapshot consistency and equal-stamp limits, including same-size edits within timestamp granularity, mapped writes and missing-input create/read/remove. Converted public tests assert the racing value, next-call observation, body counts and published entries read back. Their hooks now use actual final-check order; the original hook failed 10/10 under forced opposite map iteration, and fixed controls passed 60/60 independent runs.

2. RV361-E was fixed by validating original result types/shapes/positions before normalization. The exact main error witness and expanded nested/null/URL/primitive-array cases fail old code and pass the corrected source, including cleanup and same-key recovery.

3. Stamp observations now obtain Unix kind/size/full-resolution mtime/ctime/file key together. A public single-input counter changed from four attribute calls to two, fail-first 3/3. The 2,000-file/five-warm-call workload counted main 32,000, original PR 48,000 and bulk head 38,000 attribute reads per warm call. Uninstrumented medians were main 224.342 ms, original 300.235 ms, bulk 304.180 ms. This proves reduced metadata work, not improved wall time or the cause of CI timeouts. The old five failing-shard names were all absent on main, so no main duration exists for them.

4. Digest/provider and provenance tests retain exact failing-base qualification: https://github.com/CodexCoder21Organization/kompile-core/commit/9355509c0c6b9feee18f2014040468c1aa1d9225. Digest returned first instead of second there; it passes current main. Provenance held one value.lock after IOException there; current main fails earlier because it lacks the missing record, so that is not the same claimed reproduction. Tests were not deleted.

5. Main https://github.com/CodexCoder21Organization/kompile-core/commit/8c63643a129694ea1a45dab0c0a8b8cf9c4ea999 landed during the first clean suite. Rebase lost basename behavior because this PR moved normalization. Two unchanged main tests failed (renamed Path hit; Second.kt yielded FirstKt.class). Phase A now owns the basename once and definition regular-file keys include it. A link-name control passes main, fails the initial port, and passes the fix. It covers direct/chained file links through File, Array, List, Set, Map value/key, plus directory-link relocation. Generic/legacy/location-preserving keys retain behavior, with no added stat/read/callback/JSON parse. Main namespace v2 and all six main tests remain. That historical clean-review claim is refuted by the later RV361 review; see finding10.

6. Final source 7c13a5e28a7328f9308e0f3622d9ffccaf2741b9 passed 1,174/1,174 local tests, zero failures/retries. Fetch/rebase preserved that exact head before source and draft-PR pushes. Prior 1,167-test results are separately qualified as pre-main-update evidence. Source tree is clean; investigations/build outputs are absent from the PR diff. The PR is OPEN and draft, with WHY-first description and the required footer.

7. CI run https://github.com/CodexCoder21Organization/kompile-core/actions/runs/37190538308 ended red. Fat-JAR and shards 1/2 passed; shards 3/4 and aggregate failed. Shard 3 https://github.com/CodexCoder21Organization/kompile-core/actions/runs/37190538308/job/111401582276 timed out at 30 seconds in testWarmInputContentReadsStayWithinMainBudget, testCompiledDependencyEditAfterConsumptionCannotServeOldClasses, testExplicitDirectoryBuildPrimesPreparedDependencyAfterReopen. Shard 4 https://github.com/CodexCoder21Organization/kompile-core/actions/runs/37190538308/job/111401582405 timed out in testMavenChildSourceEditDuringBuildInvalidatesParent, testExplicitBuildFailureIncludesTestResults, testExplicitBuildSuccessMatchesExecute, testWarmManyInputBuildMeasurement. Full traces span hashing/realpath, Kotlin compilation/classloading and nested JVM launch. They do not establish a common cause. Three test files are byte-identical to main (explicit-directory, explicit-failure-results, explicit-success); four are PR-added. Recent successful main run https://github.com/CodexCoder21Organization/kompile-core/actions/runs/37186761612 includes those three, but successful logs have no per-test durations and artifacts are manifests only; do not infer duration from job elapsed time.

8. Three subsequent unchanged local targeted rounds of the shard-3 tests passed 9/9, 6,998–8,416 ms, with Temurin 21.0.12.1 matching CI. That is not a fail-first or reliable reproduction. No timeout cause is established and no speculative fix was made. Do not treat host load as a root cause. No CI was rerun. All limits, iteration counts and test behavior remain unchanged.

9. L2 independently reproduced the seven original30-second timing symptoms through genuine selected public scenarios on candidate7c13 rebased to main32761. Runs https://buildtest.kotlin.build/run?id=54aa67d8 and https://buildtest.kotlin.build/run?id=27deb650 have0PASS/7FAIL each, exactlyoneattempt pertest:14/14 failures. Every new main thread is inside Kotlin compilation; no lock cycle or common named defect is proved. These remote children use Ubuntu OpenJDK21/C2, so the older client's C1 cap is not necessary. Complete historicalCLI0.119/C2 jobs from https://github.com/CodexCoder21Organization/kompile-core/actions/runs/37132755924 also show the three unchanged victims failing before this snapshot implementation. C1-only and snapshot-only remedies are refuted for that inherited class. A selected warm-input singleton passed1/1 in25.178s; this is not a fix gate.

10. New review https://github.com/CodexCoder21Organization/kompile-core/tree/wip/RV361-2026-10-04 refutes clean review: completing-parent input propagation, legacy error context, worker protocol omissions and fixed-output writes have concrete witnesses. The separate F361e lane saved count/callback/legacy fixes and a recoverable patch, with31 eventual targeted passes and15/15 final controls. Its full remote suite d3fc4abd was canceled by the published client's30-minute wait:25PASS/0FAIL/1164CANCELED, no new candidate witness ran. It did not update PR source. F361f recovered candidatebe32ee9ad415fdf2bed7cfb0b1479e82d7b04834 and its explicitly authorized local full suite failed1184PASS/7FAIL/1191TOTAL. It stopped without pushing source or dispatching candidate CI; inspect its evidence before another suite. Do not duplicate or overwrite those lanes' source. The two worker key/value-only rows remain explicitly excluded, not fixed.

11. L2 found a new source-review hypothesis: target normalization removes .. before the filesystem resolves an earlier link in the target. Real filesystem and a public-cache program against published manager0.0.203 demonstrate caller targetactual/source versus normalized targetrouting/source; the main library correctly invalidates after actual bytes change. Candidate captureInputManifest and rememberInputStamps normalize the target, and F361e's new source projection consumes those records. A public declared-input witness is authored at50bf5727 but not yet executed successfully against candidate. Do not claim a confirmed candidate bug or write a fix before its fail-first result. It is unrelated to the seven compiler timing symptoms until evidence says otherwise.

12. The new candidate test's first submission never dispatched: pinnedCLI0.112 archived the primary clone's403MB .git object pack into an in-memory compressed byte array and ran out of512MB client heap. Current source excludes .git and streams to a temporary file; use an identical worktree with a small .git pointer for this old client. The thin-worktree targeted attempt ended exit124 at its1800s outer cap around12:46UTC, with no visible run ID/XML. Its24-minute client dump waits for getBuildRun RPC in Libp2pRpcStream.sendRequest; remote result/cancellation remains unknown. The current-main control was found by its actual labelL2-main-control, notkompile-core: https://buildtest.kotlin.build/run?id=cd2a94c9. Authoritative stateCANCELED/no droplet/zero attempts and "kompile-cli process interrupted" confirms no tests executed. Its frontend still saysPENDING. Worktree records omit gitContext; clean local source and exact submission command establish provenance, rather than a server-side head field.

13. Public bounded RPC retrieves known runs/results/attempts/log chunks even when frontend results return503. Frontend rows omit gitContext, so branch-name searches cannot locate runs; label/time controls must be verified by authoritative getBuildRun. A bounded candidate77ca090d row was another lane, and authoritativeCANCELED contradicted frontendPENDING. Frontend generation40282e95 was replaced by live5c2fd57d and its cursor needs reset; a watcher NOT FOUND is not authoritative absence. Missing process-tree/run-queue telemetry is not zero demand. Logical four-lane admission on two CPUs is owner-defined; production CPU admission is deliberately OBSERVE pending measured before/after comparison. No lane/capacity/flag/deadline change is justified as a timeout fix.

14. F361f terminal failures are three PR publication witnesses with inactive edit triggers (testDefinitionPublicationUsesBodyInputSnapshot, definitionSiteValidationChildProcessEditCannotPublish, definitionSiteValidationThreadEditCannotPublish), and four unchanged-main fixture assertions (testPreparedBatchUsesPreparedWorkspaceRuleBody, testPreparedBatchDoesNotRescanBuildScriptsOnExecute, testBuildMavenSameVersionDeletedPublicMemberInvalidatesArtifactCache, buildLocalRuleInvokesZeroArgWorkspaceRuleTest). Every terminal failure is below30seconds. testRV361KindsCaptureTransitions failed its first built-in attempt, then passed its second. These are new failed gates requiring mechanism verification, not a reason to weaken assertions or manually retry. Full XML/stacks/patch are preserved on F361f evidence. L2 independently wrote test-comprehensiveness and adversarial review sections; neither claims the full PR is clean.

15. Active F361g continuation has reproduced three old candidate trigger failures, proved four fixture failures on unchanged main, and recorded identical before/after directory stamps for the retry-only case. Its own authorized repairs force metadata boundaries and distinct revisions while retaining behavioral assertions and main tests. Existing README warmup is used for final validation. Do not duplicate this lane; re-fetch its full gate and source state. L2 static public-path check confirms manifests capture original spelling, so canonical path storage does not refute the parent-traversal hypothesis; its candidate verdict remains missing.

Relevant PRs / refs

| kompile-core | https://github.com/CodexCoder21Organization/kompile-core/tree/wip/F361g-2026-10-04 | d156a4da189d2cbde3f4e2bcdcf1b8639219517e | https://github.com/CodexCoder21Organization/kompile-core/pull/361 | Deterministic seven-assertion diagnosis and narrow fixture repairs; candidate patch saved | Active continuation; final gate pending; keep |

All necessary local experiments are preserved as patches/probe source or exact final source on these branches; superseded local commit graphs are not needed to resume. Basename baseline b753 and link baseline56, old/fixed hook patches and exact final patch are included in evidence. Earlier author evidence is linked from the PR body under wip/F361b-stamped-inputs-2026-10-04 and wip/F361c-rebased-evidence-2026-10-04. Nothing was deployed or published in this lane; command/history evidence contains only builds, tests, Git branch pushes, PR edits and handoff creation. No production route or Maven version changed.

Next steps

1. Supervisor assigns a fresh continuation from this checkpoint. Re-verify live PR/main and all evidence refs. Keep the draft PR and all listed evidence; nothing is proved obsolete. No PR closure is recommended. Original snapshot-order, error, basename and operation-count work is useful; F361e/F361f callback/count/legacy work is salvageable but its full candidate gate failed, so adopt it only after diagnosis and verification.
2. Locate L2's thin-worktree candidate run ID/result before submitting a replacement: actual labelL2-link-control, local candidate50bf5727 /branchwip/L2-link-parent-transport /submissionaround12:15UTC. Its client did not print an ID and worktree submissions omit gitContext. Main controlcd2a94c9 is already verifiedCANCELED with zero attempts; it supplies no timing verdict. Frontend freshness is unreliable; use authoritative bounded public RPC. No result or remote cancellation is inferred from a local outer deadline.
3. Recover investigation/L2/unconfirmed-link-witness.patch onto candidate7c13 rebased on current main. Obtain an actual candidate fail-first verdict. Published manager0.0.203 public control passes; OS path discrepancy alone does not establish a candidate cache failure. If confirmed, preserve intermediate physical link traversal before .., while testing cycle termination by physical entry identity without following the final link. Cover actual edit MISS, decoy edit HIT, persisted reopen, direct/chained links, cycles, missing targets and source identity. No production fix exists in L2.
4. Re-fetch active F361g continuation first: it already reproduced the three inactive edit triggers, all four fixture failures on unchanged main and equal-stamp directory precondition. Review its repairs and final full-suite outcome before duplicating any work. If a gap remains, diagnose with deterministic targeted controls on candidate and main. Preserve all existing expected behavior, test counts and time limits. The new metadata-only validation might bypass digest-based hooks, but this is an unverified explanation; force the exact public ordering. Independently review retry-only mixed-capture failure. Then review the recovered F361e patch against those proofs; do not present1184/1191 as a passing full gate.
5. Independently isolate the seven original compiler timing failures: two selected batches reproduce14/14 with exact attempt evidence; the named cause remains open. Historical C2/pre-snapshot controls refute C1-only and snapshot-only explanations for the three inherited victims. Use compiler/work counts and actual ordering/descendant telemetry, not a CPU spinner, machine calibration, host-load claim, deadline change, new retry, or lane-width change. If upstream, reproduce at both layers and fix upstream.
6. After fail-first proofs and fixes, run targeted controls and two independent review passes. Rebase immediately before builds/pushes; exact-head full suite must pass via --remote under L2 constraints before pushing a shared contract. Check PR OPEN before updating it, retain WHY-first description, and watch the six bld-* checks with a tracked bounded build-watchman command in plain mode. This repository is intentionally excluded from both kotlin.build checks by kotlin-build-ci README “GitHub-runner-only repositories”; do not manufacture a missing-check gate. Supervisor alone decides final review, merge, enqueue, publish, deploy, and handoff completion.

Operational details

Use gh repo clone; plain unauthenticated git ls-remote is unsuitable here. List wip refs with gh api repos/CodexCoder21Organization/kompile-core/git/matching-refs/heads/wip/. Keep findings incremental and checkpoint the first milestone. On the L2 host do not run a full local suite or large local fat-JAR build: use bounded remote runs. Pinned scripts use their own working JVM Coursier launcher. Prepend /code/ws/bin to PATH. Preserve pre-execution RPC stalls separately from timed test failures. Prepare fixtures as README says, using an empty scenario destination and retaining KOMPILE_SCENARIO_CACHE for the test call. A used warmup destination is rejected before work starts.

The first watcher unexpectedly ended with exit130; sender is unknown. Its terminal log is preserved, and the replacement used a shorter deadline within the original window. The replacement exitedRED1 after terminal check failures. No CI dispatch or retry occurred. The original lane test/watch processes were checked with actual ps -eo pid,args -ww and were gone. That sentence described the original author. L2 claimed and reported this handoff under its brief, and updated this body; no complete command was used.

L2 final checkpoint: all owned long test, RPC-reader and watcher processes have ended, verified with ps. No source PR push, CI dispatch, merge, queue, deployment, artifact publication or handoff completion was performed. The claim will be released only after the CHECKPOINT update. L2 evidence-only branch preserves the public witness/patch, main control, full errors and resume directions; it has no production fix.