Handoff: Find and fix repeatable time limits in four FiledropEmbedded tests

Written 2026-10-23 UTC.

RE-VERIFY: This is a write-time snapshot. Fetch current main, verify the evidence branch with git ls-remote, and query https://github.com/CodexCoder21Organization/FiledropEmbedded/pull/29 before continuing. Read investigation/laneFLK29-findings.md on the branch for the incremental record.

Design requirements, 2026-10-23

The lane brief requires: "Never raise timeouts, cut iterations, or weaken assertions." It also requires: "Never merge, enqueue, publish or deploy." Local targeted commands must use --local. The one complete suite, once a correction is proved, uses --remote; allow it to finish. A correction requires a reproducer with at least half of first attempts unsuccessful, then an exact deterministic condition, a reversal, and fifteen first attempts for each of the four original scenarios. Open a PR only after that work; do not monitor CI after opening it. The PR must use the provenance and footer specified in the original lane brief.

Mission

The owner reported that https://droplets.wasmserver.com/ "is super unreliable" and asked for the underlying cause to be fixed. This lane concerns four existing FiledropEmbedded scenarios reaching their 30-second test limit. The task is incomplete: there is no verified correction or fix PR.

Observed chain

1. Read the supplied R29b findings and evidence, fresh project README, required TESTING.md including Never Calibrate a Reproducer to a Machine, and relevant engineering guidance. Original main was 089a6efc47cb49bc4142c93a1d0bb838b7419cfa.
2. Prior remote first attempts have process CPU/lifecycle wall milliseconds: bounded 8370/31147, updates 4690/31569, lock check 8540/31129, delete/update 10390/31369. Samples span storage/JSON work and runner completion. The prior R29b 57/60 main and 58/60 head results are separate from this lane's runs.
3. A separate initial bounded-work invocation reached its limit on attempt 1 at 32261 aggregate milliseconds, then passed on attempt 2. Samples span normal filesystem operations and post-commit listing checks; no single 30-second application wait was observed.
4. Ten four-test selections at 089a6ef passed 40/40 first attempts. After rebase to bd3a6f9e15a8f450cc7fbb21d027b7dcfb685929, five selections gave bounded 3/5, updates 3/5, lock check 5/5, delete/update 5/5. Combined matrix: 56/60 first attempts. The separate baseline remains separate. Each timing case with local limits is only 2/5 at the later revision, below the required half.
5. Returned-update round 11 has a sampled 31.882-second main-thread interval: 11.405 execution, 18.379 runnable waiting, 2.098 other waiting. Its original report records main CPU 11200.84 ms at 31.40 seconds elapsed. This locates the interval but does not identify why that runnable waiting occurs.
6. Temporary phase copies show costly bounded-work setup and listing checks, repeated large JSON processing in updates, and costs throughout twelve delete/update rounds. The lock check's JVM recording contains 690 ThreadDump operations taking 5.165 seconds. tests/testNoStorageCallRunsWhileHoldingAFiledropLock.kts:41 requests monitor and owned-synchronizer details on every storage call. OpenJDK 21 implements the owned-synchronizer request with a heap survey. This is proved instrumentation cost, not a proved cause of all four limits.
7. A FakeFileSystem comparison is unsuitable as a correction: concurrent variants encountered internal map/open-state errors, and the other two retained substantial cost. Original tests and assertions were not edited. Eight temporary measurement copies were removed after recording results.
8. Fresh kompile-core main 84396b347669e11532726a9c3396cb1120ea5f60 is version 0.0.211 and supports worker-slot declarations. The pinned manager 0.0.207 does not. It already configures one GC worker of each kind and C1 compilation. No incorrect admission calculation or CPU reservation has been proved; no annotation or toolchain change is selected.
9. Independent evidence review checked the counts and attribution. Structured data and eighteen original XML reports are on the evidence branch. No product correction, reversal, corrected-head fifteen-run gate, complete suite, or fix PR exists. No deployment or artifact publication was attempted by this lane.

Relevant branches and PRs

https://github.com/CodexCoder21Organization/FiledropEmbedded/pull/29 is MERGED at 2026-10-23T07:58:01Z with commit https://github.com/CodexCoder21Organization/FiledropEmbedded/commit/bd3a6f9e15a8f450cc7fbb21d027b7dcfb685929 . This lane made no merge or queue action. Other fresh clones were read-only and clean. No code from this lane is running or published anywhere.

Next steps

1. Re-verify current state and read the branch findings, investigation/flk29/README.md, and original XML attempt histories. Do not count later automatic passes as first-attempt passes.
2. Identify an exact condition for an original limit while observing runner admissions and per-thread scheduling together. The 18.379-second runnable interval is measured; its cause remains unknown. Do not substitute a test-serialization change or a capacity claim for that missing mechanism.
3. Obtain a consistent original-path reproducer with genuine concurrent public callers, then express the exact condition deterministically. Keep the current limits and assertions.
4. Correct the owning layer, verify the reversal, and run each original scenario fifteen times at one corrected revision with original-runner XML.
5. Run the one complete suite under the lane rules, then fetch/rebase, run targeted checks once more, push and open a PR with the required provenance/footer. Do not merge, enqueue, publish, deploy, or monitor CI after opening it.

Operational notes

Use a fresh own-lane clone. Run targeted checks with scripts/test.bash --local --test <name> --log <file>.xml; multiple --test flags select a batch. XML time fields are aggregate launch-inclusive values; lifecycle and CPU measures are distinct. Successful XML omits the phase output, so the separate disk-phases.json is authoritative for phase data. Replay and bounded capture tools live in investigation/flk29. Do not copy captured diagnostic text into messages.

GitHub REST returned a rate-limit response during initial document lookup; fresh clones supplied private documentation. A full-history kompile-core clone accumulated about 597 MB before checkout; a fresh shallow clone completed promptly. Only this lane's work was changed. No original scenario, timeout, iteration count, or assertion changed.

FLK29b continuation snapshot, 2026-10-23

RE-VERIFY: evidence-only branch checked against remote at 1ad7c25134a8c663c4c63de0fcf92ab1a0394833. Product source and original tests have no lane changes. Main advanced from bd3a6f9e15a8f450cc7fbb21d027b7dcfb685929 to 2b5f3978f232ff1dad8d5da808129e8f24d3562a; only build.kts artifact coordinates changed from 0.1.28 to 0.1.30, with source/tests and dependency coordinates unchanged. No correction or fix PR exists.

Remaining work: obtain a read-only host process/thread CPU view for this environment, account for CPU outside this cgroup, establish the condition in the owning code, then write a deterministic public-API reproducer and correction. Do not change runner concurrency, test limits, iteration counts or assertions. Run the reversal, each original scenario fifteen times with first-attempt counts, then one complete suite under the original brief. The final complete suite is remote by default and must be allowed to finish. No merge, queue entry, publication or deployment.

Observed results: five four-case selections at original main have returned-update 3/5 first-attempt passes; bounded-work, lock-observation and delete/update each have 5/5. Round 4 returned-update passes only on attempt 2; round 5 has three unsuccessful attempts. Two standalone returned-update controls each pass on attempt one. Thus matrix first-attempt total is 18/20, and all original invocations together are 20/22. This does not meet the reproduction bar or corrected-head gates.

Measured intervals: returned-update round 4 first attempt is 31.102 s with main CPU 8.938 s and runnable wait 19.248 s. Round 5 first attempt is 31.909 s with main CPU 6.673 s and runnable wait 22.247 s. Every other thread in those child JVMs uses less than 0.7 CPU seconds. JSON parsing and quoting dominate the main execution; allocation estimates for complete returned-update runs are about 4.13 GB, and collection pauses are about 0.55 to 1.45 s. Later attempts still exceed the limit after the other selected children have finished, so four-test runner parallelism is not a necessary condition. No product background maintenance worker accounts for the wait.

The lock-observation case is different: VM Thread execution is 2.498 to 6.575 s in four saved profiles, while main uses 0.973 to 1.392 s. This agrees with the earlier 690 thread inspections at tests/testNoStorageCallRunsWhileHoldingAFiledropLock.kts:41. The other scenarios show filesystem and concurrent-commit work. These are measured costs, not one established shared cause.

Machine visibility: a 231.735 s window has 697.269 cgroup CPU seconds and 3428.000 busy machine CPU seconds. No child cgroup directories are exposed. Most machine execution is outside this cgroup and cannot be named from this process view. During one over-limit interval, visible competing processes include a BuildTestEmbedded test process (17.12 CPU s), two WUI runner processes (16.66 and 16.45 CPU s) and the other selected Filedrop children (7.22, 7.17 and 6.10 CPU s). They do not account for the machine total. No other process or checkout was changed. A host visibility question remains unanswered.

The detailed findings are at https://github.com/CodexCoder21Organization/FiledropEmbedded/blob/wip/laneFLK29b-evidence/investigation/flk29b/findings.md . Read architecture/TESTING.md at https://github.com/CodexCoder21Organization/DocumentationRepository/blob/main/architecture/TESTING.md before writing tests. CPU capture uses investigation/flk29b/sample_cpu.py; its counter file and JFR recordings are local .git outputs until summaries are exported. Original targeted command is scripts/test.bash --local --test <scenario> --log <file>.xml; attempt-history is a sibling of the final test element and must be read separately. CPU samplers have fixed windows and explicit stop files. Raw recordings, compiler outputs, dependency caches and binary build outputs are intentionally omitted from Git.

Operational note: status uploads sometimes report UrlResolutionException with no peer for service handoff and registry size 31. A normal WUI page request followed by a fresh CLI get succeeded later. The report-challenge CLI was not invoked because it automatically merges a report PR, contrary to this lane's no-merge rule. No production process was restarted or changed.