← Priority list

Finish BuildTestEmbedded phase E NetLab owned routing

NetLab API 0.0.19 is merged and published. Both downstream PRs are open; NLMS is green, while BTE passed every phase-E/phase-D scenario but its overall check is blocked by a reproduced deployed-scheduler timeout wave affecting unrelated tests. Awaiting supervisor review and direction; neither PR is enqueued or merged.

hf-2026-07-30-finish-buildtestembedded-phase-e-netlab-owned-routing

Edit handoff

Add dependency

Complete this handoff

Moves it out of every priority list and into ArchiveArea.

Handoff document

Markdown

Finish phase E NetLab owned-environment routing in BuildTestEmbedded

RE-VERIFY: This is a write-time snapshot from 2026-07-31 01:32 UTC. Re-check pull-request state, remote branch heads, published coordinates, and CI before acting.

Current state

The API prerequisite is complete. NetLabApi pull request https://github.com/CodexCoder21Organization/NetLabApi/pull/29 is merged, and netlab.api:netlab-api:0.0.19 is published on kotlin.directory. It was resolved by both downstream builds and was not republished.

The two downstream pull requests are open:

  • BuildTestEmbedded: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/448
    • branch: feature/netlab-owned-environment-runner
    • pushed head: a600f96d4717e26e1ab562f92697f18e020064e1
    • artifact version: buildtest.embedded:buildtest-embedded:0.0.61309
  • NetLabManagerServer: https://github.com/CodexCoder21Organization/NetLabManagerServer/pull/130
    • branch: feature/owned-environment-public-client
    • pushed head: 1c6a8f3f5af3d63c19322d39d9b8e8fccc6dc5b6
    • artifact versions: server 0.0.76, bundled client 0.0.13

NetLabManagerServer CI is green: remote run 3c45616b passed 129/129 tests. BuildTestEmbedded remote run bf5d41da is red with 804 passed and 80 failed out of 884. All seven phase-E scenarios and all 16 phase-D cross-checks passed in that remote run. The failures are pre-existing, unrelated scenarios that timed out in one short dynamic-dispatch wave; details and the blocker are recorded below.

Neither downstream pull request was enqueued or merged. They are waiting for supervisor review.

Verification

Both repositories were cloned fresh, their feature branches were checked against the live remote tips, and they were fetched and rebased on current origin/main before building and again immediately before pushing. Their final working trees were clean and passed git diff --check.

Both repositories were built through the same fresh task-specific cache:

/tmp/claude-1000/-code/6aa4b0bd-7f88-4135-b24a-864ef6727f9b/scratchpad/cache/r43-netlab-continue

The cache has a task-local Coursier mirror configuration because Maven Central returned HTTP 429 during the first NetLabManagerServer build. With the mirror, the same published coordinates resolved and both builds passed.

The seven BuildTestEmbedded phase-E scenarios passed individually:

  1. netlabLeaseRunReleaseHappyPath
  2. netlabReleaseOnTestFailure
  3. netlabReleaseOnCancellationMidRun
  4. serviceCloseReleasesActiveNetLabLeasePromptly
  5. netlabLeaseFailureReportsFullError
  6. netlabReleaseFailureFailsTestLoudly
  7. defaultRunnerTestDoesNotLeaseNetLabEnvironment

All 16 phase-D routing cross-checks from https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/446 passed individually:

  1. allRoutedBuildRuleSurvivesRestartAfterNamedResult
  2. allRoutedSuiteStillExecutesRequestedBuildRule
  3. allRoutedSuiteUsesNoDefaultDropletEndToEnd
  4. annotatedTestRoutedToNamedRunner
  5. executionEnvironmentQueryParamsDeliveredToRunner
  6. executionEnvironmentRoutingHonorsExplicitTestFilter
  7. functionLevelEnvironmentOverridesFileLevel
  8. malformedExecutionEnvironmentUrlReportsDescriptiveError
  9. mixedRoutedAndDefaultSuiteEndToEnd
  10. routedAndDefaultFunctionsInSameFileUseDisjointSelectors
  11. routedFailureDoesNotPoisonDefaultPoolDispatch
  12. routedRunnerDeathFinalizesFailure
  13. routedRunnerInterruptionRemainsRecoverable
  14. routedTerminalResultSurvivesRestartExactlyOnce
  15. unassignedRoutedTestRestartResumesNamedRunnerOnly
  16. unreachableExecutionEnvironmentFailsLoudly

The added NetLabManagerServer public-client integration scenario, ownedLifecycleThroughPublishedClient, failed against current main with the expected NoSuchMethodError and passed on the feature branch. It uses a real in-process UrlProtocol sandbox connection, the bundled TopologyServiceClientImpl, and the manager RPC handler without mocks.

No machine-wide full suite was run while the one-minute host load exceeded the limit. Each required scenario was selected and run separately.

Reviews and findings

A dedicated test review found that the manager branch needed coverage through its typed bundled client rather than only through raw RPC calls. The public-client integration scenario above was added and verified failing on main and passing on the branch.

A separate code review found no remaining source defect. The manager forwards and validates the owned-lifecycle fields, and the BuildTestEmbedded connector keeps one lease per dispatch, preserves the first terminal reason, restores interruption after release, retries an unacknowledged release at close, and releases active leases during service shutdown.

Two operational findings were recorded:

  • Maven Central HTTP 429 from a fresh cache: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-30-2225-fresh-task-cache-triggered-a-maven-central-http-429.md
  • A shared kompile cache substituted a feature artifact into a historical baseline: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-31-0048-kompile-shared-cache-substituted-a-feature-artifact-into-a.md
  • The deployed dynamic scheduler can launch 16 test JVMs per worker based on memory alone: https://github.com/CodexCoder21Organization/PlanRepository/blob/main/challenges/2026-07-31-0130-buildtest-dynamic-dispatch-can-launch-16-test-jvms-per.md

The unreliable extra historical BTE baseline test was removed instead of being presented as evidence. The requested seven phase-E tests and 16 phase-D tests are the BTE compatibility evidence.

CI blocker

BuildTestEmbedded run bf5d41da failed 80 unrelated scenarios. Seventy-eight failed attempts began from 00:57 through 00:59 UTC, and none began after 01:00:49. Most reached the existing 30-second test bound. One captured JVM had 31.16 seconds elapsed but only 2.73 seconds of main-thread CPU while still doing ordinary SSH key generation. Independent run 524c6391 produced 78 timeouts with the same scenario names and timing shape.

The deployed scheduler computes droplet-side runner parallelism from predicted test heap and droplet RAM, up to the 16-test lease cap. It does not cap child-test concurrency by provider-reported CPU capacity. The consumer branch cannot change the already-deployed scheduler that executes its CI. No check was re-requested because doing so without a scheduler correction would only gamble on a different scheduling outcome.

A durable correction needs a separate, reliably failing runner-path test, a CPU-aware scheduler fix in BuildTestEmbedded, publication, and an explicitly authorized production deployment. That work is outside the two downstream feature pull requests, and this task explicitly did not authorize a deployment.

Incremental findings are in:

/tmp/claude-1000/-code/6aa4b0bd-7f88-4135-b24a-864ef6727f9b/scratchpad/out/r43-netlab-continue.md

References

  • Merged and published API: https://github.com/CodexCoder21Organization/NetLabApi/pull/29
  • Manager owned-routing reference: https://github.com/CodexCoder21Organization/NetLabManagerServer/pull/127
  • BuildTestEmbedded phase-D routing reference: https://github.com/CodexCoder21Organization/BuildTestEmbedded/pull/446

Next action

The supervisor should review the two open downstream pull requests and the recorded scheduler blocker. NetLabManagerServer is ready for review. BuildTestEmbedded's feature scenarios are green locally and remotely, but its overall check cannot honestly be called green until the deployed scheduler problem is corrected and the check is then re-requested. Do not merge or enqueue either pull request without explicit instruction.