Workstream: Kotlin Build (kompile)

Status: Planned · Component: Maximize developer productivity

Goal

Kotlin Build (kompile) is the toolchain at the very bottom of component 1: build logic as plain Kotlin functions in .kts files, compiled and cached, producing JARs / Maven artifacts / fat JARs, locally or remotely over url://buildtest/. Everything else in the plan is built with it, so making it faster and more flexible compounds across every other workstream and every agent.

This workstream tracks the next round of changes to the build system itself — the three evolutions of kompile-cli and kompile-core that have the highest leverage:

1. A shared build cache, so a build rule compiled once is reused by every other machine, CI runner, and agent.
2. The ability to build against several different versions of Kotlin, so the toolchain isn't pinned to one compiler.
3. Build rules published as Maven artifacts and executed in isolation, so a build.kts can be shared like any library — and so imported rules run in their own context (own dependencies, own Kotlin version) rather than inside the importer's JVM. This is the mechanism that makes (2) composable.

Current state

Kotlin Build exists and is self-hosting (it builds itself with itself). The relevant pieces for this workstream:

Repository	Role
kompile-cli	The CLI entry point. Discovers build rules, builds locally (--local) or remotely (--remote via url://buildtest/), runs tests, launches artifacts. Selects the cache directory via -c, --cache-location PATH.
kompile-core	The Workspace abstraction and the build cache it owns; also WorkspaceCoordinateResolver, which satisfies Maven-coordinate references from workspace source.
kompile-buildcache-interfaces	The cache-storage abstraction: BuildCache (umbrella), ContentAddressedStore, and BuildCacheIndex — interfaces only, so each backend owns its own consistency and atomicity.
kompile-buildscript	Parses and caches build.kts / tests/*.kts (the BuildscriptCache), SHA-256 content-hashed.
kompile-buildscript-stubjar	Generates a stub jar for a build script — plain JVM bytecode, one method per public rule, dispatching to the bound workspace rather than carrying rule bodies.
kompile-runtime	Executes compiled build scripts; provides simpleCache(inputs) { ... } for caching build outputs.
build-kotlin-jvm	The standard build-rule library (BuildKotlinJar, BuildMavenArtifact, …); the published artifact whose version currently pins the ecosystem's Kotlin compiler.
BuildTest	The remote build service behind url://buildtest/: provisions an ephemeral droplet per run, builds/tests there, destroys the droplet.

How the cache works today. The cache-storage abstraction exists: Workspace is constructed with an injectable BuildCache, and the default FileBuildCache is a per-machine, on-disk implementation — kompile-cli roots it at ~/.buildcache (overridable with -c, --cache-location); parallel kompile child JVMs share a ~/.aibuildcaches/ directory. It composes HashToFileCache (build-rule outputs) and BuildRuleResultIndex (a cross-process-safe, lock-free map from build-rule name to output files, one JSON file per entry — keyed by rule name alone, with no fingerprint of the rule's inputs, so a source or version change is served stale until the cache directory is cleared: a known correctness gap tracked in kompile-core #208, workaround rm -rf the cache location — planned fix in Automatic Build Rule Caching); the BuildscriptCache (parsed/compiled scripts) lives alongside, not yet behind the abstraction. Cache entries are keyed by a deterministic hash of the build inputs (sources, classpath, compiler flags — via CacheKeyBuilder), so an entry is not self-verifying: the key does not attest to the output bytes, which is why cache integrity rests on who may write (see Decisions). No remote backend exists yet — every machine, CI runner, and agent that hasn't built a given rule before rebuilds it from scratch. Notably that includes the remote build service itself: BuildTest droplets are ephemeral and start cold, so today every --remote build and every CI run rebuilds everything.

How Kotlin versioning works today. The compiler is pinned by the published build-kotlin-jvm artifact, which compiles via a fixed kotlin-compiler-embeddable dependency (currently 1.8.20 — pre-K2). There is no first-class way for a module to declare "build this with Kotlin X" and have the build run that compiler.

How build-rule publishing works today. A build rule declares the Maven coordinate it produces with @MavenArtifactCoordinates("group:artifact:version") (function-level, or file-level for a whole build.kts; the version may be left empty and locked at release time). Within a workspace, WorkspaceCoordinateResolver plus a custom Coursier protocol (community.kotlin.coursier.bldbinary) satisfy a @file:WithArtifact("g:a:v") reference by building the claiming rule from source, using the stub jar as the compile-time surface. Publishing to kotlin.directory is a manual act — build the buildMaven()-style output and publish it — and rule libraries like build-kotlin-jvm must today be laid out as ordinary src/ modules to be publishable; a plain build.kts cannot yet be published as an artifact. Imported build-rule artifacts currently load directly into the importer's JVM like any library, which is exactly what evolution (3) changes — and the isolated-execution mechanism that changes it has now landed in kompile-core (see the Publish build rules milestones below).

Why it accelerates developers

• The cache is the single biggest lever on build latency. In a world of agent swarms building in parallel across many ephemeral containers, a local cache is wasted work multiplied by the fleet size. A shared cache turns "the swarm built this once" into "nobody ever builds it again," which is exactly the compounding the plan is after. The first and largest beneficiary is the remote build service itself, whose ephemeral droplets currently cold-build every run.
• Multiple Kotlin versions remove a ceiling. Pinning to one compiler blocks adopting language features, validating upgrades, and supporting modules that legitimately need different versions. Making the version a property of the build lets the ecosystem move forward without a flag-day migration.
• Published, isolated build rules make build logic a library ecosystem. Any build.kts becomes shareable infrastructure with normal Maven versioning, and isolation means adopting someone's rules never entangles your build with their dependencies or compiler version.
• All three are toolchain-level wins. Because everything is built with kompile, an improvement here is felt by every other workstream and every agent, not just one service.

Plan / roadmap

The design decisions underneath these milestones are settled (see Decisions below); the milestones themselves are ordered but unscheduled.

Shared build cache

• [x] Abstract the cache storage. Done: kompile-buildcache-interfaces defines BuildCache / ContentAddressedStore / BuildCacheIndex, and kompile-core's Workspace accepts an injected BuildCache with the local FileBuildCache as the default.
• [ ] Add a SimpleFileSystem-backed cache implementation. Back the cache with the remote virtual filesystem service (FileStorage) over url://simple-filesystem/: the input-hash cache key is the entry's directory path, and the rule's output tree lives under it. An entry is visible if and only if its manifest file exists — the manifest (listing the entry's files) is written atomically last on publish and deleted first on evict, so partial or failed builds are never observable and readers treat any mid-read disappearance as an ordinary miss. This is how the local "a named result's files exist on disk" invariant translates to a remote store, and it is also the cross-host concurrency answer: SimpleFileSystem's atomic single-path write commits the manifest, and concurrent writers of the same key produce identical-by-construction entries.
• [ ] Wire it through kompile-cli. Extend cache selection so --cache-location (or a new companion flag) can accept a url:// shared cache alongside the local path, with the local cache acting as a fast L1 in front of the shared L2. Publishing to the shared cache is best-effort: a quota-full or failed publish tosses a warning effect and the build still succeeds; an unreachable cache degrades builders to cold builds, never to build failures.
• [ ] Adopt the shared cache in BuildTest. Droplets consult the shared cache as their L2 and, as the trusted writers, publish successful outputs back to it — making remote builds and CI warm for the first time, and letting shard droplets within one run reuse each other's compilations. (Droplet provisioning and toolchain install remain cold-start costs outside this workstream's scope.)
• [ ] Eviction janitor. A CI-side sweeper (only CI writes) applying the eviction policy from Decisions: TTL by write time plus a low-water-mark sweep as usage approaches the filesystem's size cap.

Build against several Kotlin versions

Progress (2026-07-10): this section is complete. The four implementation items landed together in build-kotlin-jvm PR 57 (merged 2026-07-08 once the redeployed buildtest fleet ran kompile-cli 0.0.78), published as kompile:build-kotlin-jvm:0.0.31. The health-check validation landed in KompileCliHealthCheck PR 9 (merged 2026-07-10), published as kompile-cli-health-check:0.0.4, registered, and verified with a production SUCCESS execution.

• [x] Make the Kotlin compiler version a property of the build. The compile rules in build-kotlin-jvm gain an optional kotlinVersion parameter; when unset, the default is the one carried by the imported build-rule artifact's version (see Decisions — there is deliberately no workspace-level default file). (Landed 2026-07-08: PR 57, published in kompile:build-kotlin-jvm:0.0.31.)
• [x] Resolve compiler versions on demand (kotlin-compiler-embeddable:<X> via kotlin.directory / Coursier) and cache them, the same way build-script dependencies are resolved, executing each requested compiler in its own isolated context. (Landed in PR 57: on-demand resolution + isolated-classloader execution, e2e-tested with metadata-version discriminators.)
• [x] Make the cache version-aware. The effective concrete compiler version simply becomes one more input to CacheKeyBuilder — no special machinery; today it is implicitly constant, which is why it is absent from the key. A version chosen by default and the same version chosen explicitly share cache entries. (Landed in PR 57, including the distinct-cache-entries e2e test.)
• [x] Diagnose mixed-version dependency violations. For ordinary library dependencies, a consumer's Kotlin version must be ≥ each Kotlin dependency's version (Kotlin metadata is backward- but not forward-readable); kompile detects violations up front and fails with a descriptive error naming both modules and both versions, rather than surfacing the compiler's opaque metadata-version error. The runtime classpath takes the highest stdlib in the graph. Build-rule imports are exempt — isolation (below) removes them from this constraint entirely. (Landed in PR 57 with the older-consumer-of-newer-dependency e2e test.)
• [x] Validate against the existing health checks (KompileCliHealthCheck) so the toolchain proves it can compile a trivial program under each supported version. (Landed 2026-07-10: KompileCliHealthCheck PR 9 compiles under the bundled default and explicit kotlinVersion = "2.4.0" through a real kompile.cli:kompile-cli:0.0.79 subprocess and proves which compiler ran by parsing kotlin.Metadata.metadataVersion from the produced class bytes (major 1 vs major 2), with 11 e2e tests. Published as community.kotlin.healthchecks.kompilecli:kompile-cli-health-check:0.0.4, the production registration updated, and a production execution recorded SUCCESS (89.6s; default → metadataVersion 1.9.0, explicit 2.4.0 → metadataVersion 2.4.0). The supported set — the ecosystem default carried by the pinned build-rule artifact and migration target 2.4.0 — is now operationally defined by this check.)

Publish build rules as Maven artifacts, executed in isolation

• [x] A build rule that builds the current build-rule-file into a publishable Maven artifact. The artifact pairs a consumer-facing outer jar with an isolated implementation payload: the compiled rule bodies travel as META-INF/kompile/impl.jar together with build-rule-manifest.json, while the POM depends on the kompile runtime — not on the rule's own dependencies. Publishing stays manual, exactly like any other Maven artifact, with the version locked at release time via the existing empty-version @file:MavenArtifactCoordinates form. (Landed 2026-07-08: PR 57 and PR 59 merged with an ASM-stub outer jar; buildPublishedBuildRulesArtifact publishes the stub + isolated payload with the Kompile-Build-Rule-Stub: true manifest marker, as kompile:build-kotlin-jvm:0.0.31. The decided generated-Kotlin-facade outer format remains follow-up work; see Decisions.)
• [x] Imported build rules always execute in isolation. When a .kts imports a build-rule artifact, kompile detects the stub and executes the rule in its own context — own classloader/JVM, own classpath resolved from the embedded manifest, own Kotlin runtime — never linked into the importer's JVM. The landed marshaling surface is String, boxed primitives, File, and List/Set/String-keyed-Map thereof; the Automatic Build Rule Caching migration extends that boundary with structural records and URL-as-String rather than redesigning signatures. (Landed: kompile-core #204, published kompile:manager:0.0.109 via kompile-core #207; consumed by the CLI in kompile-cli #125, kompile.cli:kompile-cli:0.0.78.)
• [x] One dispatch path, two bootstraps. The stub dispatches through the same seam whether or not kompile is running: under kompile, the bound workspace executes the rule; imported as normal code in a plain JVM app, the stub lazily bootstraps an embedded kompile runtime (the POM's kompile dependency makes this work out of the box), resolves the isolated classpath, and behaves as if the build system had been asked to run the rule. (Landed: the getInstanceOrBootstrap bridge seam in kompile-executionenvironment-bridge-interface:0.0.7 and the published-stub generator generatePublishedStubJar in kompile-buildscript-stubjar:0.0.3.)
• [ ] Publishable rules take all inputs as parameters. Zero-arg rules that read workspace-relative paths (File("src")) are meaningful only in-workspace. (Verified 2026-07-23 that PR 59 shipped no warn/reject behavior: PublishedBuildRulesArtifact.kt contains no such check. For now, the publisher tosses one detailed warning effect per zero-required-parameter rule using the existing parameterlessBuildRules classification — no body analysis, exclusion, or hard failure yet — explaining how to parameterize it. This may be tightened later, so the box remains unchecked.)

Cross-cutting

• [ ] End-to-end tests following the testing standards: a real Workspace against a real in-process SimpleFileSystem ServiceServer (never a fake), asserting that a second builder gets a cache hit for a rule the first builder compiled; that two Kotlin versions produce distinct, non-colliding cache entries; and that a published build-rule artifact executes in isolation from both a .kts importer and a plain JVM consumer.
• [ ] Provision capabilities into tests (and possibly build rules) via W3Wallet. A test or build rule that must reach an external API or a credential-bearing service should receive an injected, scoped W3Wallet capability — exercised without the secret ever entering test source or the test process — rather than relying on ambient credentials. This is the principled form of today's "tests use production url:// services for infra," and it is the same mechanism that provisions the shared-cache write capability into CI builders.
• [ ] Documentation. Keep the kompile-cli and kompile-core READMEs and the Kotlin Build project page in sync as these land.

Decisions

Previously-open questions, now settled:

• Cache scope & trust. The shared cache is global — one cache for all users and orgs, not per-org or per-identity — because fleet-wide reuse is the entire point. Cache keys contain the checksum of the inputs, so distinct builds cannot collide. Reads are open (USE on the filesystem's fs-read capability); writes are restricted to trusted CI builders — BuildTest droplets and CI runners holding an instance-scoped, auto-expiring fs-write capability provisioned by the platform (W3Wallet hosted-workload provisioning). Because entries are keyed by input hash, they are not self-verifying, so integrity rests entirely on this writer restriction — which suffices: the writers are the same attested infrastructure the org already trusts to merge code. Developer machines are read-only against the shared cache (they keep their local L1); the residual risk of a flaky build rule publishing a bad binary exists at every cache scope, including single-user, and so does not drive the design.
• Relationship to the remote build service. One backing store. The shared cache is the artifact store; the BuildTest/CI droplets are simultaneously its primary readers and its only writers. A local build reading the shared cache and a --remote build therefore hit the same artifacts by construction — there is no second store to reconcile. (The premise of the original question — that the remote builder already maintains a cache — was stale: BuildTest droplets are ephemeral and today cold-build every run.) The per-run channel for retrieving a named output artifact from a build run is a delivery mechanism, not a cache, and stays as-is.
• Eviction & quotas. The cache is one SimpleFileSystem filesystem whose required maxSizeBytes is the hard quota backstop (initial values operator-tunable, on the order of 100 GB). SimpleFileSystem's expiration is whole-filesystem, so per-entry eviction is the cache's own job: TTL by write time (~30 days initially — the first CI build after expiry rebuilds and republishes, so a hot entry costs at most one rebuild per TTL; true LRU is deliberately rejected because reads must stay read-only) plus a low-water-mark sweep of oldest-written entries when usage nears the cap. Eviction is safe at any moment because of the manifest-visibility contract in the roadmap: no manifest, no entry, and disappearance mid-read is just a miss.
• How many Kotlin versions, and chosen how? Per-module granularity via an optional kotlinVersion parameter on the compile rules (which incidentally gives per-rule control, since each build.kts invocation site may set it). Precedence: explicit parameter → the default carried by the imported build-rule artifact's version. Upgrading the build-rule artifact is how a module adopts a newer default compiler; explicitly passing kotlinVersion is the escape hatch to take new rules while staying on an older compiler. There is deliberately no workspace-level default file — two sources of defaults invite confusion, and per-module choice already falls out of each build.kts picking its build-rule version. The supported set is two versions at any time — the ecosystem default and the migration target (today: 1.8.20 and the latest stable 2.x) — defined operationally as the versions the health check compiles under; other Coursier-resolvable versions may work but are not promised. Mixed-version dependencies follow the consumer ≥ dependency rule for libraries, and are unconstrained across build-rule imports thanks to isolated execution.
• Publishing build rules. Publishing is manual, not a CI pipeline (for now): a user runs the publish rule and pushes the artifact like any other Maven build. Working assumption (2026-07-23): the generated, kotlinc-compiled Kotlin facade described in Automatic Build Rule Caching becomes the outer published format for all build-rule artifacts, retiring the ASM-stub outer jar while retaining ASM stubs for in-workspace compile-surface use; the .kts-pipeline-compiled bodies remain the isolated payload, never raw bodies on the importer's classpath (which would re-couple consumers to the publisher's dependencies and compiler version).

Related

• Automatic Build Rule Caching — a correctness/ergonomics evolution of the local cache this workstream already relies on: bytecode-rewriting every build rule (default and user-supplied) to get a complete, transitively-correct cache key automatically, instead of a hand-written simpleCache(...) input list.
• Pluggable Execution Environments — builds directly on this workstream's shared cache: it makes test execution a distributed, pluggable service (@ExecutionEnvironment("url://...")) where compilation happens once and any runner pulls the already-built classpath from the shared cache instead of rebuilding. That workstream also reframes this one's url://buildtest/ remote service as the default test runner rather than the only one.
• SimpleFileSystem — the remote filesystem primitive the shared cache is built on; the Kotlin Build shared cache is the headline adopter in that workstream's plan.
• LambdaServer — its LambdaServerApi implements kompile.Workspace for remote build/test execution, so cache changes to the Workspace contract touch it too.
• W3Wallet — both the capability that authorizes writing the shared cache and the capabilities provisioned into tests/build rules so they reach credential-bearing services without secrets in source.
• Kotlin Build project page — the canonical description of what exists today.

Graduation

This remains an active workstream until the shared cache is in production use (a second builder demonstrably reusing a first builder's output over url://), multi-version builds are supported and tested, and at least one build-rule artifact is published and consumed in isolation from another repository. Because Kotlin Build already has a project page in the Documentation Repository, "graduation" here means folding these capabilities into that existing page rather than creating a new one.