← Workstreams

Workstream: Automatic Build Rule Caching

Status: In progress — mechanism phase landed 2026-07-08 (roadmap items 1-3); remaining: JvmBuildRules .kts migration + manual-simpleCache strip and the cross-repo graduation e2e tests · Component: Maximize developer productivity (part of Kotlin Build)

Goal

Automated bytecode rewrites on build rules so that all build rules — user-supplied ones in a build.kts, and the default JVM ones in build-kotlin-jvm's JvmBuildRules.kt — get caching enabled by default, without a build rule author having to call simpleCache(...) by hand.

This solves three problems with today's manual simpleCache(...) calls:

  1. Boilerplate/complexity. A build rule author has to know the caching API exists and wire it up themselves.
  2. Incomplete cache keys. simpleCache(...) takes an explicit list of inputs; if an author forgets to pass one of the function's parameters, the cache key under-describes the function's actual inputs and the cache can silently return a wrong (stale) result.
  3. Caching should be mandatory, not opt-in. Every build rule should be cached by default — a rule author should not be able to (accidentally or otherwise) ship an uncached rule, the way simpleCache(...) today requires an author to actively choose to call it.

The cache key must also include a hash of the build rule's own function and, transitively, any function it calls, so that changing a build rule's implementation (or code it depends on) automatically produces a different cache key — today, changing a build rule's body does not, by itself, change any simpleCache(...) input, which is the same class of correctness bug as (2) above.

Pointers to the current implementation

File:line references into the relevant repos, for anyone picking this up cold:

  • build-kotlin-jvm/src/JvmBuildRules.kt — the default JVM build rules (BuildKotlin, CompileJava, BuildJar, buildSimpleKotlinMavenArtifact, …). BuildKotlin (line ~1527) is today's hand-written example: simpleCache("BuildKotlin", src, classpath, buildAsJar, jvmTarget, pluginClasspaths) { ... } — a manual rule-name string plus a manually-curated list of the function's own parameters.
  • kompile-runtime/src/PrivilegedBuildAssistant.kt — the untrusted-code-facing simpleCache(...) / newCacheKeyBuilder() free functions build rules call; they delegate to whatever assistant is currently bound.
  • kompile.executionenvironment.bridge.interface — PrivilegedBuildAssistantHolder (an InheritableThreadLocal-scoped instance, set by trusted code around a build rule's invocation) and PrivilegedBuildAssistantUnprivilegedView (the interface exposed to build-rule code, including simpleCache and visitMethodInsn).
  • kompile-executionenvironment/src/kompile-executionenvironment.kt (ExecutionEnvironment.executeSync, ~line 70) — where the trusted assistant is bound into thread-local scope immediately around a build rule's (possibly cross-JVM) execution.
  • kompile-core/src/PrivilegedBuildAssistant.kt (simpleCache, ~line 116) — the trusted implementation: builds the cache key via newCacheKeyBuilder().add(inputs).getCacheKey() and get-or-computes through HashToFileCache.
  • CacheKeyBuilder/src/CacheKeyBuilderImpl.kt — walks the given inputs recursively (arrays/collections/maps included), hashes File/Path inputs by content (via FileMetadataCache), and SHA-256s the accumulated string into the final key.
  • HashToFileCache/src/util/cache/hashtofile/HashToFileCache.kt — the on-disk get-or-compute store: fan-out directory layout keyed by hash prefix, JVM-wide + cross-process locking, atomic publish via staging-dir rename.
  • kompile-buildrule-enhancer/src/BuildRuleInvocationInterceptorInjector.kt — existing, narrower precedent for exactly this kind of mechanism: an ASM ClassVisitor that rewrites call sites of a hardcoded list of four function names (resolveArtifact$default, BuildKotlin$default, buildSimpleKotlinMavenArtifact$default, resolveDependencies$default) to route through the assistant instead of calling directly. This workstream is about generalizing that mechanism — to all build rules, not a hardcoded list — and making it compute a complete, transitively-correct cache key rather than relying on a hand-passed input list.
  • kompile-buildscript/src/BuildscriptCache.kt:1465-1518 — the existing, PSI-level rule discovery already used by kompile-cli: any non-private function in a .kts file whose declared return type is exactly File/java.io.File is classified as a BuildRule (publicBuildRules; the zero-required-param subset is parameterlessBuildRules, the CLI-invocable ones). Anything else (Unit, ambiguous/inferred type, other explicit type) either becomes a test rule or is a hard compile-time diagnostic error — this is an enforced classification, not folklore.
  • kompile-buildscript/src/BuildscriptCache.kt:787-818 — where BuildRuleInvocationInterceptorInjectorClassVisitor is actually invoked today: as post-processing immediately after compiling a single staged .kts file. This step runs only for .kts scripts compiled through this pipeline — it does not run when an ordinary Kotlin module (like JvmBuildRules.kt today) is compiled, which is why calls within JvmBuildRules.kt are invisible to it.
  • kompile-core/src/BuildRuleResultIndex.kt, consulted in Workspace.kt:1179 and :1234 — a second, separate cache layer sitting in front of HashToFileCache/simpleCache. Maps a build rule's name string alone to its output files, with no fingerprint of the rule's inputs at all (a fast path to skip a ~2-3s child-JVM launch when re-invoking an already-built named rule). This is a documented correctness gap, kompile-core #208: a source or version change is served stale until the cache directory is cleared by hand. In scope for this workstream — see Decisions.
  • KompileDocumentation/docs/caching.md — the authoritative write-up of how the caching layers work today.

Decisions

Previously-open questions, now settled:

  • Which functions get automatic caching, and how are they identified? Metadata-aware: when kotlin.Metadata is present, eligible means a publicly declared function (not a property accessor and not internal) returning File/java.io.File, plus the $default wrapper of an eligible function; only metadata-less classes (ASM stubs and plain Java) retain the structural ACC_PUBLIC + returns-File rule. The pure structural rule is not faithful to the PSI classification it generalizes: in production it intercepts the File.jar/File.pom extension getters — making every cache hit content-hash the whole artifact directory and copy the jar to a fresh scratch directory — and similarly over-captures Kotlin-internal functions, which are public and mangled in bytecode. Regression coverage must exclude getters and internal functions while including $default wrappers of eligible functions.
  • Where does the transitive "hash of any function it calls" stop? At external Maven-artifact boundaries. The transitive walk covers only code within the kompile build-rule ecosystem's own source (the rule's defining module plus any other workspace/published build-rule source it calls into); a call into a class from a different Maven artifact (the Kotlin compiler, the JDK, Coursier, …) stops the walk, because that artifact's identity is already pinned — by version for a published dependency, or by content-hash for a classpath entry already passed as a parameter. This matches the precedent already set by Kotlin Build's own in-flight "make the cache version-aware" milestone, where the Kotlin compiler's version is folded into CacheKeyBuilder as an explicit input rather than a bytecode walk into the compiler's own code.
  • Where does the rewrite/wrapping happen — call site or function definition? Definition site. At .kts compile time the enhancer renames each wrapped function's body to $impl and emits a caching wrapper keyed on function identity + materialized argument values + transitive bytecode hash. This covers method references, supertype-typed calls, non-pipeline callers, and private helpers by construction; materializing defaults before keying makes defaulted and explicit calls share entries, whereas intercepting call-site $default shapes would split keys on nulls and bitmasks; and it avoids reflective dispatch. The wrap scope is all functions defined in a .kts, including private/internal helpers so cross-rule work such as resolveArtifact remains shared, with two implementation carve-outs: Unit-returning/side-effecting functions are not skipped on a hit, and results must be serializable (trivial pure factories may be exempted when caching costs more than recomputation). Call-site interception remains only as the routing seam into isolated execution for imported published rules; the fingerprint-validated BuildRuleResultIndex fast path avoids child-JVM launches on hits.
  • Caching is mandatory, not opt-in, with no opt-out. Every cacheable wrapped function is cached unconditionally — there is no per-rule choice to skip it and no escape-hatch annotation. A function that cannot tolerate this must expose its varying state as an explicit parameter; Unit-returning/side-effecting functions still run on hits as described above rather than bypassing instrumentation.
  • Fate of existing manual simpleCache(...) calls. All are stripped during the JvmBuildRules.kt → .kts migration, including calls in private helpers: definition-site wrappers replace them one-for-one with caller-independent keys that also include the body's transitive hash, closing manual keys' stale-on-body-change gap.
  • Scope of the JvmBuildRules.kt → .kts migration. In scope for this workstream (not an external prerequisite tracked only in Kotlin Build), sequenced after that workstream's "publish build rules as Maven artifacts, executed in isolation" milestone lands.
  • Shape of the JvmBuildRules .kts migration. The .kts is the single source of truth. The outer jar published at the existing kompile:build-kotlin-jvm coordinate becomes a generated, kotlinc-compiled Kotlin facade with full Kotlin metadata (defaults, named arguments, extension properties, and type-use annotations): File-returning rules dispatch through PrivilegedBuildAssistantHolder.getInstanceOrBootstrap(), while the importer-side vocabulary/resolution API (MavenPrebuilt data class + MavenPrebuilt2 factories, resolveDependencies2, Manifest, type annotations, extension properties, validators, and effects) retains real bodies. The .kts-pipeline-compiled bodies continue to ship as the isolated implementation payload (META-INF/kompile/impl.jar + manifest). The existing ASM-stub format cannot be a Kotlin compile surface because it lacks metadata/defaults/named arguments; the builtin-jar bootstrap requires JvmBuildRulesKt.class; and release N+1 self-compiles against ordinary release N, so replacing the coordinate with a stub would break all three (investigated 2026-07-23).
  • Marshaling surface. Extended, not signature-redesigned: MavenPrebuilt and Manifest cross the isolation boundary as structural records, and URL is marshaled as String. The parser/stub-generator textual type check must normalize annotations and nullability — today it rejects every JvmBuildRules signature, including @Jar File and String? — and buildPublishedBuildRulesArtifact must fail loudly with an aggregate error for any rule it cannot faithfully represent, rather than silently publishing stubs that throw UnmarshalableBuildRuleParameterException when invoked.
  • BuildRuleResultIndex (kompile-core #208) is in scope. Its rule-name-only fast path sits in front of HashToFileCache and would otherwise silently bypass this workstream's correct, complete cache key for any rule reached through it — leaving "all build rules cached correctly by default" untrue end-to-end. This workstream replaces it (or re-keys it) so there is exactly one cache-key computation, not two with different correctness properties.

Plan / roadmap

  • [x] Generalize BuildRuleInvocationInterceptorInjectorClassVisitor to detect eligibility by bytecode signature (ACC_PUBLIC + return descriptor Ljava/io/File;) instead of today's hardcoded four-name list. (Landed 2026-07-08: kompile-buildrule-enhancer PR 19 — BuildRuleInvocationEligibilityIndex (ASM-only, deterministic, platform/JDK/runtime owners excluded) + predicate-driven visitor, published as kompile.buildrule.enhancer:kompile-buildrule-enhancer:0.0.4; wired into the per-.kts post-compile pipeline by kompile-buildscript PR 64 — staged classes + builtin-rule jars + Kompile-Build-Rule-Stub: true manifest jars feed the index; legacy four names still intercepted; published as kompile.buildscript:kompile-buildscript:0.0.30.)
  • [x] Extend the cache-key computation to walk the transitive call graph of the invoked function (own bytecode hash + callees, stopping at external Maven-artifact boundaries) instead of relying on a hand-passed input list. (Landed 2026-07-08: kompile-core PR 210 — TransitiveMethodBytecodeHash (stable instruction serialization, invokedynamic/lambda walking, cycle-safe, inherited-method resolution, classpath-order shadowing) feeds the trusted PrivilegedBuildAssistant.visitMethodInsn, which now auto-caches every intercepted eligible call through HashToFileCache keyed on target identity + argument values + transitive bytecode hash; failures are never cached; published as kompile:manager:0.0.110.)
  • [x] Replace or re-key BuildRuleResultIndex's name-only fast path (kompile-core #208) so its child-JVM-launch-avoidance check uses the same complete cache key, not the rule name alone. (Landed 2026-07-08 in the same kompile-core PR 210: versioned entries {formatVersion: 2, rule, fingerprint, inputFileHashes, paths} — fingerprint = the same shared transitive-bytecode-hash computation + buildscript content hash + sorted dependency pins, validated together with re-hashed recorded input files at consult time; old-format entries are misses; WorkspaceCoordinateResolver validates the same way; issue #208 is CLOSED.)
  • [ ] (Dependency satisfied 2026-07-08: build-kotlin-jvm PR 57 and PR 59 merged — buildPublishedBuildRulesArtifact is on main, published as kompile:build-kotlin-jvm:0.0.31.) Migrate JvmBuildRules.kt (and any other .kt-based build-rule library) to a single .kts source of truth, publishing the generated kotlinc-compiled facade at the existing coordinate and the pipeline-compiled bodies as its isolated implementation payload.
    • [ ] Extend the marshaling surface for structural MavenPrebuilt/Manifest records and URL-as-String; normalize annotations/nullability in textual type checks; make buildPublishedBuildRulesArtifact aggregate and fail on every signature it cannot faithfully represent.
  • [ ] Strip all now-redundant manual simpleCache(...) calls during that migration, including calls in private helpers.
  • [ ] End-to-end tests: a rule with zero manual caching code gets a correct cache hit/miss; changing a rule's own body (or an in-ecosystem function it calls) invalidates its cache entry with no parameter list touched by hand; a call into an external Maven artifact does not trigger a transitive-hash walk into that artifact's own code; a stale BuildRuleResultIndex-style fast-path entry is never served after a source change; metadata-aware eligibility excludes extension getters and internal functions while including $default wrappers; a producer → consumer flow imports build.kotlin.jvm.*, uses omitted defaults and named arguments, and executes in isolation; and a double self-build proves release N+1 pinned to ordinary release N closes the bootstrap loop.

Follow-ups (non-blocking)

  • Remove the redundant double content-hash in recordExistingFileInputs.
  • Avoid copy-on-every-hit result materialization in simpleCache deserialization.
  • Remove the legacy hardcoded four-name interception list once index coverage is proven.

Related

  • Kotlin Build — the parent workstream; this depends on its "publish build rules as Maven artifacts, executed in isolation" milestone (needed so a .kts-defined rule library like the migrated JvmBuildRules.kt can still be published/imported), and is a correctness/ergonomics improvement to the same simpleCache/CacheKeyBuilder/HashToFileCache machinery its shared-build-cache milestone extends remotely — and also fixes kompile-core #208, which that workstream's "how the cache works today" section already documents as a known gap.

Graduation

This remains an active workstream until a build rule with zero manual caching code — neither a user-defined build.kts rule nor a default JVM rule (post-migration to .kts) — gets a correct cache hit/miss purely from automatic bytecode rewriting; a change to a build rule's own body (or an in-ecosystem function it calls) is demonstrated to invalidate its cache entry without any parameter list being touched by hand; metadata-aware eligibility has regression coverage for getters, internal functions, and eligible $default wrappers; a producer → consumer e2e imports build.kotlin.jvm.*, uses omitted defaults and named arguments, and executes in isolation; a double self-build proves release N+1 pinned to ordinary release N closes the bootstrap loop; and kompile-core #208 remains closed.