Workstream: Serialization (@Serializable / @Immutable)

Status: Planned · Component: Maximize developer productivity

Goal

Make "this value can cross the wire" a compiler-verified property of the code itself — for classes and for functions. A new Kotlin compiler plugin (the community.kotlin.serialization project) provides two annotations:

• @Immutable — a claim, verified by the compiler, that a class is deeply immutable: every property is a val, and every property's type is itself @Immutable or on a built-in whitelist of known-immutable types (primitives, String, enums, read-only collections of immutable elements). Annotating a class that violates the rules is a compile error, so immutability never regresses silently and needs no runtime enforcement.
• @Serializable — the compiler adds serialization support to the annotated class or function. A @Serializable class must be either @Immutable (serialized by value) or a recognized reference type (serialized by reference — e.g. an Observable serialized as the URL it can be re-materialized from). There is deliberately no snapshot serialization of mutable state: mutable state crosses the wire only as a live reference, never as a silently frozen copy.

The headline capability is that functions are serializable too. Placed on a function or lambda, the annotations make the function value — its code identity plus its captured variables — serializable:

@Immutable @Serializable
data class Region(val name: String, val minLat: Double, val maxLat: Double)

fun regionFilter(region: Region): (@Serializable @Immutable (Event) -> Boolean) =
    { event -> event.lat in region.minLat..region.maxLat }   // capture of `region` is
                                                             // verified immutable +
                                                             // serializable at compile time

• @Serializable @Immutable on a function is the common case: the compiler verifies every captured variable is itself immutable and serializable (the same recursive rules; capturing this is allowed only when the enclosing class qualifies), so the lambda serializes fully by value — a violation is a compile error at the capture site.
• @Serializable without @Immutable on a function is also legal and means the captures may be serializable reference types: a lambda that captures an Observable serializes the reference (its URL), and the receiving side re-connects to the live projection rather than receiving a frozen copy.
• @Immutable on a function speaks only about its captured state, not its behavior — the body may still perform I/O or toss algebraic effects; purity and determinism are not claimed and not verified.

Together the two annotations give the ecosystem one invariant it currently lacks: any annotated POJO or closure can be handed to another process with compile-time certainty that the transfer is well-defined.

Why a compiler plugin — not a library, convention, or reflection

• The rules need enforcement at compile time. Deep immutability and capture-eligibility are exactly the properties that rot silently under a convention-based approach (today's hand-written toJson()/fromJson() value objects) and surface as runtime surprises under a reflection-based one. Reflection is also a non-starter structurally: the ecosystem's code conventions bar it, and url:// client code — where marshaling runs — executes inside an SJVM sandbox with no reflection access at all. Only the compiler can reject a mutable capture at the capture site.
• Closures are compiler-internal. The captured variables of a lambda are not part of any public API a library could walk; enumerating them, verifying them, and generating a capture-aware serializer is inherently the compiler's job.
• The precedent exists in-house. The ecosystem already builds Kotlin compiler plugins: Observable's observable-compiler-plugin (IR-level instrumentation of @Observable classes) and MutableImmutable (community.kotlin.datalayer.mutableimmutable — persistent, structurally shared object graphs with scoped edit {} mutation). MutableImmutable is the closest relative, and this workstream commits it to a migration to real immutables so its graphs qualify.

Structure first, format second: pluggable serializers

The compiler plugin's product is not a wire format. It is the verified structural decomposition of an annotated class or function — its property graph, its captured variables, its code identity — exposed so that pluggable serializer implementations decide how the structure becomes bytes. Different serializers make different choices, and the architecture explicitly supports:

• Code separated from captures. Most serializers keep a serialized function's code distinct from its captured values, so code is deduplicated and cached independently of the (small, per-instance) captures.
• Blobs by reference. A serializer may externalize large payloads — bytecode above all — into another repository and embed only a reference, e.g. a content hash resolved via Blobstore (whose plan already names bytecode JARs as a payload class), the kotlin.directory Maven repository, or the url:// fabric's existing sandboxed-bytecode channel. The receiving side loads code from its own classpath when present and fetches by reference when not.
• References by URL. Reference types serialize as the URL they re-materialize from — the Observables case above.
• Node identity and deduplication. A serializer may assign IDs to nodes so an object or reference sent repeatedly (within a message or across a session) is encoded once and referenced thereafter.

A JSON reference serializer ships first (aligned with the ecosystem's org.json toJson()/fromJson() convention and the portable JSON value facade planned in Kotlin Multiplatform), but every such choice — format, blob externalization, dedup, reference strategies — belongs to the serializer layer, not the plugin. Deserialized code is still foreign code: executing it stays inside the platform's existing sandboxed client-bytecode machinery, and who may ship code to whom is a W3Wallet capability question, not something a serializer decides.

Role in the plan

This is a reusable primitive for component 1, and every committed consumer is a load-bearing part of the plan:

• LambdaServer — invoke a lambda. Today "run this code remotely" means registering a Maven artifact, JAR, or built git repo. A serializable function collapses that to its ideal form: hand the platform a @Serializable @Immutable lambda and invoke it — the lambda is the deployment unit. This also answers the serialization-contract question the LambdaServer plan had left open: parameters and return values are @Serializable values under this workstream's rules.
• EventLog / EventSourcingStreams — events and shipped operators. @Immutable @Serializable is the natural shape of a domain event (the Event Log's entries are immutable by contract). And a derived stream's filter, join, and projection logic are functions — serializable functions let a subscriber ship its predicate or fold to where the stream lives instead of moving the stream to the code.
• url:// RPC marshaling (UrlResolver). Typed proxies marshal arguments as ad-hoc JSON maps today, which cannot carry concrete data classes — let alone closures. Generated serializers give openSandboxedConnection proxies typed, compile-verified marshaling, including passing callbacks and lambdas as RPC arguments.
• Observables — by reference. Observables are the canonical serializable reference type: capturing one in a serialized lambda or passing one through RPC transfers the URL, and the receiver re-materializes the live projection.

One serialization story, replacing kotlinx.serialization

The end state is a single serialization story for the whole ecosystem. Today's landscape is split three ways: hand-written toJson()/fromJson() value objects on the JVM, kotlinx.serialization in the multiplatform protocol repos (W3WalletProtocol, RpcProtocolApi, LightroomPlus), and no answer at all for functions. This plugin subsumes all three: generated serializers retire the hand-written convention, and once the class side is mature — including multiplatform coverage — the protocol repos migrate off kotlinx.serialization, which this workstream replaces over time. Delivery is JVM-first (function serialization is code-shaped, and code identity on the JVM means bytecode); multiplatform class-side support is the explicit bridge to the kotlinx migration, sequenced with the Kotlin Multiplatform workstream. The annotations live in their own community.kotlin.serialization package, so coexistence during the migration is an import choice, not a conflict.

MutableImmutable: migrating to real immutables

MutableImmutable (community.kotlin.datalayer.mutableimmutable) is the ecosystem's update-ergonomics layer for immutable object graphs: an edit {} scope with imperative setter syntax, path-copying structural sharing (originals are never modified; a no-change edit returns the original instance), and deep, cycle-safe, hash-cached graph equality. Its semantics are exactly what this workstream wants to serialize — but its mechanism today is runtime-policed mutability: properties are real vars whose instrumented setters throw outside a thread-local edit session, construction mode disables every check on the thread, mutable collections bypass the setters entirely, and the runtime leans on reflection (so it cannot run in the SJVM sandbox where url:// marshaling executes). Such objects are observationally immutable in the disciplined happy path, but the compiler cannot prove it — so they do not qualify as @Immutable, and there is deliberately no trusted special case for MutableImmutableBase in the verifier.

The committed direction is to migrate MutableImmutable — gradually — to real immutables: its compiler plugin evolves so @MutableImmutable classes compile to genuinely immutable nodes (val/final fields), with edit {} becoming compiler-generated copy-on-write that builds new instances along the edited path instead of mutating policed clones. The migration keeps the project's actual point — edit ergonomics, structural sharing, graph value semantics — while eliminating the runtime-enforcement holes and the reflection. The one genuine casualty is cyclic graphs, the single capability that depends on under-the-hood mutation; the migration restricts them or special-cases them explicitly (serializers can still encode cycles via node identity, so this is a value-model decision, not a wire-format one). Once a class is migrated it satisfies the @Immutable verifier by construction and its graphs serialize by value: @Immutable supplies the guarantee, MutableImmutable supplies the update ergonomics on top of it, and @Serializable supplies the wire mobility.

Shape of the project

A single community.kotlin.serialization repository (the community.kotlin.clocks.hierarchical naming precedent), Gradle-built like the MutableImmutable and Observable compiler plugins (a compiler plugin tracks the Kotlin compiler's own version, ahead of what kompile pins), publishing to kotlin.directory: an annotations module (the @Serializable / @Immutable interfaces and the known-immutable whitelist), the compiler plugin, a small runtime, and each serializer implementation as its own module — interfaces and implementations kept in separate modules, the same Api-vs-Embedded separation the standard layered architecture applies to services. Consumers that build with kompile need the Kotlin Build toolchain to learn to apply third-party compiler plugins — a small, explicit dependency of this workstream on that one.

Current state

Nothing of this workstream exists yet — there is no @Serializable or @Immutable annotation anywhere in the organization. What exists is the prior art it builds on: the two in-house compiler plugins named above, the org.json value-serialization convention, kotlinx.serialization in the protocol repos (the incumbent to be replaced), and the url:// fabric's sandboxed bytecode-shipping machinery (the transport a code-by-reference serializer rides on).

Plan / roadmap

• [ ] Annotations + @Immutable verification. The annotations module and the compiler plugin's verification pass: recursive class rules, the known-immutable whitelist, function-capture rules (including the this-capture rule and the reference-type relaxation for non-@Immutable functions), each proven by failing-first compile tests per the testing standards.
• [ ] Class serialization. Generated structural decomposition for @Immutable value graphs, the serializer SPI, and the JSON reference serializer.
• [ ] Function serialization. Lambdas and function references as code identity + captures — classpath-resolved code first, then code-by-reference via content hash (Blobstore / kotlin.directory), riding the existing sandboxed-bytecode machinery on the receiving side.
• [ ] Serializer strategies. By-reference serialization of reference types (Observable-by-URL first), blob externalization, and node-ID deduplication as serializer-layer features over the same structural decomposition.
• [ ] kompile integration. Kotlin Build support for applying compiler plugins, so kompile-built consumers can annotate their own types.
• [ ] MutableImmutable migration to real immutables. Gradually evolve the MutableImmutable plugin to compile @MutableImmutable classes into genuinely immutable nodes with compiler-generated copy-on-write edit {} (final fields; the construction-mode, mutable-collection, and reflection holes removed; cycles restricted or explicitly special-cased), so migrated classes qualify for @Immutable / @Serializable by value — see above.
• [ ] Consumer adoption. LambdaServer invoke-a-lambda; EventLog / EventSourcingStreams event types and shipped stream operators; typed url:// RPC marshaling (UrlResolver); Observables reference serialization.
• [ ] Multiplatform + kotlinx migration. Multiplatform class-side support (sequenced with Kotlin Multiplatform), schema evolution at the serializer layer, then migrating the protocol repos off kotlinx.serialization.

Graduation

The workstream graduates when the plugin and its first serializers are published, at least two committed consumers run on it in production — the flagship being a LambdaServer invocation of a serialized lambda — and the project is documented in the Documentation Repository. The kotlinx.serialization migration completes on its own schedule after graduation; it is the workstream's end state, not its graduation gate.