lazily-spec
Language-agnostic wire protocol specification for the lazily reactive signals family.
This site is the rendered companion to the lazily-spec repository. It defines the canonical message schemas shared across every lazily implementation:
- lazily-rs (Rust)
- lazily-py (Python)
- lazily-zig (Zig)
- @lazily/signaling (TypeScript / Cloudflare Worker)
Cell Model
Upstream of every transport, the Cell Model fixes how a cell’s value
converges. A cell is either single-writer (local/direct, no merge) or
multi-write, and a multi-write cell carries a pluggable merge: <mechanism>
attribute. CRDT is the first multi-write merge mechanism (merge: crdt), not the
only one — lww, ot, lease, and custom are reserved alongside it. All transports
below carry cells classified by this model.
Protocol Layers
| Layer | Spec | Schema |
|---|---|---|
| IPC (Snapshot + Delta) | Wire Protocol § IPC | snapshot.json, delta.json |
| Cross-language FFI | Wire Protocol § FFI | ffi.json |
| Signaling (WebSocket) | Wire Protocol § Signaling | signaling.json |
| Distributed (CRDT) | Wire Protocol § Distributed | distributed.json |
| Capability negotiation | Wire Protocol § Capability Negotiation | inline |
Every layer in this matrix is required of every binding. The Distributed CRDT
row and the required keyed cell collections layer
are unconditional. The C-ABI FFI row is required by default with a narrow platform
carve-out (a binding whose runtime cannot host a native in-process C ABI — e.g.
browser/Worker JS — declares ffi = none and interops over the wire instead). The
thread-safe and async reactive contexts are required where the platform supports
them (a platform that structurally lacks threading or suspendable async declares
thread_safe = none / async = none); the shared-memory payload path is required
where the platform supports it, with an I/O-channel fallback (Inline payloads over
IPC/WebSocket/WebRTC) when it does not. See the
Binding Conformance Matrix for the full
MUST/MAY breakdown and the carve-out terms.
Wire Format
All messages use JSON with serde-compatible tagging ("type" discriminant). Future
binary codecs (bincode, postcard, protobuf) encode the same schemas — the JSON representation
is normative.
Schema Format
Schemas are provided as JSON Schema (Draft 2020-12). Each implementation must validate against these schemas. See JSON Schemas.
Scope & non-goals
This repo extracts the cross-language, wire-protocol, and behavioral sections
from lazily-rs/SPEC.md into a standalone reference. Every lazily-rs feature
area is accounted for here exactly once: either normatively specified (with a
link below) or explicitly marked Rust-specific. Rust-specific internals remain
in the Rust crate and are intentionally out of scope.
Covered (normative, cross-language)
Out of scope (Rust-specific implementation)
These lazily-rs features are implementation choices, not cross-language contracts. Other bindings pick their own; they MUST meet the normative contracts above but need not mirror Rust’s approach.
| lazily-rs area | Why out of scope |
|---|---|
Context / ThreadSafeContext lock strategy (ReadStrategy, inline seqlock, typed cache fast-path) | Internal scheduling/locking; each binding picks its own concurrency strategy. The existence of the thread-safe and async context surfaces is required where the platform supports it (Wire Protocol § Concurrency layers are required); only the lock internals are out of scope |
SlotId internal representation | Volatile internal handle; the wire-stable identity is NodeId / NodeKey |
instrumentation (lock-site tracking) | Rust diagnostics |
str0m_backend / str0m_net | Concrete Rust WebRTC backend (str0m crate); only the transport abstraction is cross-language |
| Performance benchmarks | Rust-specific measurement |
lazily-serde type-erasure internals | Rust serialization approach; the wire shape, not the codec implementation, is normative |
Versioning
Protocol versioning follows the IPC capability negotiation: each session exchanges
{ protocol_id, protocol_major_version, codec } before any graph state flows. A major version
bump is a breaking change; minor additions are additive.
Reactive Graph
The reactive graph is the dependency-tracking core of every lazily binding:
a set of nodes whose values are derived from each other, where a change to a
source invalidates only its transitive dependents and recomputation is pull-based
and glitch-free. This chapter fixes the cross-language behavior — the
lazily-formal kernel and the
lazily-rs / lazily-py / lazily-zig / lazily-kt implementations are the executable
references.
The reactive graph is compute, not protocol: only resolved values cross IPC/FFI as ordinary cell payloads. Every binding that ships a reactive graph MUST honor this contract.
The reactive family
A cell is a value-bearing reactive node — a node with a readable value. There are exactly two kinds of cell and one sink:
| Kind | Handle | Node | Role |
|---|---|---|---|
| source | Source<T, M = KeepLatest> | SourceCell | Written from outside. set replaces (invalidating dependents on a == (PartialEq) change; an equal set is a no-op); merge folds an op ⊕ under MergePolicy M. M defaults to KeepLatest, so Source<T> is a plain source cell and Source<T, M> with M ≠ KeepLatest is what used to be called a MergeCell. |
| computed | Computed<T> | ComputedCell | Computed from upstream. Tracks its dependencies automatically, computes on first read, caches, and recomputes only when read after an upstream invalidation. Guarded, always: an equal recompute suppresses downstream invalidation (matches TC39 Signal.Computed). |
| effect | Effect | — | A side-effecting sink — no readable value, so nothing can ever depend on it. Reruns whenever a tracked dependency invalidates; an optional cleanup closure runs before each rerun and on dispose. It sits outside the cell hierarchy, by capability, not by current degree. |
Cell is the value-bearing-node concept, never a handle. The word Cell
names “a reactive node with a readable value”; the two kinds of cell are the
SourceCell and the ComputedCell (the arena nodes Node::Source(SourceNode)
and Node::Computed(ComputedNode)). A caller never holds a Cell — the two
handles are the concrete types Source<T, M> and Computed<T>. There is
no Cell<T, K> genus struct and no phantom kind parameter: Source’s policy
M is a real parameter of the Source<T, M> handle, present only where writes
exist. Reactive is the umbrella adjective — the reactive graph is cells plus
effects — never a type and never a synonym for one kind.
The partition is one axis with both sides covered and no leftover: a cell’s
value comes either from outside it (a SourceCell) or from upstream of it (a
ComputedCell). Effect has no value to read, so it cannot be folded in — the
sink position is a real boundary, not a naming accident.
A plain cell — Cell ≡ Source<KeepLatest> — is the keep-latest instance of the
source kind; this is a default type parameter, not a spec assertion. A binding
MAY implement Source<T> as that instance or keep it as a distinct fast path with
identical semantics.
The eager construct is an eager Computed, not a distinct kind. The
eager construction is computed(compute).eager(): a guarded Computed plus a
puller Effect that reads it on creation and after every invalidation, so its
value is materialized by the time the invalidating set/batch returns (readers
never see an intermediate unset state — relaycell-backpressure-analysis.md
§4.0). .eager() is declarative and idempotent and returns the same
Computed handle, mutated — so the per-write puller of the old Signal cannot be
constructed (see Eager computed cells under Semantics, and §9.2.2’s theorem that
a writer is always a sink). .lazy() is the reverse transition; is_eager() is
the predicate, so the bare verbs are never confused with a query.
The normative eager semantics are four observable clauses — materialize once at
creation, fresh at mutator return, once per flush rather than once per write, and
disposal that removes only the puller. They are stated under Eager computed cells
below and fixtured in conformance/reactive-graph/signal_*.json. The
computed-plus-puller construction is the recommended way to satisfy them, not
itself a requirement.
Values are lazy by default; call .eager() on a Computed when eager push
semantics are required. Handles are the two concrete types —
Source<T, M>, Computed<T> — plus EffectHandle: lightweight, copyable ids
over a shared node table (arena slots; see Handles and identity), usable only
with the owning context.
Read is on every cell; write is on the source kind. Both handles expose
get (auto-subscribing) — and only get; there is no subscribe, because
observation is a declared dependency edge, never a registered callback (see
Reactives have no observers). Writing is not a supertype/subtype relationship
but a kind restriction: set (replace) and merge (fold under the source’s
policy) live on the inherent impl for Source<T, M> alone, so
computed.set(…) is a compile error — no method found, no trait in sight. A
Computed reads and never writes; a Source does both. The payoff
(relaycell-backpressure-analysis.md §4.0): a composite reader that needs to
accept either kind takes a small per-binding read-only view (an enum, or Go’s
Cell[T] interface below) and the backend chooses the impl — pull-computed,
push-fed source, or polling-computed — behind one type, so ownership of the
mutation (not the type) decides the invalidation source. Where a binding has no
way to restrict methods by type parameter (Go), a read-only interface Cell[T]
carrying Get is reintroduced, with SourceCell/ComputedCell as structs — the
same compile error under a different mechanism (§4 of the design).
API surface
Two constructors and one transition, symmetric with the kernel. source /
computed / .eager() replace the eight old constructors (cell, merge_cell,
computed, memo, slot, signal, get_signal, dispose_signal) — and
because every Computed is now guarded, the old computed-vs-memo distinction
is gone: memo is removed and computed is the guarded derivation.
| Method | Description |
|---|---|
source(value) | Create a Source<T, KeepLatest> — a plain mutable source cell |
source::<M>(value) | Create a Source<T, M> whose write folds under policy M (was merge_cell) |
get(handle) | Read either kind — a Source or a Computed — computing/refreshing a Computed if necessary (auto-subscribes the running computation) |
get_shared(handle) (binding-specific; Rust Context::get_rc) | Optional unified shared-owner read of either kind, avoiding a value clone while preserving get value, refresh, and dependency-tracking semantics (#lzrsgetarc) |
set(handle, value) | Update a Source and invalidate dependents (no-op on ==) — a compile error on a Computed |
computed(compute) | Create a lazy derived Computed, guarded (an equal recompute suppresses downstream — the guard is never optional, and memo folded into this). T: PartialEq, the same uniform bound as source |
computed(compute).eager() | Make the Computed eager (attach a puller Effect). Declarative and idempotent; returns the same Computed handle (was signal) |
computed(compute).lazy() | Revert an eager Computed to lazy (removes the puller only, keeps the value) — replaces dispose_signal; exists only if a binding needs the reverse transition (§9.3.4) |
is_eager(handle) | Predicate: whether a Computed currently has a puller attached |
merge(handle, op) | Fold op into a Source under its policy (⊕; routes through set, so the == guard + store-without-cascade apply) — a compile error on a Computed |
effect(run) | Register a side-effecting computation (a sink); run may return a cleanup closure |
dispose_effect(handle) | Deschedule, drop edges, run cleanup |
dispose(handle) | Tear down any node kind: detach edges in both directions, clear the node, recycle its slot id. Disposing an eager Computed also tears down its puller (§9.3.4) |
scope() | Open a teardown scope: scope.source, scope.computed, and scope.effect create nodes disposed together when the scope ends |
scope.disarm() | Disarm a scope — ending it disposes nothing; its nodes revert to context ownership |
batch(run) | Coalesce several source updates into one invalidation + effect flush |
Semantics
- Pull-based, glitch-free refresh. A
Computedthat reads other cells always observes values consistent with the current inputs. Onget, it first refreshes its own dependencies (recursively, lazy pull), then recomputes only if any dependency actually changed — it never observes a half-updated graph. - Unified shared-owner reads (
#lzrsgetarc). A context may expose a binding-specific shared-owner form ofget(Rust’s local spelling isContext::get_rc). When exposed as a generic alternate read, it is a read mode, not another cell kind: it MUST accept bothSourceandComputed, MUST return the same current value asget, MUST refresh a dirtyComputedin the same way, and MUST register the same dependency edge when invoked through a running computation. It MUST NOT add aClone/deep-copy requirement merely to read the value. Allocation identity, reference-count operations, and inline value fallbacks are binding-level mechanics and are not wire-visible. The formal pins areReactive.readShared_eq_readCell,Reactive.trackedSharedRead_eq_trackedRead, andReactive.trackedSharedRead_registers_edgeinlazily-formal. - Every cell is guarded — one rule, two sides. A cell suppresses an equal
value. On the source side this is the
==guard onset: setting an equal value is a no-op, no downstream cascade fires. On the computed side it is the equality guard on recompute: an equal recompute suppresses downstream invalidation — theComputed’s value version does not bump, so subscribers see no change (matching TC39Signal.Computed). Equality is structural/value equality, not reference identity, so two distinct-but-equal values suppress invalidation.T: PartialEqis the uniform bound on every cell, source and computed alike — the guard is not a mode a caller opts into. - There is no unguarded cell, and no
equals:falseescape. The guard is never wrong: if it suppresses an update you wanted, the value did not encode the change. To always propagate, make the value genuinelyPartialEq-distinct (encode the distinction you care about into the value), or use a merge policy to express accumulate/always-apply semantics. This is not a library toggle, and there is no unguarded constructor —memo(the old guarded form) has been removed becausecomputedis now guarded and the two are the same thing. - Dynamic dependencies. A tracking stack auto-discovers edges on each
recompute: every
getread inside a runningComputed/effect registers a dependency. Stale dependencies from a previous run are removed before re-registering; aComputedthat reads a different set of inputs on rerun has its edge set updated to match. There is no manual subscribe/unsubscribe.
Implementation note. The dedup that keeps edge registration idempotent is an implementation concern, not an observable one — the contract fixes the edge set, not how membership is tested. A binding MAY dedup by linear scan while a node’s degree is small; above a wide-fanout threshold it SHOULD promote to a hash-indexed edge set, so registration stays amortized O(1) in node degree and a wide-fanout graph does not degrade to O(n²) per propagation. The threshold matters in both directions: below it the linear scan is measurably the faster of the two, so an unconditional hash set is a regression on the common low-degree case (
#lzspecedgeindex).The threshold is not portable — measure it per binding. Naively it is where a scan of contiguous ids crosses the cost of one hash lookup, so it moves with both, and the same number can be right or wrong in the same language depending on unrelated choices. In
lazily-rsthat crossover measured near degree 170 with the standard library’s SipHash and near degree 40 once ids were hashed with a multiply-shift finalizer — a 4x shift from changing the hash function alone.lazily-dartmeasured 60 under AOT and 96 under JIT, a 1.6x spread from compilation mode with no code change at all. Copying another binding’s constant is how a promotion threshold ends up making mid-degree nodes slower than the scan it replaced. Across bindings the measured thresholds so far are 32, 64, 128 and 160 — there is no family constant.Measure the hybrid, not the pure crossover. The crossover between a pure scan and a pure index is the wrong number, because a list arriving at degree T pays the full scan plus the one-time index build, with no indexed insert at exactly T to amortize against. Every candidate threshold therefore parks a regression on its own width, and the pure crossover cannot predict where. Sweep the real implementation against the unfixed tree across candidate thresholds and take the knee —
lazily-dart’s pure crossover said 60–96, but the hybrid sweep put the worst-case regression at 1.63x for T=64 and 1.31x for T=128, so 128 shipped.Two ways to get this measurement wrong, both observed:
- Comparing always-indexed against always-scanning inflates the crossover. It charges the index’s overhead on every one-element dependency list to the wide list.
lazily-cppfirst estimated 96 this way against a true 32 — a 3x error. Sweep the threshold constant alone, with everything else fixed.- A width ladder’s narrow rungs cannot answer the low-degree question. A few hundred registrations is noise;
lazily-dartmeasured 1.4x run-to-run variance at widths 2–4 and nearly reported a phantom regression at width 96 from ladder data. Low-degree behavior needs a separate high-repetition harness, not the tail of the ladder.Two further hazards, both observed:
Demotion needs hysteresis, or it thrashes. A dependent list oscillates by one on every recompute, because edges are removed and re-registered, so a single shared promote/demote boundary makes a list sitting at the threshold rebuild its index on every recompute. Demote well below the promote threshold, or do not demote at all.
The cost is severe where the hazard exists and varies far more than the mechanism suggests: ~4x in
lazily-rs, 7.67x inlazily-js, and 21.5x inlazily-kt— each at exactly threshold+1 and within noise at every neighbouring width, which is why a ladder must cluster rungs there or it will not see this at all.It is not universal, and the reason is structural: the hazard needs a list that oscillates by one.
lazily-zighas no demotion path, so nothing can thrash.lazily-dartclears its dependent list wholesale during cascade rather than removing one edge at a time, and no thrash was observable at threshold±1 (0.99–1.05x) — it keeps hysteresis as cheap insurance on the detach path, not because measurement forced it. A binding SHOULD check which shape its own recompute has before assuming either result.A recycled id must not inherit an index. Where the index is held outside the node — a side table keyed by owner — its entries have to be dropped whenever the list is cleared or the owner is torn down. A binding that recycles ids will otherwise alias a stale index onto an unrelated node.
-
Disposal is explicit. Handles are copyable ids, not owners, so dropping every handle to a node reclaims nothing: without an explicit disposal call the node and its edges live as long as the context. A binding whose nodes can outlive their usefulness — anything with subscribe/unsubscribe churn — SHOULD expose disposal for computed and source cells, not only for effects, or a workload whose live size is constant still grows without bound in both memory and propagation cost. Disposal detaches edges in both directions; reading a disposed node afterwards is an error, the same contract as disposing an effect (
#lzspecedgeindex).Detection of a stale handle is bounded, deliberately. A binding that recycles ids and checks only the node’s kind catches a handle naming a disposed node of a different kind, but cannot catch one whose id has been reused by a new node of the same kind — the classic ABA case. Closing that requires generational ids or reference-counted handles, and this spec does not require either: both cost either handle size or the copyability that fan-out depends on. Conforming bindings therefore reject the cross-kind case and MAY admit the same-kind one. Callers must not rely on read-after- dispose failing.
Garbage collection does not substitute for this. The reverse edge set is a strong reference to each dependent, so a long-lived source retains every node that ever read it — the same unbounded growth a manual binding has, arrived at by a different route. A tracing binding that wants reclamation to follow reachability MUST make its back-edges weak; until it does, its disposal story is exactly the explicit one above.
-
Teardown scopes (
scope). A scope records what was created through it and disposes that set when it ends. It bounds teardown, not visibility: a scope’s nodes read parent- and sibling-owned nodes freely, and scoping never restricts what an edge may point at.Ending a scope MUST be observationally equal to disposing each of its members individually — a scope introduces no disposal semantics of its own, it only names a set and a moment. Proved as
disposeScope_eq_disposeAllin the standalonelazily-formalLazilyFormal.Reactivemodule. A binding is therefore free to implement the scope as a bulk sweep rather than a loop, and SHOULD, since reading each node’s kind from the arena at teardown is cheaper than a per-node dispatch.The resulting graph state depends only on the set of members, not on their order or multiplicity (
disposeAll_order_independent). Effect cleanups are a different matter: they are side effects, so the order they run in is observable, and a binding MUST tear a scope down in reverse creation order — dependents before what they read — so a scope never transiently dangles inside itself and every binding runs cleanups in the same sequence. Order is therefore free for the edge bookkeeping and fixed for the cleanups.A scope carries the same hazard as
dispose: ending it tears down its nodes even if something outside the scope still reads them. A binding MUST NOT present scope teardown as safe against that; only reference-counted handles close it, and they cost copyable handles. -
Scope and reachability are different questions. Disposal and teardown scopes answer “this work is over — free it now”, deterministically, at a point the program names. Weak back-edges and reference-counted handles answer “is anyone still using this?”, and answer it whenever the collector or the last release gets around to it. Neither subsumes the other, and a binding SHOULD offer both rather than picking one:
binding class scope reachability non-GC, no destructors (zig) scopes, ended at an explicit deinit/defernone available; explicit disposal is the whole story non-GC, destructors (rs, cpp) scopes, ended by the scope’s own destructor reference-counted handles, opt-in — they cost copyable handles, which fan-out needs tracing GC (js, kt, py, go, dart) scopes, ended by an explicit call weak back-edges, which need no user-facing API Reference-counted handles are opt-in in every class because a source read by two dependents cannot be moved twice, so making the handle an owner forces a clone at every fan-out capture site. Per-node destructor ownership is not a third option: a node an arena-stored closure can capture must outlive the capture, so a handle that borrows the context can only ever be a leaf — the constraint that makes the scope, not the node, the right unit of teardown.
-
Cycle detection. A
Computedthat depends on itself (directly or transitively) is detected during refresh and throws — the graph is acyclic by construction. -
batchcoalesces. Multiplesetcalls insidebatch(run)queue their invalidation roots; at the outermost batch exit the roots propagate and effects flush once. Mutation inside a batch is synchronous; only the invalidation propagation is deferred to the boundary. -
Effects are scheduled, not inline. An effect rerun is scheduled when a tracked dependency invalidates and runs in the subsequent flush (which may be the same tick, at batch exit). A rerun does not start until the previous cleanup completes. Disposal removes pending reruns, runs the current cleanup, and unsubscribes all dependency edges.
-
Eager computed cells (
.eager()). Eagerness is not a kind — it is aComputedwith a pullerEffectattached, produced bycomputed(compute).eager(): the effect reads it on creation and after every invalidation, forcing it to re-materialize. Because the puller runs inside the invalidatingset/batch’s effect flush, the value is fresh by the time the mutator returns..eager()is declarative and idempotent — a second call is a no-op, so aComputednever acquires two pullers and the per-write over-compute is structurally unrepresentable..lazy()(or disposing theComputed) reverts it to lazy behaviour (the backing value stays readable but is no longer eagerly kept fresh);is_eager()reports the current state.Normative eager semantics. The four clauses below are what a binding conforms to. They are stated as observations a caller can make, so that any implementation strategy satisfying them conforms — see Composition is recommended, not required below. (The conformance fixtures still carry the historical
signalfilenames; the concept is an eagerComputed.)- Creation materializes once.
computed(compute).eager()MUST runcomputeexactly once at creation and MUST NOT expose an intermediate unset state. A reader immediately after creation observes the computed value without triggering a compute of its own. - Fresh at mutator return. After a
setthat invalidates the eagerComputed’s dependency cone returns, its value MUST already equal whatcomputeyields from the current sources, with no intervening read. This is the clause a lazyComputeddoes not satisfy, and it is the operational meaning of “eager”. - Once per flush, not once per write. Inside
batch(run), an eagerComputedwhose dependencies are written N times MUST re-materialize once, at the outermost batch exit — not once per write. The puller is an effect and obeys Effects are scheduled, not inline; aComputedthat re-materializes during invalidation rather than during the flush violates this clause even though it satisfies (2). N writes inside a batch MUST produce exactly one compute. .lazy()removes only the puller. Reverting an eagerComputedto lazy MUST dispose the eager puller and MUST NOT dispose the backing value. After.lazy()the value remains readable, remains correct on read (it reverts to lazy recompute-on-read), and MUST NOT re-materialize on write. This is why the operation is a state transition back to lazy rather than a teardown, and why the olddispose_signalnaming was an inaccuracy.
Composition is recommended, not required. Clauses 1–4 are observable. “A guarded
Computedplus a puller effect” is the construction that satisfies them and is what every binding SHOULD use — it is five lines over the public API and needs no teardown special case. It is deliberately not aMUST, because which nodes exist internally is not something a caller can observe, and this specification does not mandate unobservable representation (the same rule that lets a binding choose weak back-edges). A binding that welds eagerness into its computed-cell invalidation path conforms if and only if it satisfies all four clauses — and clause 3 is the one such a binding is most likely to fail, because re-pulling during invalidation is earlier than the flush.A binding MUST NOT make eagerness a node kind in its graph representation: the kernel’s node enumeration is
SourceCell,ComputedCell,Effect— the two value-bearing cell kinds plus the sink. An eagerComputedis aComputedCellwith anEffect, not a fourth kind to dispatch on. This is now aMUST NOTrather than the oldSHOULD NOT, becausecomputed().eager()makes the composition the only way to build eagerness: the handle a caller holds is theComputeditself, read with ordinaryget, and there is noSignaltype left to ship. The kernel stays closed because the DAG positions are closed — the same reason this specification declines to mandate weak back-edges is why it need not police a construction that cannot be written.Conformance:
conformance/reactive-graph/signal_*.json(historical filenames).Measured 2026-07-20. What replaying the three signal fixtures against every context each binding ships actually found. Recorded as measurements, not as inferences from which types exist — the discipline established by the capability table above.
Binding Clauses 1, 2, 4 Clause 3 Construction lazily-jspass pass composed lazily-ktpass pass composed lazily-cpppass ( Context)pass composed lazily-pypass failed, fixed welded → composed lazily-dartpass failed, fixed welded → composed lazily-gopass failed, fixed welded → composed → Memo-backed lazily-zig4 fails on AsyncContextfailed on ThreadSafeContext, fixedcomposed Clause 3 caught four bindings across two unrelated mechanisms, and every one of them produced correct values while doing 2–3× the computes — which is why none of the other 11 reactive-graph fixtures saw them.
Mechanism A, welded eagerness (py, dart, go). Three bindings independently grew a signal-specific slot subclass whose invalidation handler re-pulled the signal inline. Re-pulling during invalidation is earlier than the effect flush, so the compute count scaled with the number of changed sources. No effect existed anywhere in the construction. Nobody coordinated this; the same wrong design was reached three times in three languages.
Mechanism B, a batch that does not bound the flush (zig).
setflushed effects unconditionally whilebatchonly nested a depth counter, so a correctly composed puller still ran once per write. A composition is not sufficient if the batch boundary does not gate the flush.The async surface is the family’s weakest, systematically.
lazily-cpp’sAsyncContextexposes no signal API at all and its async slots carry no dependency graph;lazily-zig’s async context has no lazy mode, so every derived slot is eager whether or not a puller is attached and clause 4 is unobservable there;lazily-pyandlazily-dartship no async signal constructor. A binding MAY omitsignalfrom a context — clause conformance is per surface, and a context that does not offer the constructor is not non-conformant. What it MUST NOT do is offer it and diverge silently.lazily-gowas the deepest case and is now resolved. ItsMemoimplemented the==guard by recomputing during invalidation, so a memo-backed signal could not satisfy clauses 3 and 4 while a slot-backed one lost equal-recompute suppression — neither could hold, because the cascade consumed reverse edges as it walked. Marking a dependent clean without recomputing meant it never re-registered, so its source could no longer reach it and the next write was lost at depth two, which ishybrid_serves_stale_value_at_depth_two.Its sync plane now uses a non-consuming mark-frontier walk with a pull-time guard, converged on
lazily-rs’s model rather than a new design. Two details from that convergence are worth recording for any binding attempting the same migration, because both were initially judged redundant and both were forced back by the corpus:- Notifying dependents when a slot’s value actually changed is what makes the pull walk order-independent. Without it, a dependent that refreshes in an unlucky order observes “no dependency changed” and keeps a stale cache. Full-cone marking does not subsume it.
- A force-run flag on effects. Without one the guard cannot reach effects at all: a transitively scheduled effect runs before the memo can suppress it. The pull-time re-mark must not re-schedule effects, or an effect re-runs itself mid-flush.
Its
AsyncContextwas deliberately scoped out and still does value-only suppression where the sync plane suppresses downstream entirely. That divergence between a single binding’s two planes is recorded in the binding rather than resolved here. - Creation materializes once.
Reactives have no observers (#lzdartobservercow)
No reactive exposes an observer API. Not a Source, not a Computed,
eager or lazy — no kind of cell. No subscribe, no on_write, no on_change, no
add_listener, no callback collection of any kind attached to a reactive node. A
binding MUST NOT provide one, and MUST NOT carry per-node storage
reserved for one.
The clause is stated on reactives rather than on Cell because it was first
written too narrowly and a second registry survived it: lazily-py carried a
Signal.subscribe — documented in its own docstring as “an external
(non-reactive) change callback”, with the real graph edges tracked separately —
that a Cell-only prohibition did not reach. It deduplicated by equality via a
set, the same defect this section elsewhere calls a MUST NOT. If the rule is
worth having it is worth having on every node kind, so it is written that way.
This is a MUST NOT about a mechanism rather than a behavior, which is unusual
for this document. It is stated that way because the mechanism cannot be made
safe by constraining it — every constraint below was tried, written down, and
abandoned.
Observation is a graph edge, not a callback
Reading a cell inside a computation declares a dependency. The tracking stack records the edge, invalidation propagates structurally, and the graph decides when dependents run. Nobody registers anything. That is the whole design, and every guarantee the family makes — batching, glitch-freedom, coalescing, cone-settled consistency — follows from the graph knowing what depends on what.
A callback list attached to a cell knows none of that. It cannot batch, because it has no notion of a cone to settle. It cannot be glitch-free, because it fires mid-update by construction. It cannot participate in scope teardown, because it is not a node. It is the observer pattern, living inside the reactive primitive and bypassing it.
This is the substance behind a common objection: “isn’t a reactive just an
observer with extra steps?” While Cell.subscribe existed the answer was
embarrassing, because in four bindings it was literally true — there was an
observer registry inside the cell, and callers could reach it. The honest answer
is that a reactive is not an observer, and the way to be able to say so is to not
ship one. Observation is declarative and structural; if a caller is registering
callbacks against a value, they have left the reactive model, and the library
should say so rather than provide a door.
How to tell an edge set from a registry
A binding auditing itself against this clause needs a test that does not depend on what a collection is called, because naming is exactly what hid these:
Anything that survives an invalidation is not a graph edge.
Dependency edges are re-discovered on every recompute — the tracking stack clears them and the next run re-registers whatever it actually reads. That is what makes dependencies dynamic. A collection that persists across invalidation is therefore not participating in dependency tracking, whatever its name, whatever its docstring, and whatever it sits next to.
This criterion is stated because it was arrived at expensively. lazily-py
carried three registries — on Cell, on Signal, and on Slot — and every
one of them was either labelled or assumed to be dependency-graph state. Two
separate readers, with the source open, misclassified one each. In every case the
registry sat beside a real edge set with a similar name (_subscribers next to
_parents), and in every case the deciding fact was a single line: the edge set
is rebound and cleared on invalidation, the registry is iterated and kept.
Apply the criterion mechanically rather than reading intent. It resolves all three without argument, and it is the only check here that does not require trusting a description.
Effect and observer are not two spellings of one thing
The distinction is worth stating flatly, because the two look interchangeable at the call site and are not:
| Effect | Observer (removed) | |
|---|---|---|
| Registration | implicit — reading a value inside the body declares the edge | explicit — hand a callback to a node |
| Position | a node in the graph | a callback list hanging off a node |
| Runs | once per settled cone | once per write |
| Batch | honours it — one run per batch | ignores it — one call per write |
== store-guard | sees the coalesced result | cannot see a suppressed write at all |
| Computed guard | respects it — no run on an equal recompute | not subject to it |
Merge ⊕ | sees converged state — the only guaranteed value | sees a flush-timing artifact (below) |
| Glitch-free | yes — inputs are mutually consistent | no — fires mid-update by construction |
| Dependencies | dynamic; re-discovered every run | none; bound to one node forever |
| Teardown | disposed with its scope | manual, and outlives its scope |
| Per-node cost when unused | zero | storage on every node |
Coalescence is the row that matters most, and it is not one mechanism but
five. This family coalesces at every layer: the == store-guard drops an equal
write entirely; the computed guard drops an equal recompute so downstream never
learns; batch folds many writes into one invalidation and one flush;
store-without-cascade skips effect scheduling for a cell whose cone holds no
effect; and the merge algebra folds a run of ops into one state through ⊕.
Every one of those is the graph deciding that some change does not need to be
propagated — which is most of what makes a lazy reactive graph cheaper than
recomputing everything.
Merge coalescence is where an observer fails hardest, and it is worth spelling
out because it is not a quality-of-implementation matter. Associativity is the
irreducible law of MergePolicy, and what it buys is variable flush points: a
bounded relay may flush at any post-merge watermark and converge identically. The
converged state is guaranteed; the sequence of intermediate values is not.
Two runs of the same program over the same ops may legitimately produce different
intermediate values under different backpressure, buffer sizes, or transports.
An Effect reads the converged state, which is the value the algebra actually
promises. An observer fires on intermediate writes — so what it receives is an
artifact of flush timing rather than data, and it is non-deterministic by
design, not by defect. A caller cannot write correct code against it, because
there is no contract there to be correct against.
Last-writer-wins sharpens this to a point. KeepLatest is old ⊕ op = op — the
new op annihilates the previous state — and Cell ≡ Source<KeepLatest>,
so every plain source cell in the family is already an LWW instance. Under a timestamped
LWW register (CrdtJoin<LwwRegister>) a losing write is dropped outright: after
convergence, it never happened. An observer that fired on that write reported an
event the system has since decided did not occur, and any side effect it took —
a log line, a queue push, a paint, an outbound message — is now describing a
state that no replica will ever agree existed. An Effect never sees it, because
it observes only what survived the merge.
That is the general shape of the whole objection, stated at its most concrete: observers report writes; the system’s actual semantics are about values that survive. Those are different questions, and in a coalescing, converging, distributed graph they diverge constantly.
An Effect is downstream of all four. It sees what the graph decided was worth
propagating, which is why it can be glitch-free and why an unobserved subgraph
costs nothing to write to.
An observer is upstream of all four and outside all of them. It fires on the raw write, before coalescence has a chance to apply. That is not a different observation strategy; it means a caller holding an observer is looking at a system whose central optimization is invisible to them, and cannot tell the difference between a change that mattered and one the graph suppressed.
Use an Effect
Everything an observer expressed, an Effect expresses:
// instead of: cell.subscribe(cb)
ctx.effect(|ctx| cb(ctx.get(&cell)))
The effect is batched, glitch-free, participates in teardown scopes, and is
disposed by handle. Where a caller wants both the previous and the new value —
the state-machine on_transition shape — the effect captures the previous value
in its closure; see lazily-rs state_machine.rs for the reference form.
The behavioral difference is real and intended: under a batch, an effect
observes the settled value. Writing A → B → C inside one batch reports
A → C. Intermediate states are not observable, because that is what a batch
asserts. A caller who did not want that should not have opened a batch.
Use a Topic when you need every transition
A Cell is a value: latest-wins, batched, glitch-free. A stream of every
transition is a different thing, and the family already has it — Topic, present
in all eight bindings, with cursors and durability. Topic.subscribe keeps its
name because a topic genuinely is a subscription: an ordered stream a consumer
reads at its own position.
The design error this section removes was a Topic hiding inside a Cell. A
consumer needing every write — a mutation log, a replication feed, a persistence
tap — should publish to a topic. That also makes the cost honest: only machines
that actually expose a stream allocate one, and a plain cell pays nothing.
Why not keep it, constrained
Recorded so the argument is not re-run from scratch. Cell.subscribe was
specified in detail before being removed, and each clause below was a genuine
attempt to make it safe:
- Firing order is registration order, because a
set- or hash-backed collection reorders on rehash andlazily-go’s map iteration is deliberately randomized. Four bindings had four answers. - No deduplication by callback identity or equality, because two components
subscribing the same bound method silently share one registration and the first
to unsubscribe cancels the second.
lazily-pydeduplicated by equality,lazily-zigby address. - Subscribe during notify is deferred, or a self-feeding observer extends the loop it is running in and never terminates.
- Unsubscribe during notify takes effect immediately, because in a
manually-managed binding
unsubscribeis routinely the step before freeing the state the callback reads, so one more call is a use-after-free. - Disposers latch, or an unlatched second call removes a later registration belonging to a caller who never asked.
- Delivery is per write and
batchdoes not coalesce it — which put the mechanism in permanent conflict with the batching model it sat beside.
Six clauses, four bindings, and the specification was still wrong twice in a single day: it omitted a violated clause in one binding, and its central argument cited two bindings that had never implemented the mechanism at all. A primitive requiring six normative clauses to be safe, that still diverges across the family, and whose delivery discipline contradicts the surrounding model, is not under-specified. It is misdesigned.
Two things settled it: memory carried by the graph itself, and semantics that were footguns at the edges.
The memory is measured, not estimated. Removing the observer API from
lazily-zig moved @sizeOf(Cell(u64)) from 168 bytes to 32 — 136 bytes,
81%, reclaimed from every cell in every program whether or not anything ever
registered. It was not only the callback collections: the reentrancy counters
(notify_depth, before_notify_depth, and two tombstone flags) and the
monotonic registration counter were all unconditional per-node state that existed
solely to make the notify loop safe. At cell-family scale that is the dominant
memory term in the graph, paid by every reactive value to support a feature with
one caller family-wide. A reactive graph’s whole value proposition is holding
many nodes cheaply; a per-node cost multiplied across the graph is the one kind
of overhead it cannot absorb.
The semantics were worse, because the failures were all at the edges where they
are hardest to find. Delivery that ignores batch while everything beside it
honours it. Firing order that depends on a collection’s rehash. Two components
sharing a bound method and silently sharing one registration, so the first to
unsubscribe cancels the second. A disposer that removes a later caller’s
registration. An observer invoked once more after asking to stop, which is
harmless under a tracing collector and a use-after-free without one. Each of
these is fine in the common case and wrong in a case the caller cannot see
coming, which is the definition of a footgun rather than a bug: correct code and
broken code look identical at the call site.
Underneath all of them is the defect no clause could patch: an observer cannot
distinguish batching from coalescence. It receives a flat sequence of callbacks
with no framing. Three invocations may be three separate updates or one logical
update whose writes were grouped, and nothing in the callback distinguishes them
— there is no signal for where an update begins or ends. Nor can it see what the
graph coalesced away: a write dropped by the == store-guard, a recompute
dropped by the computed guard, a flush skipped because the cone held no effect. So
“the value did not change”, “nothing was written”, and “the graph decided this
did not need propagating” are all the same non-event to an observer. It is an
event stream stripped of both its transaction boundaries and its elisions.
This is not a missing feature to be added. It follows from sitting outside the
graph: the graph is what knows where an update starts and stops, and a callback
list attached to one node is structurally unable to observe that. An Effect has
the framing for free, because running once per settled cone is the transaction
boundary. The two mechanisms are therefore not two ways of observing a value —
one can express “this update is complete” and the other cannot, at any level of
specification effort.
Conformance
There are no observer fixtures. The clauses above were removed along with the
mechanism, and conformance/reactive-graph/observer_*.json no longer exists. A
binding conforms to this section by not having the API.
lazily-rs, lazily-cpp, lazily-js, and lazily-kt never implemented one and
require no change. lazily-py, lazily-dart, lazily-go, and lazily-zig
carried one and remove it, re-expressing on_transition as an effect.
The reactive-graph fixtures that remain cover disposal, teardown scopes, and
eager computed cells. They require TeardownScope (ctx.scope() / disarm())
and dependency-graph introspection (dependents_of, dependencies_of,
cleanup_order). Every binding replays this corpus as of 2026-07-19.
The eager-computed fixtures (historical filenames signal_*.json) need one
observable the rest of the corpus does not: computes_of. It maps a node id to
the cumulative number of times its compute function has run, counted from the
start of the scenario. A runner MUST count every invocation of the compute,
including the one at creation, and MUST NOT reset it per step. This key exists
because an eager Computed and a lazy Computed return identical values for every read
sequence — the only caller-observable difference between them is when compute
runs, so a corpus that asserts values alone cannot distinguish computed().eager()
from computed() and will pass against a binding that implements the former as the
latter.
Three ops are specific to these fixtures. The caller-facing transitions are
.eager() / .lazy() (§Eager computed cells); the fixture ops retain the
historical signal / dispose_signal names, which a runner maps to those
transitions (create-eager and revert-to-lazy). A runner MUST accept these op
names — the runner panics on an unknown op, so this acceptance is the contract:
| op | shape | meaning |
|---|---|---|
signal | {id, reads, offset} | create a guarded Computed (compute is sum(reads) + offset, the same convention as computed) and make it eager — the computed(…).eager() construction |
dispose_signal | {id} | revert an eager Computed to lazy (.lazy()) — the puller only, not a node teardown, see clause 4 |
batch | {writes: [{id, value}, ...]} | perform every write inside one batch; invalidation propagates and effects flush once, at the outermost exit |
batch is a single op rather than a begin_batch/end_batch pair so that a
runner need not carry nesting state. Bindings whose batch API is a closure take
the writes as the closure body; bindings with explicit begin/end call them
around the writes. Note that batch also appears in the reliable-sync and
collections areas with unrelated semantics — these are per-area op
vocabularies, not one global namespace.
Fixture shape is declared, not inferred. Every ReactiveGraph fixture
carries a top-level "shape" field, either "steps" or "scenarios". A runner
MUST switch on that field rather than probing for whichever key happens to be
present, and MUST NOT special-case a fixture by filename — the first runner
written against this corpus did exactly that, which goes stale silently the
moment a second scenarios fixture is added. The schema suite cross-checks the
declaration against the keys actually present, so shape cannot drift from the
fixture it describes.
The two shapes are not interchangeable and the split is deliberate. A steps
fixture asserts a single trace. A scenarios fixture asserts a relation
between two op streams — scope_teardown_equals_fold_of_disposals.json claims
that ending a scope is observationally equal to disposing its members
individually, and a single steps array structurally cannot express “these two
paths must agree.” It names the scenarios that must agree in
expected.observationally_equal.
One trap in that fixture, stated here because every runner will hit it:
cleanup_order is cumulative across a scenario, not per-step. The
individual_disposal scenario spreads three disposals across three steps and
pins the whole resulting order on the last of them, while scope_teardown
produces all three from a single end_scope. A runner reading cleanup_order
per-step will see the scenarios disagree and report a divergence that is not
there.
The merge algebra and Source<T, M> (#relaycell)
A Source<T, M> is a source cell whose write is a merge rather than a
replace: merge(handle, op) computes ⊕(current, op) under MergePolicy M
and routes the result through set — so the == store-guard,
store-without-cascade, and batch all apply unchanged. A plain Cell is
exactly Source<KeepLatest> (the keep-latest instance, the default of the
one source kind); a binding MAY implement it as that instance or keep it as a
distinct fast path with identical semantics. The policy M is a real parameter
of the Source<T, M> handle, so it exists exactly where writes exist and is
absent on the computed side — never a third parameter every signature must spell.
Theorem — merge policies must be cheap (§9.2.1).
MergePolicy::mergeMUST NOT block and SHOULD be O(1)-ish in the size ofold. The algebra is the convergence guarantee, so the fold stays synchronous; the async or expensive work that produces an op lives in theEffectthat feeds the cell, never in the fold. In aThreadSafeContextthe fold runs under the mutex, so an expensivemergeholds the lock for its duration. This constrains implementations rather than observable behaviour, so it is review-enforced, not fixtured — the same construction the observer prohibition uses.
Feeding a Source from another reactive
A recurring question, answered here because the answer is not obvious and the failure mode is one cycle detection cannot see.
A Source never acquires a dependency edge. That is what makes it a
source: source and computed partition the graph by incoming edges, so a node fed by
the graph would be both, and Computed → Source → Computed would become
constructible. So “feed this source cell from that reactive” is not a new
capability on the cell.
Theorem — a writer is always a sink (§9.2.2). Writing is not having a value: anything whose job is to write reads something and produces no readable value, which is incoming edges and no value — the
Effectposition. So a network-fed cell, a feedback writer, and an eager puller are each anEffect, not a new node kind. The kernel is closed because the DAG positions are closed. The family reached this answer three times from three directions — eager values (proposed asSignal), feedback (proposed asFeedbackEffect), and writer encapsulation — and each was anEffectcomposed withset/merge/.eager(). Stating it once retires the question; only the ergonomics (a namedfeed_asyncconstructor) stay open, and those wait on two real call sites.
So feeding a source is an Effect that reads the reactive and calls merge:
effect(|ctx| { merge(acc, ctx.get(upstream)) })
The edge belongs to the effect. The cell stays edge-free, the partition holds, and the construction needs no new node kind — the same answer the family gave for eager values.
Delivery is per settled cone, not per write. The effect runs once per flush
carrying the post-coalescence value, so 1 → 2 → 3 inside a batch produces
one merge, of 3. This is a feature rather than a limitation: the
intermediates were never materialized, because eliding them is what a lazy graph
is for. An Effect is the only mechanism with the framing to do this correctly —
it sees a settled value at a boundary, where an observer would see raw writes
with neither.
Therefore merge granularity is flush granularity, and this MUST be stated
wherever the construction is offered. With a non-idempotent policy (+, count,
append-to-log) the accumulated result depends on how writes were batched: three
unbatched writes fold three times, the same three writes inside a batch fold
once. That is not a defect — it follows from Effects are scheduled, not inline —
but the accumulator is the case callers reach for first, and it is the case where
the difference is visible.
For an exact fold over every operation, do not drive it from a dependency
edge. Drive it from explicit merge calls or from a Topic:
| driver | merges performed | fold is over |
|---|---|---|
explicit merge() calls | one per call, batched or not | every op — exact |
Topic subscription | one per event | every op — exact |
| dependency edge, via an effect | one per settled cone | settled values — flush-granular |
The first two are exact because the caller or the topic decides how many
operations exist. The third is flush-granular because the graph decides, and
deciding not to produce an intermediate is the graph working as designed. This is
the same event-versus-state split recorded in
relaycell-backpressure-analysis.md — retained events versus coalescible state —
and it is why the answer to “I need every transition” is Topic, not a new
capability on a reactive.
merge folds synchronously inside a batch; only propagation defers. This
follows from merge routing through set and from Mutation inside a batch
is synchronous, but it is stated explicitly because “does batching lose my
merges?” is the first question the accumulator case raises. It does not. Every
merge call folds when it is called. What a batch defers is the invalidation and
the flush, so N calls inside a batch produce N folds and one
invalidation.
Feedback: the construction cycle detection cannot see
An effect that reads R and merges into M, where M is upstream of R,
closes a loop through the scheduler rather than through the graph. It is not
a dependency cycle, so the acyclicity check will not fire.
Theorem — feedback is when an argument is also a dependency (§9.2.3). An
Effectrelates to a cell in two ways that look alike and behave nothing alike. A write target is an argument: passed in, captured by the closure, known statically, creating no edge — and, becauseset/mergeare kind-restricted toSource, an effect cannot even take aComputedas a write-argument (it does not compile). A read is a dependency: discovered at run time by the tracking stack and re-discovered on every rerun. Feedback is exactly the case where an effect writes a cell it also reads — a write-argument that is also indeps(E). Nothing about that is a graph cycle (the argument is no edge), which is why acyclicity never fires and the loop closes through the scheduler. Prefer this phrasing over “an effect that writes into its own dependency cone,” which makes a scheduler property sound like a graph one.
This is deliberate and it is the family’s only way to express feedback: the dependency graph is acyclic by construction, so a cycle cannot be an edge. Closing it through the scheduler makes each iteration a flush, which bounds it in time and makes it observable — a discrete-time recurrence rather than an unbounded walk:
x_{n+1} = x_n ⊕ f(x_n)
Termination needs two properties, and the policy’s algebra supplies only one.
Where ⊕ is a join on a semilattice — idempotent, commutative, associative,
the properties a CRDT policy carries — every step satisfies x_{n+1} ⊒ x_n, so
the state traces a non-decreasing chain. That is the ascent, and it is all the
join gives you.
It does not give you termination. The chain stabilizes only if the lattice
also satisfies the ascending chain condition — no infinite strictly
increasing chains — and idempotence, commutativity and associativity imply
nothing about chain height. A G-Set or OR-Set over an unbounded domain is a
perfectly good join semilattice whose chain ascends forever if f keeps
producing fresh elements; LWW over unbounded timestamps is the same. Most CRDT
policies are not finite-height. A policy declaration therefore does not
certify that a feedback loop over it terminates; ACC is a separate property of
the value domain and must be established separately.
ACC and bounded height are different properties, and the weaker one is the one
that matters. ACC says no ascending chain is infinite. Bounded height says
there is a single constant H bounding the length of every chain. Bounded height
implies ACC; the converse fails. A witness: take ⊥, a top ⊤, and for each
n ≥ 1 a disjoint ladder (n,1) ⊏ … ⊏ (n,n), with joins across different
ladders landing on ⊤. Every ascending chain is finite — so ACC holds — while
chain lengths are unbounded, so no H exists. Termination needs only ACC. The
distinction is recorded because a flag can only ever declare the stronger
property (see below), so the two must not be written as synonyms.
Where ACC does hold, the == store-guard is the fixpoint detector: once the join
stops moving the value, nothing invalidates and the cascade ends. f need not be
monotone for the ascent, which is why this is not the classical
monotone-framework result — Kleene iteration requires a monotone transfer
function and delivers the least fixed point. This construction reaches a
fixed point and offers no leastness guarantee.
The termination condition, stated correctly, is x ⊕ f(x) == x — the loop
ends exactly when the store-guard suppresses. Two consequences that an earlier
draft of this section got wrong:
- An identity is sufficient, never necessary, and need not exist. If
f(x)is the policy’s identity the step is a no-op, but any absorbing or saturating value is equally a fixed point — aSumat its maximum absorbs every op. And aMergePolicyis only required to be an associative fold; nothing mandates a unit.KeepLatestis a right-zero band and has none, so “terminates whenfyields the identity” is not merely false there, it is undefined. - Correspondingly, a non-idempotent policy does not diverge by construction.
It has no guaranteed fixed point; whether it reaches one is a property of
fand the value domain, and the caller owns that argument.
The three termination classes
The split is not two-way, and the default policy is in the third class.
⊕ | recurrence | termination |
|---|---|---|
| join with ACC | monotone ascent, no infinite ascending chain | always halts |
| join without ACC — G-Set, OR-Set, LWW over unbounded domains | monotone accumulation | halts iff the accumulated set is finite; semi-decidable |
KeepLatest and other idempotent-but-not-commutative bands | x ⊕ op = op collapses it to x_{n+1} = f(x_n) | undecidable — unrestricted iteration of an arbitrary function |
non-idempotent — +, append, counters | accumulates | no guarantee; halts at an absorbing or saturating value |
KeepLatest is the case a caller will actually hit, because
Cell ≡ Source<KeepLatest> — an effect that reads a Computed and writes back to
a plain source cell is the most reachable feedback loop in the family. Under a right-zero
band the merge discards prior state entirely, so there is no ascent, ACC is
irrelevant, and the lattice framing does not apply at all. What remains is
x_{n+1} = f(x_n) over unbounded state with arbitrary f, which is Turing
complete: no analysis can decide in general whether such a loop halts.
A caller in that class cannot appeal to the algebra. They must supply one of: a decreasing measure, an iteration bound, or a cancellation observed in the effect body (see below).
Accordingly:
-
A caller building a scheduler-closed feedback loop MUST be able to state why it terminates. The policy’s declared algebra is sufficient evidence only for a join over a domain satisfying ACC. In every other class the caller owes an argument the specification cannot supply.
-
Bindings MUST bound the effect-drain iteration count within a single flush, and report exhaustion rather than spinning. Note this is not re-entry depth: every binding surveyed guards re-entrant flushes and returns immediately, so the loop is a flat unbounded drain at constant stack depth, and a re-entry-depth bound would be pinned at 1 and could never fire.
The bound’s value is deliberately unspecified and is not part of the contract — it is a binding’s tuning parameter, and pinning a number would make a legitimately long cascade non-conformant. What is normative is that the drain has an exit other than an empty worklist, and that taking that exit is observable: exhaustion MUST surface as a distinguishable outcome (a raised error, a reported diagnostic, a context-level status) and MUST NOT be a silent truncation of the cascade. A silently truncated flush leaves the graph in a state no clause of this specification describes — dependents marked dirty with their effects never run — which is worse than the livelock it replaced, because it is indistinguishable from convergence.
Why this was raised from SHOULD. Three flush loops were read directly (lazily-rs
Context::flush_effectsandThreadSafeContext::flush_effects, lazily-jsflushEffects). All three are re-entrancy-guarded flat drains whose only exit is an empty worklist; none bounds anything. A SHOULD that no surveyed binding implements is not a requirement, and the failure it permits is a silent livelock on a thread that has starved whatever would diagnose it. The bound is also the enabling condition for fixturing this section at all: a conformance runner cannot replay a divergent feedback loop against a binding whose only exit is convergence, because the fixture hangs the runner. This section has stood as unfixtured normative prose precisely because there was no bounded failure mode to assert against.
Where a cancellation can be observed. In a synchronous context the flush does not return to the caller between iterations, so the calling thread cannot observe an interrupt and in fact starves whatever would deliver one. The available yield point is the effect body, which is caller code running once per iteration; declining to write ends the cascade, because an empty worklist is the drain’s only exit. Async contexts additionally suspend between iterations. Unmeasured: whether any binding’s async drain observes cancellation between iterations has not been checked.
Exhaustion SHOULD say what was cycling. A bound that reports only that it was
hit sends the reader to a debugger on a thread that has just stopped livelocking.
A binding SHOULD report the nodes that re-ran repeatedly and, where cheap, the
repeating values — the difference between “drain exhausted after N iterations”
and “drain exhausted; acc and total alternated”. Only the second is
actionable. Representation is a binding’s choice.
Termination state belongs in the graph, not in a closure
The stop predicate that ends a feedback loop SHOULD be a Computed, read by
the effect, rather than a branch buried in the effect body:
done = computed(|ctx| ctx.get(acc) >= threshold)
effect(|ctx| { if ctx.get(done) { return } merge(acc, f(ctx.get(upstream))) })
This is a recommendation about where to put a condition, not new surface — it is
a Computed and an Effect, both already kinds. What it buys:
- It is inspectable. Other nodes may read
done; a UI can show it; another effect can react to it. - It is fixturable. A fixture can assert
donetransitions and that merging stops. A predicate inside an effect body is opaque to the corpus, which is part of why this area went untested. - It is reviewable. The termination argument the clause above requires a caller to state becomes a named node rather than a comment.
The shape is legal under the source/computed partition, which is worth checking
explicitly because it looks circular: acc → done → effect → acc. The final hop
is a write, not a dependency edge, so acc acquires no incoming edge and no
dependency cycle exists. It remains scheduler-closed, one iteration per flush.
It does not make termination decidable. A done that never becomes true
diverges exactly as before; the drain bound remains the backstop.
Accumulators
An accumulator is a Source<M> under an accumulating policy. That is what
the merge algebra is for.
A Computed cannot be an accumulator, for two independent reasons, and the
second is a live hazard:
- No self-edge. A
Computed‘s value is a function of its dependencies’ current values; accumulation is a function of history. Reading its own previous value would be a self-dependency, which the acyclic graph forbids. - Recomputes are elidable, so an impure workaround silently loses data. A
compute body closing over a mutable counter compiles everywhere and looks
correct. But the graph is free not to run it: the guard suppresses a
recompute whose value is unchanged, a
Computedwith no readers never runs, and a batch coalesces N invalidations into one recompute. Each of those drops an increment. A binding MUST NOT document or example this pattern.
The structural rule, stated once because it decides these questions generally:
Memory of its own past implies a
Source. AComputedmust be recomputable from its dependencies, and a value determined by history is not. This is why aSource<M>is a source even though it computes —⊕(current, op)reads its own prior state, which noComputedmay do.
The exception that confirms it: a Computed folding a Topic’s retained events
is an ordinary derivation, because the history is materialized in the topic and
the fold is a pure function of current state.
| pattern | where history lives | |
|---|---|---|
Source<M> + accumulating policy | in the cell (a source) | ✅ |
Computed folding a Topic’s retained events | in the topic | ✅ the fold is pure |
Computed closing over a mutable counter | nowhere reachable | ❌ loses increments under elision |
Recall also that driving merge from a dependency edge is flush-granular, so an
accumulator fed that way counts flushes, not writes. For an exact count, drive
it from explicit merge calls or from a Topic.
There is no fixpoint construct, and this records why
A termination construct was designed for this section and declined. Recorded briefly so it is not re-derived; this is rationale, not specification.
The proposal was a fixpoint restricted to policies declaring a new
BOUNDED_HEIGHT, plus a fixpoint_with(cell, f, measure) taking a
strictly-decreasing measure. Three reasons it failed:
- The flag would be unverifiable.
COMMUTATIVEandIDEMPOTENTare honest because they are refutable by sampling — one counterexample disproves a claim; neither is confirmable, and this document does not pretend otherwise. ACC is not even refutable: every finite chain observed is consistent with it. The stronger bounded-height-with-constant is refutable and is vacuous —Maxover a 64-bit integer has height2^64. A declared property that is unverifiable or vacuous is worse than no property, and that test generalizes past this proposal. - It could not cover the reachable case.
Cell ≡ Source<KeepLatest>, and a right-zero band has no ascent, so an ACC-restricted construct is by construction absent from the class callers actually hit. - A measure parameter adds no capability, and the construct is a composition
promoted to a primitive. A caller who can compute a decreasing measure over
their own state can compute a stop predicate over that same state — which is
the
Computedpattern above, written with kinds that already exist.Signalwas retired for exactly this shape.
What the proposal genuinely offered was diagnosis, which is retained above as a
SHOULD on exhaustion reporting.
Conformance for this section
Three fixtures are specified; they are not yet written.
| fixture | asserts |
|---|---|
feedback_drain_bound_reports_exhaustion | A divergent loop — c a source cell, m a Computed of c + 1, an effect merging get(m) into c under KeepLatest — terminates the flush and surfaces the exhaustion outcome. Asserts no iteration count, which is not contract. |
feedback_converges_below_the_bound | The discriminating negative. The same shape with m = min(get(c) + 1, 3) under Max reaches its fixed point and must not report exhaustion. A binding that reports exhaustion for every feedback loop passes the first and fails this. |
feedback_declining_to_write_terminates | Pins the caller-side exit: an effect that stops merging ends the cascade as ordinary convergence, not an exhaustion report. |
Deliberately absent: the bound’s value, the exhaustion outcome’s representation, and any assertion about how many times a given effect ran.
Unfixtured and unverified, stated explicitly. Until these exist, treat
cross-binding agreement on this whole section as unverified — the eager-computed
semantics sat in exactly this state for months while a binding shipped a
different construction under a green checkmark. Separately unmeasured: whether
any binding’s async drain observes cancellation between iterations. And the
MUST above is currently satisfied by no surveyed binding; it describes
required behavior, not observed behavior.
A MergePolicy is an associative fold ⊕ : T × T → T. The properties it
satisfies are selected by the transport contract, not fixed
(relaycell-backpressure-analysis.md §2):
| Property | Requirement | Purpose |
|---|---|---|
| Associativity | Always — the irreducible law | Regrouping a run of merged ops never changes the converged state, which is what licenses variable flush points: a bounded relay may flush at any post-merge watermark and converge identically. Not a flag; a law every policy MUST satisfy. |
| Commutativity | Per policy (const COMMUTATIVE) | The reordering tax — required only when ops may be applied out of order (concurrent producers / replicas / pages). |
| Idempotency | Per policy (const IDEMPOTENT) | The durability tax — required only for at-least-once / crash-replay. For an idempotent ⊕, re-applying an op is a no-op, which is exactly the == store-guard one layer up: free dedup. |
Policies are open. A binding MUST permit application-defined policies. The
list below is canonical, not exhaustive: a MergePolicy is any associative
fold with honestly declared flags, and a binding that ships a closed enum of
policies has implemented a subset rather than the algebra. Merge semantics are
domain knowledge — a text register, a set of tags, a running quantile, a
per-tenant precedence rule — and the family cannot enumerate them in advance. A
binding MUST expose the policy interface publicly, and SHOULD accept a
custom policy anywhere a canonical one is accepted.
With one obligation attached, and it is the whole cost of the openness. An
application-defined policy MUST declare COMMUTATIVE and IDEMPOTENT
truthfully, and a binding MUST ship the law-test harness that checks a policy
against its own declarations — the same harness that verifies the canonical
policies, exposed for application use.
A policy that misdeclares its flags does not fail loudly. It converges correctly in every single-replica test, every in-order test, and every test without redelivery — and diverges only under the conditions the flag was supposed to license: a claimed-commutative policy that isn’t will disagree between replicas that saw the same ops in different orders, and a claimed-idempotent one that isn’t will drift under at-least-once delivery. Both surface as replicas that silently stop agreeing, arbitrarily far from the code that caused it. That is why the flags are verified rather than trusted for the canonical policies, and the reasoning does not weaken for a policy an application wrote. Shipping the checker is therefore part of shipping the extension point — an unverifiable declaration is worse than no declaration, because the relay acts on it.
The canonical policies (each names its algebraic structure and flags):
| Policy | ⊕ | Structure | Comm | Idem |
|---|---|---|---|---|
KeepLatest | old ⊕ op = op | right-zero band | ✗ | ✓ |
Sum | old + op | commutative monoid | ✓ | ✗ |
Max | max(old, op) | semilattice (total order) | ✓ | ✓ |
SetUnion | old ∪ op | grow-only semilattice | ✓ | ✓ |
RawFifo | old ++ op | free semigroup (concat) | ✗ | ✗ |
CrdtJoin<C> | C::merge_from | join semilattice | ✓ | ✓ |
A replicated cell that accepts writes from more than one authority MUST use a
commutative policy. This follows from the reordering tax above, but it is
stated separately because the default is the trap: Cell ≡ Source<KeepLatest>,
and KeepLatest is not commutative. A plain cell replicated to peers that
also write it will converge differently depending on arrival order — the exact
defect the algebra exists to prevent, arrived at by writing no policy at all. The
multi-writer replicated case wants CrdtJoin<LwwRegister> or another commutative
policy, chosen deliberately.
“More than one authority” is about who may write, not about topology. A
replicated derivation pushed to a peer that only reads it has one authority and
needs no commutativity — on that peer it is read-only, exposing only get
without the Source write methods, which is the read-only replicated cell
shape. Commutativity becomes mandatory the moment the receiving peer may also write.
Derived default with user override. A recurring shape worth naming, because it looks like it needs a new primitive and does not. Where a value has a computed default that a user may override:
-
Co-located — use the graph, not a policy. A
Sourceholding the override (absent = defer), aComputedholding the derivation, and aComputedselectingoverride ?? derived. Glitch-free, self-documenting, and “reset to default” is clearing the override cell. No arbitration exists to get wrong. -
Distributed — a merge policy is required, and a valid one exists: tag ops with provenance and fold by precedence (user outranks derived, LWW within the user rank). It satisfies all three properties — associative because a higher rank absorbs a lower one regardless of grouping, commutative because rank comparison is order-free and ties resolve by timestamp, idempotent because re-applying an op of equal rank and stamp is a no-op.
The hazard is reset-to-default, and it is not obvious. Clearing an override is an op that lowers precedence, and a naive “a clear beats the user rank” rule is not commutative — a clear arriving after a newer user edit would wrongly win. The clear MUST carry a timestamp and lose to any user op newer than itself, which makes it an ordinary participant in the LWW rank rather than a special case. A binding that special-cases the clear will pass single-replica tests and diverge only under reordering.
KeepLatest (positional last-writer-wins, not commutative) is distinct from
a timestamped LWW register (CrdtJoin<LwwRegister>, commutative): both conflate,
they differ only on commutativity — the CRDT-vs-LWW branch. RawFifo cannot
conflate (order and multiplicity are meaning); its only bounded-lossless option is
Spill (Phase 3+). CrdtJoin<C> wires the existing cell CRDT units
(Merge mechanisms) into the algebra without
reimplementing their join.
Verification form. The three properties are algebraic identities over
Tvalues ((a⊕b)⊕c == a⊕(b⊕c),(a⊕b)⊕c == (a⊕c)⊕b,(a⊕b)⊕b == a⊕b), so a binding pins them with property-based law-tests (associativity for every policy; commutativity/idempotency asserted exactly when the flag is set, plus a counterexample proving a cleared flag does not lie) — lazily-rs usestests/merge_laws.rs. The cross-language converged-state determinism invariant (same op multiset, any grouping → same egress) is additionally pinned by themergecell_algebra.jsoncompute fixture.
Invalidation propagation
When set changes a value (post-==-guard):
- The cell’s dependents are marked dirty (computed cells) or scheduled (effects).
- Dirty marks propagate transitively through computed dependents — a dirty
Computedmarks its own dependents, and so on. AComputedthat recomputes to an equal value stops the propagation (the computed guard). - On the next
getof a dirtyComputed, it refreshes: it pulls each dependency (recursively refreshing/ recomputing as needed), and recomputes only if a dependency actually changed value.
This is a push-invalidated, pull-recomputed graph — invalidation travels downstream eagerly (so effects fire), but the new value is computed lazily on read (so untouched branches do no work).
Store-without-cascade (the write-side dual of lazy reads). When set
changes a value whose transitive dependent cone contains no Effect, the new
value is stored (step 1’s dirty-marking of lazy computed dependents still happens, so
a future subscriber reads the current value glitch-free — late-subscribe
correctness) but no effect flush is scheduled — there is no active reactor to
run. A binding MAY skip the flush machinery entirely in this case. Combined with
demand-driven derivation on the read side, an unobserved reactive node —
pull-derived or push-populated — costs approximately its raw storage: the merge
cost law tiers the write cost by dependent kind (none → store only; lazy-only →
store + O(deps) dirty-mark, no flush; active → store + dirty + flush). A burst
of N value-changing writes with no interleaved active read pays the transitive
dirty-mark once (dirty-marking is idempotent and monotonic — an already-dirty
Computed is not re-walked), i.e. N·(==/⊕) + one dirty-propagation. See
relaycell-backpressure-analysis.md §4.0.
Handles and identity
A handle is a Source<T, M> or Computed<T> (or EffectHandle) carrying a
SlotId — and here Slot is the storage concept, not a reactive value: a
slot is the arena position that holds a node, and it holds any kind
(SourceCell, ComputedCell, or Effect). SlotId, SlotValue, and the slab
vocabulary are accurate under this storage meaning and are unchanged by the
kernel rename; the reactive-value node the arena holds is a SourceCell or a
ComputedCell. The slot persists across recycling; its occupant does
not (recycled_id_inherits_nothing).
Slot ids are minted monotonically and recycled on dispose. A disposed handle is
inert: reads on a disposed Computed return its last cached value if any;
reads on a disposed Source, or on a Computed whose eager puller was
disposed, are undefined (the caller MUST NOT retain a handle past disposal).
Re-entrancy of a disposed effect is prevented by removing it from the schedule
before running cleanup. Detection of a stale handle is bounded (see Disposal is
explicit): a binding that recycles slot ids MAY admit a same-kind ABA read unless
its SlotId carries a generation tag.
Context layers
- Single-threaded — the base context (mirrors lazily-rs
Context). The graph is notSend/Sync; it lives on one thread/executor. Unconditionally required of every binding (it is the reactive core). - Thread-safe — a lock-backed counterpart (mirrors lazily-rs
ThreadSafeContext); handles are clonable and the transition function and state areSend + Sync. Effects are scheduled and flushed within the invalidatingsend/batch, preserving glitch-free pull-based ordering. Required of any binding whose platform exposes preemptive multi-threading or shared-memory concurrency — see Wire Protocol § Concurrency layers are required. - Async — a separate reactive surface for future-returning computations; see Async Reactive Context. Required of any binding whose platform exposes an async/future runtime — see Wire Protocol § Concurrency layers are required.
The single-threaded context is the unconditional base; the thread-safe and async
layers are required conditionally. A platform that structurally lacks either
primitive (a strictly single-threaded runtime, a process/actor-isolation model,
or a platform with no suspendable async computation) declares the matching
thread_safe / async capability as none and advertises it, never silently.
The flat State Machine and State Charts
compose with whichever reactive context a binding ships, with identical send
semantics.
Conformance
A reactive context conforms when:
- The kernel’s two cell kinds — nodes
SourceCell/ComputedCell, handlesSource<T, M>/Computed<T>— and the sinkEffectare implemented; an eagerComputed(computed().eager()) is the eager construct — not a fourth kind. - Every cell is guarded on
T: PartialEq:setis==-guarded (equal value is a no-op) andcomputedis guarded on recompute (an equal recompute suppresses downstream). There is no unguarded mode. - Refresh is pull-based and glitch-free: a
Computedobserves consistent inputs; untouched branches are not recomputed. - Dependencies are tracked dynamically through a tracking stack (edges re-registered each recompute; no manual subscribe).
- Cycles are detected and throw.
batchcoalesces into one propagation + effect flush at the outermost exit.- Effects fire scheduled (not inline), cleanup runs before each rerun and on dispose, and disposal unsubscribes edges.
- An eager
Computedis materialized by the time the invalidatingset/batchreturns (eager push). - Read on every cell, write on the source kind. Reads (
get) are on every cell —SourceandComputedalike. Any generic alternate read surface (for example the optionalget_shared/ RustContext::get_rc) MUST cover the same two kinds and preserve ordinary-read value, refresh, and dependency-tracking semantics. Writes (set/merge) are onSource<T, M>alone — a compile error on aComputed, enforced by the type rather than a trait (a read-only interface in Go, §4 of the design). There is nosubscribe— see Reactives have no observers. - The merge algebra (
#relaycell).merge(handle, op)folds under an associativeMergePolicyand routes through the==-guardedset(so an idempotent policy’s no-op merge fires no cascade).Cell ≡ Source<KeepLatest>. Every policy is associative; theCOMMUTATIVEandIDEMPOTENTflags match the policy’s algebra (verified by law-tests); the converged egress state is independent of merge grouping/order for a commutative policy (verified bymergecell_algebra.json).
Declared context capabilities
A binding MUST declare, per context it ships, where that context sits on two independent axes. The declaration exists so the conformance harness can decide which fixtures apply. It has no other purpose, and the constraints below are part of the requirement rather than commentary on it.
Axis 1 — read discipline. Either blocking (reads return values) or async (reads return futures/promises). Mutually exclusive per context.
Axis 2 — concurrent access. One of:
| Level | Meaning |
|---|---|
none | Single-threaded; no concurrent access contemplated. |
serialized | Concurrent callers are linearized against per-thread or per-realm graphs. No graph is shared. |
shared-graph | One reactive graph is genuinely accessed from multiple threads. |
The three-way split on axis 2 is not pedantry — it was added because a two-way
thread-safe: yes/no flag would have been satisfied by four bindings offering
materially different guarantees. Measured 2026-07-19: lazily-rs and lazily-py
share one graph across OS threads; lazily-js shares an Atomics-backed lock
across worker realms while each realm keeps its own graph; lazily-dart is a
single-isolate reentrancy guard with no cross-isolate anything. Every one of those
is individually honest and documented, but a fixture asserting concurrent-access
behavior is meaningful only at shared-graph and vacuous at serialized. A flag
that cannot tell them apart produces exactly the failure this chapter exists to
prevent: a suite that passes while testing nothing.
Feature-group rule. A serialized context is a meaningful execution flavor
when it supplies a distinct linearization boundary for reentrant or concurrent
logical callers, even though its graph is realm-local. It joins every portable
Core feature test that its surface supports (ordering, membership, atomic move,
materialization). It does not join shared-graph stress/model-check tests.
A binding may instead declare the context absent; then it is staged out of that
peer group. Merely publishing a duplicate type is not support: an incomplete
wrapper remains partial/absent, and a test runner MUST NOT fill its gaps with a
base-context object, ad hoc mutex, or ordinary dictionary.
The axes are independent. Read discipline does not imply a concurrency level
and vice versa. Single-threaded async is fully concurrent while requiring no
thread-safety at all, and a blocking context may be shared-graph. A binding
MUST NOT infer one axis from the other.
The declaration is not wire-visible. Context capability MUST NOT appear in
any protocol message, influence sync behavior, or be observable by a peer. The
reactive graph is compute, not protocol; only resolved values cross IPC/FFI, and
convergence is a property of the merge algebra rather than of any replica’s local
execution model. Associativity already licenses variable flush points, so local
scheduling is unconstrained by the protocol — which is precisely why the execution
model producing that scheduling is irrelevant to a peer. A replica running an
async context and one running a shared-graph context converge because ⊕ says
so, not because they agree about threads.
This constraint is stated because the failure mode is quiet. Nobody sets out to leak the execution model into the protocol; it happens when a sync path needs a decision and a capability declaration is conveniently in scope. The declaration answers exactly one question — which fixtures run — and the moment it answers a second, the layering is broken.
Thread-safe context conformance
The lock-backed context (Context layers) is required of any binding whose platform exposes preemptive multi-threading or shared-memory concurrency. A binding’s thread-safe context conforms when it holds these deterministic properties under concurrent access:
-
Handles are clonable, and the transition function and source/computed node state are
Send + Sync; one reactive graph is shared across OS threads.This clause applies only to a context declaring
shared-graph(see Declared context capabilities). Aserializedcontext —lazily-js, whose graph is per-realm, orlazily-dart, whose guard is single-isolate — cannot satisfy it and is not required to. Conformance for those is the linearization property alone: concurrent callers are ordered, and none observes a half-updated graph. Requiring a shared graph of every “thread-safe” context would have made two bindings permanently non-conforming for having chosen a design their platform actually permits. -
Effects run glitch-free under concurrent access, preserving the same ordering as the single-threaded context — a concurrent reader never observes a half-updated graph. This mandates glitch-free ordering (every effect that runs sees the fully settled cone, never an intermediate state), not literal in-lock dispatch. A threaded binding MAY defer effect dispatch out of the graph lock, so a callback may re-enter the context, provided the ordering invariant holds: effects are delivered in dependency order and none observes a mixed state.
(This clause previously governed observer callbacks, under
#lzspecobserverclarify. Observers were removed from the family — see Reactives have no observers — and the requirement now attaches to effects, which are the only remaining way user code runs inside an invalidation wave.) -
The
==(PartialEq) source guard and theComputedequality guard both hold under concurrent mutation: an equal write invalidates nothing, an equal recompute suppresses downstream work. -
The graph lock is released before user compute/effect/cleanup callbacks run, so callbacks may re-enter the same context without deadlock.
-
An in-flight recompute is parked on a per-slot generation/condvar sidecar; a stale completion (the slot was invalidated during compute) is discarded and the waiter retried against a fresh value rather than observing a mixed state.
Verification form. Concurrent interleaving is not a deterministic load/replay sequence, so it is not pinned by a portable conformance fixture in
lazily-spec. Each binding verifies its lock-backed context with a synchronization-model checker over the same semantics above — lazily-rs uses Loom (tests/thread_safe_loom.rs, behind theloomfeature). The five properties above are the contract that model check exercises; a binding with no such tooling MUST at minimum exercise 1–4 under a threaded stress harness.
Implementation status
The single-threaded reactive context is required of every binding that
advertises the reactive core. lazily-rs, lazily-py, lazily-zig, lazily-kt, and
lazily-js implement it. The thread-safe and async counterparts are required of
any binding whose platform supports them (see Wire Protocol § Concurrency
layers are required); a platform
that structurally lacks either declares the matching thread_safe / async
capability as none and advertises it, never silently.
Measured 2026-07-19. Shipping a type named AsyncContext or
ThreadSafeContext is not the same as implementing the capability, so this
records what was found by reading each implementation rather than by counting
type names:
| Binding | async | thread_safe | Note |
|---|---|---|---|
lazily-rs | full | shared-graph | reference; the only binding replaying the reactive-graph corpus |
lazily-py | full | shared-graph | AsyncContext added 2026-07-19 (d115000); it was the last binding without one |
lazily-dart | full | serialized | single-isolate reentrancy guard; isolates share no memory |
lazily-js | full | serialized | Atomics/SharedArrayBuffer mutex shared across worker realms, graph is per-realm |
lazily-go | full | unmeasured | |
lazily-kt | unmeasured | unmeasured | |
lazily-zig | full | unmeasured | cascade rides the publish path rather than the invalidate path |
lazily-cpp | stub | unmeasured | see below |
lazily-cpp’s async context is a stub and MUST NOT be counted as implementing
the capability: AsyncSlotNode carries no dependents or dependencies
fields, get_async unconditionally recomputes on every call, and the
synchronous get() returns a cached value that nothing ever invalidates. Depth
tests “pass” through get_async only because nothing is ever memoized. Per the
rule above it should declare async: none until it has a dependency graph, or
implement one.
The unmeasured entries are deliberately not guesses. Absence of a finding is
not a finding — that lesson is recorded in Reactives have no observers, where
three bindings’ omission from a divergence table turned out to mean nobody had
looked.
lazily Cell Model Specification
Normative source of truth for cell kinds and the multi-write merge mechanism.
This chapter defines the cell-kind model that the Wire Protocol serves.
It is upstream of every transport: IPC, FFI, signaling, and the distributed plane all
carry cells whose convergence semantics are fixed here. The
Distributed: CRDT Cell Plane section
specifies merge: crdt — the first multi-write merge mechanism defined below.
The single axis: writer count
A lazily graph is a set of cells. The only axis that determines how a cell’s value converges is how many writers can concurrently produce a value for it — not whether those writers are local or remote, in one process or many.
| Kind | Concurrent writers | Convergence | Merge |
|---|---|---|---|
Single-writer (local / direct) | exactly one | direct reactive push/pull | none |
| Multi-write | potentially many | merge: <mechanism> ingress | mechanism-defined |
The kind is a static property of what the cell represents, chosen at definition time. There is no dynamic per-write mode switching: a multi-write cell stays multi-write even when only one writer is currently live, and a single-writer cell never becomes multi-write because a value happens to arrive over a wire.
Single-writer cells (local / direct)
A single-writer cell has exactly one writer:
- this runtime’s own derivation graph (a derived cell is always single-writer — see Derived cells), or
- a single owning runtime that one-way mirrors the cell to other runtimes via a
delta projection (today’s
lazily-rs → lazily-kt#lazilystatesyncpush is exactly this shape).
Propagation is direct reactive push/pull over the IPC or FFI channels. A mirror is still single-writer: the receiving side observes, it does not write back. No merge step exists, and none is permitted — a single-writer cell that receives a concurrent remote write is a conformance error, not a merge.
A cell mirrored one-way to N readers is single-writer. Locality does not make a cell multi-write; a second concurrent writer does.
Multi-write cells
A multi-write cell admits concurrent writes from multiple replicas. New values arrive as remote ops merged through the cell’s declared mechanism; the merged result is fed into the reactive graph as an ordinary cell update, after which propagation is identical to a single-writer cell.
A multi-write cell carries a merge: <mechanism> attribute. The mechanism is a
parameter of the cell, not a separate cell kind:
Cell = SingleWriter
| MultiWrite { merge: MergeMechanism }
This framing is deliberate. An implementation MUST model multi-write as one cell
category parameterized by a merge mechanism, and MUST NOT hardcode a single crdt
cell kind. New mechanisms slot in without a new kind, and every mechanism shares the
ingress boundary, the merge-unit granularity, and the downstream propagation rules
below.
Merge mechanisms
crdt is the first mechanism this spec defines and the only one with a normative
wire schema today (distributed.json). The mechanism slot is open by
construction so later mechanisms slot in alongside it. Every mechanism MUST be
deterministic — replicas applying the same op set MUST reach the same value
regardless of arrival order or lag.
merge | Status | Convergence strategy |
|---|---|---|
crdt | normative (first) | Conflict-free replicated data type (yrs/Yjs-family registers); converges without coordination. See Distributed: CRDT Cell Plane. |
lww | reserved | Last-writer-wins by HLC/Lamport timestamp. |
ot | reserved | Operational transform (server-ordered op rebase). |
lease | reserved | Lease/lock-serialized single-live-writer; degenerate-concurrency convergence. |
custom | reserved | Application-supplied deterministic merge function. |
crdt is chosen as the first mechanism because it converges without coordination —
no central ordering authority, no lock round-trip — which is the property the
editor-as-replica use case needs. Reserved mechanisms are named to fix the extension
shape; an implementation MAY reject any mechanism it does not implement, but MUST reject
it explicitly (capability negotiation) rather than silently treating it as crdt.
A multi-write cell with a single live writer degenerates to a near-free merge (no
concurrent ops to reconcile) under every mechanism. This is why the kind is chosen
statically by representation: a shared cell that is usually edited by one replica still
declares merge: so that the moment a second writer attaches, convergence is already
guaranteed.
Merge is an ingress operation on root cells only
The merge mechanism is an ingress step at the boundary where remote ops enter a replica. It applies only to root (input) cells.
Derived cells are never multi-write
A derived cell is a deterministic function of its inputs. It has exactly one writer — the derivation — and therefore is always single-writer. Replicas converge on a derived cell because they converge on its roots, never by merging the derived value itself. An implementation MUST NOT replicate or merge a derived cell directly.
Propagate guard (computed / computed_ripple_when / slot)
A derived cell carries a propagate guard: after it recomputes, the guard decides whether the recompute is forwarded to its dependents. The guard governs propagation, never computation — a derived cell is always recomputed when it is dirty and read (laziness is the only compute gate), so a reader MUST always observe a value current with the inputs. The guard only decides whether dependents are invalidated. This is the glitch-suppression that lets a diamond re-converge without re-running consumers whose input did not meaningfully change.
Three constructors expose the guard; all always compute, and differ only in the
guard predicate changed(old, new) (propagate iff true):
computed(f)— the default. Guards by the value’s natural equality:changed = (old ≠ new). An equal recompute MUST NOT invalidate dependents. This is the#lzcellkernelguarded computed; equality semantics are the host language’s (PartialEq/==/equals), so for reference-typed values (e.g. a fresh array each recompute) suppression is host-defined and MAY not fire — this is permitted (an unsuppressed cascade is a missed optimization, never incorrect).computed_ripple_when(f, changed)— the same guard with an explicit, purechanged(old, new)predicate: a cheaper/custom equality (dedup a large value by a version/hash field; epsilon compare; hysteresis; monotonic gate; or “propagate every N” when the counter lives in the value).changedMUST be a pure function of(old, new); value-carried state is a permitted input, external mutable state is NOT (it would key off recompute/read frequency and break determinism).computed(f)≡computed_ripple_when(f, ≠).slot(f)— pass-through: no guard (changed ≡ true). Every recompute propagates. The escape for values with no cheap equality (!PartialEq, non-comparablecollections) that a binding is fine re-firing. This is a derived constructor and is distinct from the internal storage-senseSlotthat acomputednode occupies.
The guard is proved in lazily-formal as recomputeSlot_equal_preserves_dependents
(equal recompute leaves dependents’ dirty flags untouched) and its specialization
recomputeSlot_ripple_when_false_preserves_dependents for a custom changed.
Dependency tracking (the fortified compute view)
A derived cell discovers its dependencies dynamically: on each recompute it runs
its compute function and records every cell that function reads. The recorded set is
re-bound every recompute (not accumulated), so a conditional read
(cond ? a : b) drops the branch it did not take — an implementation MUST NOT retain
a dependency a recompute did not read.
The identity that a read must attribute to — which node is being recomputed — is
carried into the compute function as a value, through a per-recompute compute
view (lazily-rs: Compute, the sole implementor besides Context of the
ComputeOps operations subset). It is NOT ambient (thread-local / module global).
This is normative because ambient state is clobbered across suspension: an async
compute that reads a dependency after an await would attribute it to whatever else
ran on the executor. A value threaded through the closure survives suspension (it is
captured), so it is the only mechanism that tracks correctly post-await — and the
only one that works where no ambient carrier exists (browser JS has no
AsyncLocalStorage). Bindings whose runtime does provide a suspension-surviving
ambient carrier (Python contextvars, Dart Zone, Node AsyncLocalStorage) MAY use
it; all others MUST thread the value.
The compute view SHOULD be fortified so misattribution is prevented by construction, not convention:
- Sole tracking surface — a tracked read is available ONLY through the compute view; reading through the owning context registers no edge (the explicit untracked escape). A normal read therefore cannot silently miss tracking.
- Non-escapable — the view MUST NOT outlive the recompute (lazily-rs binds it by
lifetime and makes it
!Send), so it cannot be stored and later replayed to register an edge against the wrong node. - Generation-stamped — a read against a node disposed/recycled mid-recompute is detected, never misattributed.
Edge-attribution invariant (normative): every dependency edge registered during
the recompute of node n has n as its dependent. Because the node is a value
parameter of the compute view, this holds by construction. Proved in
lazily-formal as registerReads_dependent_is_recomputing_node.
Effects stay single-writer
Effects (irreversible external actions — send email, charge card, fire webhook) are not multi-write. State convergence does not authorize an effect to fire on every replica. Effects MUST be gated behind a single-writer authority (a designated peer or small consensus group) that decides when the effect fires, at-most-once. See Single-writer effect authority.
remote ops
│
▼
┌──────────────────────────┐
│ merge: <mechanism> │ ← ingress, ROOT cells only
│ (crdt | lww | …) │
└──────────────────────────┘
│ merged value as ordinary cell update
▼
root cell ──► derived cells ──► effects
(single-writer) (single-writer authority)
└─ direct reactive propagation, identical for all kinds ─┘
Cell = merge unit
The cell is the unit of merge. Each multi-write cell converges independently; a merge mechanism operates within one cell’s value and MUST NOT move content across cell boundaries.
This is normative because the cell graph already supplies the natural merge boundaries: making the cell the merge unit makes cross-cell contamination impossible by construction. A whole-document merge that splices one logical region’s content into another (for example an agent’s console output bleeding into a queue region) is a conformance violation — each region is a distinct cell and converges on its own.
An implementation MUST scope every merge to a single cell. Coarser-grained merge (whole snapshot, whole document) is permitted only as an optimization that is observably equivalent to per-cell merge; if it can produce a result no sequence of per-cell merges could, it is non-conforming.
The per-cell merge ⊕ is characterized as an associative fold by the
merge algebra
(#relaycell): a MergeCell<T, M> is a cell whose write folds under a
MergePolicy M, and a plain Cell is MergeCell<KeepLatest>. The
merge mechanisms below are the semilattice policies
(CrdtJoin<C>) of that algebra; associativity is the invariant that lets a
bounded relay flush at any watermark and still converge (Phase 2+).
Liveness vs mechanism
Whether a cell is multi-write (merge: present) is static. How many writers are
live against it at a given moment is dynamic, governed by an attach/detach
authority state machine (e.g. an editor plugin attaching or detaching). The authority
SM governs only liveness — it never changes a cell’s kind or mechanism:
- No live writer → the cell still declares its mechanism, but with zero concurrent ops the merge is inert and a durable replica MAY be ephemeral (rebuilt from a checkpoint on demand).
- One or more live writers → ops flow and the declared mechanism reconciles them.
Mechanism is a property of the cell’s meaning; liveness is a property of the current session. Conforming implementations MUST keep these independent.
Cross-process liveness as a CRDT cell. When session liveness itself must cross a process
boundary — “editor pid X has doc Y open”, “pid X holds the owner lease” — it is modeled as an
ordinary multi-write cell on the CRDT plane, not as out-of-band state: an OR-set for open-set
membership (observed-remove, so a re-open wins over a lagging close) and an LWW register for the
per-pid alive flag / lease. The derived “is this doc live” aggregate is then a plain reactive
memo over that liveness keyed map (the #lzfamilysync derived-aggregate contract), and an OS
process-exit event is just the highest-stamp write to alive[pid]. This keeps liveness on the same
convergent, idempotent, frontier-resumable substrate as every other replicated cell. Normative
semantics: protocol.md § Liveness cells.
Conformance summary
An implementation conforms to the cell model when:
- Every cell is classified single-writer or multi-write, statically, by representation.
- Multi-write cells carry a
merge: <mechanism>attribute; multi-write is not modeled as a hardcodedcrdtcell kind. crdtis implemented as the first mechanism; unimplemented reserved mechanisms are rejected explicitly, never silently aliased.- Merge is an ingress step on root cells only; derived cells and effects are never merged.
- Every merge is scoped to a single cell (cell = merge unit); no cross-cell content movement.
- Downstream reactive propagation is the same direct mechanism regardless of cell kind.
- The attach/detach authority governs writer liveness only, never a cell’s kind or mechanism.
- A keyed cell collection (
ReactiveMap— theSourceMap/ComputedMapspecializations) is implemented — entries are ordinary cells, a dedicated membership cell tracks the key set, and the value / set-membership / order reactivity-independence, stable-handle, and atomic-move invariants below hold. Collections are required of every binding, not optional. - An ordered keyed tree (
SourceTree) is implemented, inheriting the per-cell merge and atomic-move guarantees node-by-node (required of every binding). - Keyed reconciliation emits the move-minimized
{insert, remove, move, update}op set (LIS over prior indices preserved), and a stable entry is not invalidated by a sibling reorder (required of every binding). - A reactive queue (
QueueCell) is implemented — a FIFO collection whose reactive shell invalidates by reader kind (head / length / empty / full / closed), backed by a pluggableQueueStoragebackend. The shell / storage split, closure observable contract, bounded-queue backpressure, and ordering guarantees below hold (required of every binding). - Materialization is eager by default — a
ComputedMap’s derived entries are pre-minted over the keyset; lazy materialization (get_or_insert_withmint-on-access) is opt-in, keyed, and observationally transparent — identical read values, allocation deferred only (see Materialization). It is a behavior, not a mode flag. Lazy evaluation (bounded-viewport recompute) is provided either way and is never conflated with lazy materialization.
Materialization (a caller-provided recipe)
Cell kind (above) fixes how a cell converges. Materialization is an orthogonal axis: it fixes when a derived cell’s backing node is allocated — not what it computes, not how it converges, not how it merges. It trades memory and first-touch latency against cold full-scan cost, and it MUST NOT be observable through the value of any cell.
Why a behavior, not a mode. Materialization was first pinned as a bespoke
ReactiveFamilytype carrying an eager/lazy mode — a reaction to one binding (lazily-zig) implementing lazy materialization in the spreadsheet benchmark and thereby diverging from the others. Standardizing a type with a mode invites re-divergence: each binding builds it slightly differently, and most never need it. What must agree is the observable behavior the benchmark measures — transparency (a lazy read equals an eager read) and deferral (an unread lazy entry costs nothing) — not any type or flag. So materialization is normative as a behavior of the keyed primitive: it is simply whatComputedMapdoes —get_or_insert_withmints a derived slot on first access (lazy); a pre-mint loop over the keyset is eager. There is no materialization mode and no family type — the family types (ReactiveFamily/CellFamily) are removed;ComputedMap(aReactiveMapspecialization) is the vehicle.
The materialization recipe
Materialization is caller-provided: a keyed collection (a
SourceMap or any keyed address space) plus a per-key factory
whose return type is the materialization choice. Nothing new is required beyond the
cell/slot/signal primitives a binding already has:
- Eager entry — the factory yields an input cell or an eager
signal(a memo-slot + puller effect): the node is allocated/pulled up front; a read is a direct node access. - Lazy entry — the factory yields a lazy
slot: the node is allocated on its first observe, addressed by key. A never-observed lazy entry is never allocated.
So the caller provides materialization through two levers it already owns — whether it
observes a key (unread lazy entries stay unallocated) and what the factory returns (slot ⇒
lazy, signal ⇒ eager) — with no mode flag and no per-read toggle. This is the same
“lazy by default, eager when asked (via signal)” model lazily already uses at the single-cell
level, applied per key. Entry kind is the pinned axis:
- Cell entries (
H = Source) are input nodes — always materialized; an input has no derivation to defer. Minting an input on firstgetis a collection concern, not materialization. - Slot entries (
H = Computed) are derived — the ones deferral governs: an eager factory allocates them up front, a lazy factory defers each to first observe.
Entry kind is orthogonal to the materialization choice (proved in lazily-formal’s
Materialization module as cell_entries_materialized_in_every_mode /
slot_entries_deferred_under_lazy): lazy defers only slot entries, never cell entries.
Normative rules on the recipe:
- Eager is the default. Absent an explicit lazy opt-in, derived entries are eager — a read is a direct node access and a full recompute pays only compute. A binding MUST make eager the default.
- Lazy is an explicit opt-in overlay on the eager core, addressed by key, never
the default and never a per-read toggle on an eager handle. The first observe of key
kconstructs the same node the eager build would have, then caches it — a keyed overlay, not a second graph engine. A binding that offers lazy MUST expose it as an explicit opt-in (e.g. a keyed factory / keyed-context constructor).
Observational transparency (normative)
For every node and every read, the observed value MUST be identical under either mode. Materialization mode is not observable on the value axis — it changes allocation timing and memory, never results:
observe(build(eager, spec), id) = observe(build(lazy, spec), id) = spec.val(id) ∀ id
This is proved in lazily-formal’s Materialization module
(observe_canonical, eager_lazy_observationally_equivalent). An implementation MUST
preserve these consequences:
- Same values. A lazy read returns the value an eager read would (
observe_canonical). - No churn from allocation. Materializing one node MUST NOT change any other node’s
observed value (
materialize_preserves_observe). - Deferral, not de-allocation. Lazy materialization only grows the materialized set;
a materialized node is never silently dropped, and the lazy set is a subset of the eager
set (
materialize_present_monotone,lazy_present_subset_eager). - Reactivity is orthogonal. Lazy evaluation — leaving off-viewport derived cells dirty and never recomputing them (the microsecond bounded-viewport read) — is required of both modes and is independent of materialization. Eager materialization still evaluates lazily; lazy materialization additionally defers allocation. An implementation MUST NOT conflate the two.
Execution-context flavors (thread-safe / async)
ComputedMap runs against a context, and the context is a third axis orthogonal to both entry
kind and materialization: it fixes where and how the graph executes, not what it computes.
The materialization laws hold over each context a binding provides — the ReactiveMap line has
one flavor per context, and each carries the context-specific law below:
- Single-threaded (
ComputedMap, over the baseContext) — the reference semantics above. - Thread-safe (
ThreadSafeComputedMap, over a lock-backed context) — aSend + Syncmap that can live in a cross-thread owner (e.g. a hub behind a global mutex, where anRc-based map cannot go). It carries the same materialization laws, plus materialization confluence: the present set and every observed value are independent of the order in which keys are materialized. This is what makes lock-serialized concurrent materialization safe — any order the lock admits yields the same observable map. Proved inlazily-formal’sMaterializationmodule (materialize_present_comm/materialize_observe_comm). - Async (
AsyncComputedMap, over an async context) — derived (slot) entries resolve asynchronously, so a non-blocking read returns an optional value (Nonewhile pending,Some(v)once resolved). Observational transparency weakens to eventual transparency: once a node resolves, its observed value is the canonical value — identical to what the synchronousComputedMapobserves. Input cells are resolved at build. Proved inlazily-formal’sAsyncMaterializationmodule (eventual_transparency,async_resolved_matches_sync; a pending read is never a stale value,observe_pending_is_none).
A binding SHOULD keep the per-key factory uniform across flavors, so the flavors
differ only in execution context — a derived async slot wraps that factory in a
ready computation. The factory SHOULD receive the entry’s own tracking view
(Fn(&Compute, &K) -> V, not Fn(&K) -> V): without it a derived entry cannot
read another reactive and have that read register a dependency edge, which
silently reduces ComputedMap to a cache.
A flavor MUST preserve the entry-kind and materialization laws above, MUST expose the Core surface defined under “Core surface vs. binding extensions” below, and adds the context-specific guarantee (confluence for thread-safe, eventual transparency for async). Ordering and atomic move are not exempt: they are Core, and they bind every flavor the binding advertises. A binding that advertises a thread-safe or async map but ships Core only on its single-threaded map is non-conforming on that advertised flavor. A context declared absent is staged out of that peer group; a runner-local lock, dictionary, or synchronous stand-in MUST NOT be used to make the missing flavor appear covered.
When to opt into lazy
Lazy pays off only for sparsely-touched large keyed address spaces — e.g. a
10,000,000-cell spreadsheet where a session reads ~1% of the derived cells: it lowers peak
memory and makes “open” cost O(inputs) rather than O(derived cells). It costs a
keyed-cache lookup per read instead of a handle dereference, and a cold full scan pays
allocation and compute together (eager_materializes_all vs lazy_defers_slots).
Handle-based graphs that read most of what they build SHOULD stay eager. The choice is a
per-context construction decision, not a per-cell or per-read one.
Keyed cell collections
A keyed cell collection is a composition of cells, not a new cell kind. It maps keys K
to per-entry reactive nodes and adds a dedicated membership cell tracking the set of keys.
There is one keyed primitive, generic over the entry’s handle kind:
ReactiveMap<K, V, H>— a mutable reactive keyed dict: reactive membership + order,get_or_insert_with(mint-on-access),remove,move.His the entry handle kind. Its two specializations are the concrete types a binding exposes:SourceMap<K, V>=ReactiveMap<K, V, Source>— input-cell entries. Addsset(key, value)(an input is settable). Minting is eager-by-value.ComputedMap<K, V>=ReactiveMap<K, V, Computed>— derived-slot entries.get_or_insert_with(key, factory)mints a slot on first access (lazy materialization); a slot’s value is derived, soComputedMaphas noset. Eager materialization is a pre-mint loop over the keyset; lazy is mint-on-access — there is no eager/lazy mode flag.
Deprecated spellings.
SourceMapandComputedMapwere previouslyCellMapandSlotMap, withThreadSafeCellMap/ThreadSafeSlotMapandAsyncCellMap/AsyncSlotMapas their per-context variants. The rename finishes the v2 kernel migration: the node kinds becameSourceandComputed, and the map names now say which kind they hold instead of naming a vocabulary the kernel no longer uses. A binding SHOULD keep the old names as deprecated aliases of the new ones rather than removing them, so a caller that has not migrated still compiles. Runners MUST accept the oldmodelspellings in a fixture ("CellMap","SlotMap") alongside the new ones — the same dual-accept the corpus already uses for thesignal/eageranddispose_signal/lazyop names. The fixture FILE names keep their historical spelling; a file name is not a type name, and renaming them would invalidate every binding’s replay ledger at once for no semantic gain.
set(key, value) is therefore cell-only (lives on the SourceMap specialization); the shared
surface — get_or_insert_with / remove / move / membership / order — lives on the generic
ReactiveMap. There are no family types: the “keyed materialized family” is ComputedMap + the
mint recipe, and the “auto-mint keyed default” is get_or_insert_with — neither needs a separate
type (see § Materialization).
Required. The keyed cell collections layer is normative for every lazily binding — it is not an optional lazily-rs extension. A conforming binding MUST implement
ReactiveMap(at least itsSourceMapspecialization;ComputedMapwhere the binding supports derived slots), the ordered keyed tree (SourceTree), and keyed reconciliation, and MUST validate against the canonical fixtures inconformance/collections/. The single-writer / multi-write classification,merge:mechanism, and ingress rules below are exactly those defined above — the collection adds no new merge unit.
It conforms to the cell model when:
- Each entry is an ordinary cell — its single-writer / multi-write classification,
merge:mechanism, and ingress rules are exactly those above. The collection adds no new merge unit; cell = merge unit still holds per entry. - Value, set-membership, and order reactivity are independent: writing one entry’s
value MUST NOT invalidate membership or order readers; adding/removing a key MUST NOT
invalidate readers of unrelated entry values; and a pure reorder (atomic move) MUST
NOT invalidate set-membership readers (
len/contains) — only order readers (keys). - A key resolves to a stable handle for the key’s lifetime; membership and order changes are signalled by their dedicated cells, never by mutating sibling entries.
- Atomic ordered move (
move_to/move_before/move_after): reordering a key MUST keep the entry’s same cell handle, dependents, and lineage (not remove + re-mint) and bump only the order signal once.
Exact-key dependency availability (#lzdependencyavailability)
A non-minting observe(key) correctly reports that an entry is absent, but it cannot
register an edge to a node that does not exist. A computed that must react when an
exact key is published later therefore uses a dependency reactive:
DependencyReactive<T> = Unavailable | Available(T)
DependencyMap<K, T> is the SourceMap<K, DependencyReactive<T>> specialization
whose lifecycle is explicit:
observe_dependency(key)materializes one stable input cell for the exact key, seeded withUnavailable, and observes it. This materializes only the dependency handle; it does not fabricateTor run an expensive value factory.publish(key, value)is an ordinary source transition toAvailable(value). If observation happened first, publication invalidates exactly that key’s consumers. Publishing an unrelated key does not invalidate them.unpublish(key)transitions the same source back toUnavailable; it does not remove or re-mint the handle. A laterpublishreuses the same identity.- Multiple first observers, including concurrent observers on thread-safe and async flavors, converge on one stored handle without orphan nodes.
- The dependency handle lives until its owning graph scope is torn down. Logical unavailability is a value transition, not disposal.
This API is deliberately separate from observe(key): the existing Core observation
remains pure and non-minting. Implementations MUST NOT approximate exact-key
availability by observing the whole membership cell, polling/retrying, sleeping, or
consulting durable storage. Those mechanisms either invalidate unrelated readers or
move live transition authority outside the graph.
Core surface vs. binding extensions
The clauses above are laws. This section says which methods a binding must expose for those laws to be observable, and — just as importantly — which methods are not spec surface at all.
The distinction exists because the family drifted without it. A survey of all
nine bindings’ single-threaded maps found method counts from 11 to 32, and the
drift splits cleanly in two: methods present in 8 or 9 bindings are laws with a
fixture behind them and one binding that never implemented them, while methods
present in 7 or fewer are conveniences with no fixture and no law —
is_empty is len() == 0, len_untracked is a tracking-discipline escape
hatch, reconcile belongs to a higher layer, insert is set under another
name. Without this split, “kt is missing seven required methods” and “cs also
ships a reconcile helper” are indistinguishable: both read as divergence, and
the coverage matrix marks both green.
Core — REQUIRED
A conforming binding MUST expose all of the following, on every flavor it ships (single-threaded, thread-safe, async), spelled in the binding’s own naming convention:
| group | methods |
|---|---|
| construction | construct bound to a context |
| materialization | get_or_insert_with (lazy mint-on-access); materialize_all (eager pre-mint) |
| kind specialization | set (SourceMap only); materialize_all (ComputedMap only) |
| entry read | observe(key) -> Maybe<V> — pure and non-minting; handle(key) -> Maybe<H> |
| reactive reads | keys (order-tracked), len and contains_key (membership-tracked) |
| materialization plane | present_keys, present_count, is_present — non-reactive |
| order | position, move_to, move_before, move_after |
| membership | remove |
| introspection | entry_kind |
Two Core entries are load-bearing in ways that are easy to miss:
observeMUST NOT mint. A read that takes a factory and allocates is a different operation with a different contract; it cannot express “is this key absent” and it cannot be called from a reader without a write side effect.handleis not optional. Clause 3’s stable handle and clause 4’s “same cell handle” are unassertable without a way to observe entry identity. A map whose entries are plain cached values rather than reactive nodes cannot satisfy this, and MUST be recorded as divergent rather than marked green.
The kind specializations bind per entry kind, not per flavor: set is
SourceMap-only on all three flavors, materialize_all is ComputedMap-only
on all three. Ordering, by contrast, binds every flavor — it touches no entry
handle and awaits nothing, so it is neither thread-coloured nor async-coloured.
Extended — OPTIONAL, non-normative
The following ship in some bindings and are explicitly not spec surface. A binding MAY offer them, MAY name them differently, and MAY omit them entirely; their absence is not a conformance gap and MUST NOT be scored in the coverage matrix.
entry / entry_with (eager value-minting convenience), is_empty,
len_untracked, get_or_insert_handle, reconcile, insert, Try*-prefixed
naming variants, and any binding-local accessors for the membership/order
signals themselves.
Ordered keyed tree
An ordered keyed tree (SourceTree) is a further composition: each node is
(stable id, value cell, ordered keyed child collection). It conforms when per-node value
reactivity holds (editing a node invalidates only that node’s readers), per-level
membership/order reactivity holds (a sibling subtree or descendant change MUST NOT
invalidate an unrelated level’s child readers), and child reorder inherits the atomic-move
guarantee. The tree is still a composition of cells — not a new cell kind — so per-cell
merge applies node-by-node.
This is the runtime substrate for stable keyed/wire addressing of collection entries
(see the protocol spec’s node-key addressing) and for keyed reconciliation of document
trees (minimal {insert, remove, move, update} ops per item → per-cell CRDT merge).
Keyed reconciliation
Reconciling a level diffs two keyed sequences by stable key, not position, emitting the
minimal {insert, remove, move, update} op set. It conforms when reordering is
move-minimized (keys already in relative order — the longest-increasing-subsequence over
their prior indices — MUST NOT move; only the remainder emit move), and when applied to
the reactive collection a stable entry (unchanged value, in the LIS) MUST NOT have its
value cell invalidated by a sibling reorder. Applying this minimal op set per-cell is the
enabling step for per-cell CRDT merge of a document tree — it replaces whole-subtree
replacement with proportional-to-the-diff work.
Memoized semantic tree
The syntactic tree holds input cells; a semantic tree (unresolved prompts, drainable
heads, summaries) is a layer of memoized computeds derived from it — one memo slot per
node folding (node value, child derived values). It conforms when the derivation is
incremental and glitch-free: editing one node recomputes only its ancestor chain (a
sibling subtree’s derived value stays cached), and a node edit that does not change the
folded result MUST NOT re-run a downstream consumer (memo equality guard). Semantics are
derived, not materialized eagerly — cost is proportional to the diff, not the document.
Manufactured identity for text
Markdown has no inherent node ids, so reconciliation keys are manufactured from text in
three layers: in-band anchors (exact, survive a body rewrite), content-derived hashes
of normalized text (survive reflow/reorder, change on edit), and alignment by similarity
(word-LCS ratio) to distinguish an edit (key inherited from the matched predecessor → an
update) from a genuine insert. A true rewrite legitimately reads as insert+remove. This
is why the controlled skeleton uses in-band markers: stable identity is the linchpin that
keeps keyed reconciliation from degrading to whole-document replacement over unstable text.
Free-text CRDT + re-parse
For anchorless prose under concurrent edits the merge unit drops to characters: a Fugue/RGA-style character CRDT (each char an element with a unique id + left origin; deletes tombstoned) whose order is a pure function of the element set, so merge is commutative, associative, idempotent and concurrent same-point inserts converge with both preserved. The structural tree is then a projection of the merged text (re-parse → manufactured-identity keys → reconcile), not the merge unit. Honest floor: a true rewrite is a replace — no character identity survives it. The anchored layer keeps per-node lineage; the free-text layer’s guarantee is “merge the text, re-derive the tree.”
Delta sync (#lztextsync)
Whole-replica merge requires transporting the entire element set; a conformant binding also
exposes delta synchronization so replicas converge by exchanging only what a partner
lacks. Three operations, over the same element set:
version_vector()→{peer → counter}— the greatest [OpId] counter this replica holds per originating peer, taken over both insert ids and tombstone (delete) ids. It is the compact frontier a replica publishes; an op(c, p)is unknown to a partner iffc > their_vv[p](absent peer = 0).delta_since(their_vv)→[TextOp]— the ops this replica holds thattheir_vvhas not observed: elements whose insert id is newer, plus elements whose tombstone id is newer (a fresh deletion of an already-shared element). EachTextOpis the transport form of one element —{ id, ch, origin, deleted }. A whole-state snapshot isdelta_since(∅).apply_delta(ops)— applies a delta op list with the same algebra asmerge: a new id adds its element (preserving that id); an incoming tombstone is merged sticky-minimally (concurrent deletes keep the smaller delete id); the local Lamport counter advances past every observed id. It is therefore commutative, associative, and idempotent, and re-applying a delta is a no-op.
Identity preservation is the load-bearing property: rebuilding a replica by apply_delta-ing a
snapshot onto a fresh buffer keeps every character’s OpId, so a later concurrent edit merges
without duplication — unlike re-parsing the text, which would mint fresh ids and double content
on the next merge. This is what lets a canonical replica fork per-member replicas from an
encoded snapshot and keep them converged by bidirectional delta_since/apply_delta.
The three operations and their convergence/idempotence/identity invariants are pinned by the
compute fixture conformance/collections/textcrdt_delta_sync.json
(#lztextsync): version-vector shape, bidirectional exchange, whole-snapshot fork identity, and
no-op re-apply.
Move-aware sequence order
Sibling order under concurrency is a separate composition above per-cell value merge: a move-aware sequence CRDT (fractional-index positions tiebroken by peer). It conforms when a move is a single LWW reassignment of an element’s position — not delete + reinsert — so two concurrent moves of the same element converge to the later one without duplication, and a concurrent move + value-edit of one element both apply (position and value are independent registers). Removal is an LWW tombstone. This is the order layer beneath keyed reconciliation; it lives only at the multi-writer boundary, leaving the single-producer Snapshot/Delta mirror unchanged.
Forking a replica (#lzzigforkhlcpeer). A fork copies the element set under a new peer
identity, and the two halves of its clock are governed separately. The forked replica MUST
inherit the source’s causal position — the HLC’s last observed wall time and logical counter —
because it has already observed everything the source holds; a fork that restarts its clock at
zero mints stamps behind state it already carries the moment its next local op supplies a
backwards-skewed now, and since every register adopts only on a strictly greater stamp,
that replica’s own write is silently rejected rather than applied. The forked replica MUST
NOT inherit the source’s peer id: the peer is the stamp’s final tiebreaker, so two replicas
stamping under one id can mint an identical (wall, logical, peer) triple, neither adopts the
other, and they diverge permanently. Both halves are pinned by
conformance/collections/seqcrdt_convergence.json
(fork_carries_the_clock_so_a_backwards_skewed_write_survives,
fork_stamps_with_its_own_peer_so_equal_wall_edits_converge). A same-peer copy — clone —
inherits both, because it is the same replica continuing rather than a new one.
Tombstone garbage collection
Tombstones (both the sequence-CRDT LWW flag and the character-CRDT sticky delete, which carries the delete’s own id) accumulate without bound — the standard set-CRDT memory-bloat cost. Conformant GC is causal-stability-gated: a tombstone is collectable only once every replica has observed the deletion (the version-vector frontier supplied by the distributed plane, never a single replica’s clock). The sequence layer drops a stable tombstone directly (observationally inert: order/contains already skip it, re-merge re-adopts it as a tombstone, a genuine resurrection wins by LWW). The character layer is conservative: it collects a stable deleted element only when nothing references it as a left origin, so removal never orphans a survivor; interior tombstones are reclaimed bottom-up. Bloat is bounded to the multi-writer plane — the single-producer Snapshot/Delta mirror accrues none.
Scheduling (#lzspecgcdefer). The safety contract fixes which tombstones are
collectable; it does not mandate when collection runs. A binding MAY defer GC to idle,
batch it under memory pressure, or reclaim incrementally per anti-entropy round, provided no
collectable tombstone is re-examined after reclaim and unbounded accrual is surfaced via
instrumentation. Deferral changes only memory footprint, never observable values.
Cascade depth is scheduling, not safety (#lzspecgcreferencedtombstone). “Bottom-up”
describes the ORDER reclamation is safe in, not how much of it one call must perform.
Collecting a leaf can unreference its origin, making that origin collectable in turn; a
binding MAY run a single pass and leave the newly-unreferenced tombstone for the next
call, or iterate to a fixpoint within one call. The family is split roughly evenly between
the two and both conform — the difference is memory footprint at a point in time, which the
clause above already places outside the contract. Conformance fixtures therefore assert
which tombstones survive a collection, never how many passes it took, and a runner that
pinned a cascade count would be encoding one binding’s schedule as a requirement.
What is not optional is the conservative rule itself: a stable tombstone that any
surviving element still names as its left origin MUST be kept, or removal orphans a
survivor. That half is pinned by
conformance/collections/textcrdt_convergence.json
(gc_keeps_a_tombstone_that_is_still_a_left_origin) — deleting an INTERIOR character, since
a scenario that deletes the last one has no referenced tombstone to keep and so cannot tell
a conforming collector from one that ignores origins entirely.
Reactive queues
A reactive queue (QueueCell) is a FIFO collection composed of cells — not a new cell
kind — that adds queue semantics (push to tail, pop from head) to the reactive graph. Like
the keyed collections above, it adds no new merge unit; each element’s value is an ordinary
cell subject to the same single-writer / multi-write classification.
The distinguishing property of a reactive queue is that invalidation is scoped to reader
kind, not to individual positions: a push invalidates length/empty/full readers (and the
tail signal); a pop invalidates head/length/empty/full readers. The head reader observes the
current head value — after a pop, the head reader sees the next element (or empty), not a
stale value. There is no random-access queue[N] reader; per-position reactivity is the
domain of SourceMap, not QueueCell.
QueueCell — SPSC primitive with MPSC usage rule
QueueCell is specified as a single-producer, single-consumer (SPSC) primitive: one
writer owns the tail, one reader owns the head. The producer is the natural FIFO sequencer
(push order = delivery order).
MPSC (multi-producer, single-consumer) is a usage rule on the same primitive, not a
separate type. Multiple producers push to the same tail inside a batch(); the batch
boundary serializes the pushes into a deterministic order. A conforming implementation
MUST document the MPSC usage rule and MUST NOT introduce a separate MPSCQueueCell type.
Naming discipline. The cardinality of producers/consumers is not a type parameter.
SPSCQueueCellwould implyMPSCQueueCell/SPMCQueueCell/MPMCQueueCellsiblings — but those shapes differ in semantics (invalidation model, handoff exclusivity), not cardinality. See § Future queue primitives for the genuinely distinct primitives (TopicCell,WorkQueueCell).
The queue family — two axes (semantics defines the primitive)
QueueCell, TopicCell, and WorkQueueCell are one family of reactive cursor-stream cells,
separated by two orthogonal axes. Only the first defines the primitive; the second is a usage
tier every primitive shares.
Axis 1 — consumer delivery semantics (the primitive axis). Where does each pushed element go?
| Delivery semantics | Each element goes to | Consumption | Primitive |
|---|---|---|---|
| single | the one consumer | destructive pop | QueueCell |
| competing | exactly one of N consumers | destructive, exclusive handoff | WorkQueueCell |
| broadcast | every subscriber | non-destructive cursor read | TopicCell |
Axis 2 — topology / ordering (a usage tier, not a type). Producer and consumer counts — fan-in and fan-out — never change which primitive you have; they only set the ordering guarantee and its cost. Opt into exactly the tier you need:
| Ordering tier | Guarantee | Mechanism | Cost |
|---|---|---|---|
| per-producer FIFO (default) | each producer’s substream in push order; interleave arbitrary | none — ≡ multiplexing N single-producer channels at the consumer | free |
| agreed total order | one global order across producers | a single leader sequencer | one hop, single point of failure |
| agreed total order + HA | total order surviving node death / partition | consensus (Raft/Paxos) | quorum latency |
Why not name by cardinality or by fan-in/fan-out. Both are the same mistake — naming a primitive by topology instead of by delivery semantics:
SP*/MP*mixes two axes:QueueCellalready covers SPSC and MPSC (both single-consumer; MPSC is the multi-producer usage).WorkQueueCellandTopicCellare each usable single- or multi-producer, soSPMC/MPMCcannot tell them apart.FanOutCell/FanInCellfails identically. Fan-in (many producers → one consumer) is justQueueCellused multi-producer — the ordering tier, not a distinct primitive; aFanInCellcollapses back intoQueueCell. Fan-out (one → many consumers) is ambiguous between broadcast and competing —TopicCellandWorkQueueCellboth fan out, and oneFanOutCellname cannot say whether an element goes to all or to one. And fan-in and fan-out are not mutually exclusive (a topic or work queue may be many-producer and many-consumer at once), so they are orthogonal descriptors of wiring, not primitive identities.
So fan-in and fan-out are real and useful — but as the topology axes describing how a primitive
is wired, not as names: fan-in = the multi-producer ordering tier (Axis 2); fan-out =
multi-consumer, which subdivides into broadcast (TopicCell) vs competing (WorkQueueCell) by
Axis 1. A primitive is always named by its Axis-1 delivery semantics, which carries meaning a
topology name cannot.
Naming rationale (suffix). The shared family suffix is Cell; Queue appears only where
consumption is destructive and exclusive — QueueCell and WorkQueueCell. A TopicCell is a
non-destructive broadcast log (reading removes nothing; every subscriber reads every element),
so it carries no Queue — TopicQueueCell would conflate the deliberately contrasting messaging
terms queue (consumed once) and topic (delivered to all). Parity is the Cell suffix + this
family section, not a forced Queue infix.
What multi-producer actually buys (fan-in): a single merged fan-in aggregation stream, one shared backpressure / capacity bound, and — only at the total-order tier — an agreed order. Consensus is the price of agreed total order under partition only, never of multi-producer itself; per-producer FIFO needs no sequencer (it is multiplexed single-producer channels). Full cross-replica retention (every replica holds each item until all consumers ack + compaction) is a replication/HA cost, orthogonal to producer count — a non-replicated queue retains each item once.
Distribution cost differs by primitive (Axis 1 decides it):
| Property | QueueCell | WorkQueueCell | TopicCell |
|---|---|---|---|
| Each element → | the one consumer | exactly one of N | every subscriber |
| Consumption | destructive pop | destructive, exclusive handoff | non-destructive cursor read |
| Distribution cost | leader-election HA (fence one consumer) | assignment consensus (quorum) | per-subscriber cursors, no consensus¹ |
| Slow consumer | fills queue → backpressure | routed to others; fine until all saturated | grows its retention; evict on lease expiry |
| Consumer death | queue stalls until failover | in-flight reassigned; no stall | that subscription grows; others fine |
| Ordering | total FIFO | assignment-FIFO; processing unordered | per-subscriber FIFO¹ |
| Delivery | exactly-once local / effectively-once distributed | at-least-once + idempotency = effectively-once | at-least-once per subscriber |
| Consensus? | only for HA | yes (assignment) | only for agreed-order broadcast¹ |
¹ A topic is consensus-free only at the per-producer-FIFO tier; total-order broadcast (all
subscribers agreeing on one order) is atomic broadcast ≡ consensus — the same cost as an ordered
WorkQueueCell. Quorum-intersection safety for WorkQueueCell assignment is proven
ReliableSync.majorities_intersect. Detailed semantics follow in
§ Future queue primitives.
Reactive shell vs storage backend
A QueueCell factors into two layers:
┌─────────────────────────────────────────────────────────┐
│ Reactive shell │
│ head version cell │ tail version cell │ closed cell │
│ len / is_empty / is_full / head — reactive reads │
│ invalidation scoped by reader kind │
└────────────────────────┬────────────────────────────────┘
│ QueueStorage trait
│ try_push(v) → Result<(), Full|Closed>
│ try_pop() → Result<T, Empty|Closed>
│ len() → usize
│ capacity() → Option<usize>
│ is_closed() → bool
┌───────────────────────┼───────────────────────────────┐
│ │ │
▼ ▼ ▼
VecDequeStorage RaftQueueStorage KafkaStorage
(local default) (embedded consensus; (external broker;
per distributed-queue PRD) via adapter)
The reactive shell owns the version cells and invalidation logic; it is
storage-agnostic and is what the formal model (QueueCell.lean) pins.
The reader-kind plane is pinned separately by QueueReaderKinds.lean: the
invalidation set is proven to be a function of the transition alone (the bound,
the pre-op length, emptiness, and the closed flag) and never of the elements,
which is what licenses running the identical rule on all three flavors. It also
proves the atomicity requirement above, and — as the queue-family instance of
the thread-safe confluence obligation — that invalidation is order-independent
and idempotent, because it consults membership in a set rather than a sequence.
QueueFamilyReaderKinds.lean carries the same obligation to the other two
primitives: the WorkQueueCell rule invalidates a reader kind iff that kind’s
value moved (so it neither over- nor under-invalidates) and reads only the
before/after counts; a TopicCell publish changes the observed suffix of exactly
the connected subscribers; a cursor read depends on that subscriber’s own record
alone, which is why the independence law holds structurally rather than by
discipline; and safe GC provably changes no reader’s value, which is what
licenses gc taking no context on any flavor.
The storage backend owns the actual FIFO data structure and is pluggable via the
QueueStorage trait (Rust) / concept (C++) / interface (Py/JS/etc.).
An implementation MUST split the shell from the storage. The shell MUST NOT assume a
specific storage type (VecDeque, ring buffer, broker client). A binding MAY ship multiple
backends; the default MUST be an unbounded VecDeque-backed storage.
Storage backend contract
Minimal required contract. A QueueStorage backend MUST implement exactly
try_push / try_pop / len / is_closed / close. peek and capacity are optional
capabilities with a default of “absent” (None): a backend that satisfies only the five
required methods — a raw channel, a consuming stream, a Go channel — is fully conforming. It
simply has no head reader (no peek) and no is_full reader (unbounded, capacity() → None), exactly as an unbounded backend has always had no is_full. A backend that can
cheaply inspect its head MAY expose peek to gain a reactive head; a backend that is
bounded MUST expose capacity() → Some(n). head was never in the MUST-reactive set (see
“Named observables” below), so removing peek from the required contract removes no required
reader.
Footnote —
LookaheadShim. A caller that wants aheadreader over a non-peekable backend MAY opt into a shell-level lookahead shim that prefetches (early-pops) one element into a one-slot buffer. This is SPSC-local only — early-popping is incorrect for competing-consumer or consensus backends, where an element must not be committed to one consumer before assignment. The shim is not part of the core contract.
A conforming backend MUST also satisfy:
- FIFO order:
try_popreturns elements in the order they weretry_push-ed. A backend that reorders or silently drops elements is non-conforming. - Cardinality compatibility: the backend’s native producer/consumer shape MUST be a superset of the shell’s required shape. (SPSC shell = any backend; MPSC usage requires a backend that accepts multi-writer pushes.)
- Bounded contract (optional): a bounded backend exposes
capacity() → Some(n)andtry_pushreturnsFullwhen at capacity. The overflow policy (block / drop-oldest / drop-newest / reject) is a backend property — the shell’s observable contract only distinguishesFullfromEmpty/Closed. - Position identity: invalidation is phrased over reader kind (head/len/empty/full), not over storage indices. A ring-buffer backend whose slot index wraps MUST NOT cause spurious invalidations; the shell layers its own logical reader-kind derivations above the storage.
Closure and lifecycle
Closure is an observable contract, not a mechanism:
try_popon a closed, non-empty queue returns the next element (drain continues).try_popon a closed, empty queue returnsClosed— a signal distinct fromEmpty.try_pushon a closed queue is an error, regardless of capacity.- Close is idempotent (closing an already-closed queue is a no-op) and terminal (once closed, a queue cannot be reopened).
The mechanism (a dedicated closed cell, a flag in storage, a sentinel value) is a
binding-level choice. The formal model pins closure as a monotonic flag:
Closed_then_stays_Closed.
Bounded queue and reactive backpressure
When the storage backend is bounded (capacity() → Some(n)), the reactive shell exposes
is_full as a reactive read. A consumer’s pop that transitions the queue from full to
not-full MUST invalidate is_full readers (true → false), enabling push-side effects to
react to capacity recovery without polling. This is the backpressure signal: a producer
observes is_full and backs off; a consumer’s pop invalidates the producer’s is_full
subscription and the producer resumes.
An implementation MUST expose is_full as a reactive cell when the backend is bounded.
The unbounded default (capacity() → None) has no is_full reader to invalidate.
Ordering guarantee
| Shape | Guarantee |
|---|---|
| SPSC | Total FIFO — pop order exactly matches push order. The producer is the single sequencer. |
| MPSC | Per-producer FIFO — messages from each producer arrive in that producer’s push order. Inter-producer interleaving is deterministic within a batch() but implementation-defined across batches; under distribution it converges. |
A consumer MUST NOT assume total-FIFO across multiple producers. If total order across producers is required, route all pushes through a single producer or use a consensus-backed storage backend (per the distributed-queue PRD).
Wire and snapshot shape
The QueueCell shell’s version cells have no own IPC schema — the head/tail/closed
counters are trivial and not independently serialized. A queue reconciles by two
complementary wire forms, chosen by plane:
- Snapshot plane → storage-snapshot form. Full queue state is the storage backend’s
snapshot form. The reference
VecDequeStoragebackend serializes as a JSON array (element order = FIFO order) for conformance fixtures; bindings MAY choose a more efficient binary encoding (bincode, postcard) for production. Cross-backend interop (e.g.VecDeque-backed on one peer,RaftQueueStorageon another) requires explicit storage-format agreement; the shell does not mandate a canonical storage snapshot. - Delta plane → op-log form. Incremental change is the ordered shell op-log —
QueuePush/QueuePop/QueueClose— carried in aDeltalike any otherDeltaOp(protocol.md § QueueCell op-log delta form). These are shell ops (storage-agnostic append/remove-head/close), so they need no storage-format agreement; the op-log is the form reliable-sync fuses under backpressure (a queue cannot state-supersede coalesce — order, multiplicity, and the receiver’s pop position forbid it). The op-log delta form is normative here; a distributed storage backend (per the distributed-queue PRD) remains a separate v1 non-goal.
Distribution
Distribution of a QueueCell is a storage-backend property, not a shell property. A
QueueCell is distributable iff its storage backend provides a distributed synchronization
mechanism. The shell itself is sync-mechanism-agnostic.
v1 does not specify any distributed backend. The
Native Distributed Queue PRD covers the future consensus-based
RaftQueueStorage backend (Phase 1+) and the positioning relative to external brokers
(Kafka, RabbitMQ, Redis Streams, SQS) via the QueueStorage adapter. CRDT-based
distribution is explicitly out of scope for queues — destructive pop requires agreement,
not merge (see the PRD’s § “Background: Why Consensus, Not CRDT”).
Threading, permissions, and instrumentation
-
Threading contract:
QueueCellinherits theContextthreading model. MPSC on the sameContextusesbatch(); cross-thread MPSC requires aThreadSafeContext-bound flavor; cross-process requires a distributed storage backend.This sentence was aspirational prose for the whole v1 line: it named
ThreadSafeContextas the way to reach cross-thread MPSC while no binding had a constructor that accepted one, so the only documented path to the guarantee was unimplementable. It is now normative — a thread-safe flavor is Core (see § “Core surface vs. binding extensions (queue family)”) and a binding that ships the family on a single-threaded context only is non-conforming on the other two flavors, not merely unfinished. Recording a capability as absent is fine; documenting a route to it that does not exist is not. -
Permissions: over the distributed plane, push and pop are distinct capabilities under
PeerPermissions(distributed.json). A peer MAY be granted push-only, pop-only, or both. -
Atomicity:
pushandpopare individually atomic. Multi-op transactions (e.g., “enqueue N items then close”) MUST usebatch()so a concurrent observer never sees a partial state. -
Instrumentation: a binding SHOULD expose depth / push-count / pop-count metrics via the standard instrumentation surface (parallel to
effect_queue_pushes/max_effect_queue_depth). -
Named observables:
is_emptyandlenare reactive reads dual tois_full. All three (is_empty/len/is_full) MUST be reactive when their respective conditions can change. -
Demand-driven derivation (
#lzspecdemanddriven): a reader-kind MUST be observable-consistent — agetreturns the value consistent with all preceding ops — and a binding SHOULD defer its derivation until it has a subscriber (the measured collapse is ~32× per op — 327 ns → ~10 ns — perdocs/relaycell-backpressure-analysis.md§5). A reader-kind is a derived value (aSlot), not an eagerly-written cell; an op with no subscriber to a given reader-kind SHOULD only mark it stale (O(1)) and derive it lazily on the nextget, provided the derived value is consistent with all preceding ops. This preserves the observable contract (conformance fixtures read the values and MUST stay green) while an unsubscribedQueueCellcollapses toward raw-storage cost — the reactive shell is charged only along a path an effect actually observes. A binding that eagerly derives reader-kinds remains conformant (values stay green by construction) but forgoes the write-cost win. Seedocs/relaycell-backpressure-analysis.md§5 (demand-driven reader-kinds) and §4.0 (the merge cost law).
Queue family extensions
QueueCell covers SPSC and MPSC. TopicCell is the broadcast member of the family;
WorkQueueCell remains the competing-consumer extension. They differ in
invalidation model and handoff semantics, not merely producer/consumer cardinality:
TopicCell (broadcast)
A broadcast topic: every subscriber receives every pushed element. Invalidation is “all subscribers,” not “head reader.” Each subscriber holds its own cursor; the topic retains an element until all durable cursors pass it (or a TTL expires). Reading is non-destructive — a subscriber’s advance never removes the element for others.
Relationship to QueueCell: not a multi-consumer queue. A QueueCell consumer destructively
pops; a TopicCell subscriber reads by cursor and removes nothing. Different invalidation models in
kind.
Normative state and operations
A TopicCell<T> state is the tuple (base_offset, elements, subscriptions):
base_offset: u64is the absolute offset ofelements[0];end_offsetisbase_offset + elements.len.elements: [T]is the retained append log in per-producer FIFO order.subscriptions: Map<SubscriberId, Subscription>is keyed by stable subscriber identity. Each record contains an absolutecursor(the next offset to read),durability(durableorephemeral), andconnected. Every stored cursor MUST remain inbase_offset..=end_offset. An ephemeral record MUST be connected: disconnecting it removes the record instead of persisting an offline ephemeral cursor.
The following operations and observables are required:
subscribe(id, durable)creates a new cursor atend_offset. Reconnecting an existing durableidMUST reuse its persisted cursor; it MUST NOT silently jump to the tail. Disconnecting an ephemeral subscription removes its record, so a later subscription with that id starts at the then-current tail.publish(value)appends exactly one element and leaves every cursor unchanged. Every connected subscriber whose cursor is now behindend_offsetis invalidated independently. An offline durable subscriber is not scheduled, but the element remains readable after it reconnects.read(id)observes the element at that subscriber’s cursor without changing the log or any cursor. Reads from an unknown or disconnected subscription are unavailable.advance(id)moves only a connected subscription’s cursor by one; it is a no-op for unknown/disconnected subscriptions and atend_offset. Thereforebase_offset ≤ cursor ≤ end_offsetis preserved by every non-TTL operation, and an offline durable cursor is frozen until reconnect.- The retention frontier is the minimum cursor across durable subscriptions, or
end_offsetwhen there are none. Safe GC removes only offsets below that frontier, advancesbase_offset, and leaves absolute subscriber cursors unchanged. It MUST preserve every durable subscriber’s future read stream. Ephemeral subscriptions never hold the frontier. - GC is observational maintenance and invalidates no subscriber when it stays below the frontier. TTL-forced truncation beyond a durable cursor is a separate gap/resync policy and is not part of the v1 semantic core.
The canonical replay fixtures are
topiccell_broadcast_cursor_isolation.json,
topiccell_durable_replay_gc.json, and
topiccell_ephemeral_lifecycle.json,
with connected-session boundary behavior pinned by
topiccell_offline_tail_bounds.json.
The executable universal reference is lazily-formal/LazilyFormal/TopicCell.lean.
Distribution is cheap — no assignment consensus. Every subscriber gets every element, so there is
no “who gets this” decision to arbitrate. A distributed topic is N independent per-subscriber
cursor-queues — the per-peer DurableOutbox fan-out already pinned for reliable sync.
At-least-once fan-out + idempotent apply per subscriber = effectively-once, no quorum. Caveat: this
holds only at the per-producer-FIFO tier; total-order broadcast (all subscribers agree on one
order) is atomic broadcast ≡ consensus.
Durable vs ephemeral subscriptions. A durable subscription persists its cursor and replays elements missed while offline; an ephemeral subscription sees only elements published while connected (fire-and-forget). Durable subscriptions drive retention.
Backpressure is per-subscriber, and the answer is the state-vs-event split. Each subscriber’s
delivery buffer is itself a bounded QueueCell, so a slow subscriber’s is_full fires locally;
what happens then is a per-subscription policy, and the right one depends on message semantics —
the same dichotomy as outbox coalescing:
- State topic (broadcasting a value) — old elements are worthless once superseded, so a lagging subscriber conflates to latest (drop intermediates, keep the newest): the LWW/last-value coalesce applied to a subscriber. Memory-bounded and effect-lossless — the laggard simply gets current state, and the producer feels nothing.
- Event/log topic (each element a distinct event) — no conflation is possible (order and
multiplicity are meaning), so a lagging subscriber resolves overflow one of three ways:
- Drop (lossy, isolated) —
drop-oldest/drop-newestfor that subscription; fast subscribers and the producer are unaffected. This is the broadcast default — coupling defeats fan-out. - Couple + backpressure (lossless, slowest-paces) — the subscriber withholds ack, retention grows, and the producer throttles to the slowest durable subscriber. Opt-in, for “all-must-receive-losslessly” topics; the operator accepts one slow subscriber pacing the whole topic.
- Evict (bounded) — on sustained lag past a liveness lease, drop the whole subscription (§ Partition & eviction); it full-resyncs (durable) or resumes from now (ephemeral) on return.
- Drop (lossy, isolated) —
So a TopicCell producer feels backpressure only if a subscription opts into coupling; by default
a slow subscriber conflates (state) or drops/evicts (event), and failure is isolated — a dead
subscriber grows only its retention, never stalling other subscribers or the producer. This is the
opposite of QueueCell, where the single consumer is the backpressure path.
Consumer groups = a topic of work queues. The “Kafka consumer group” shape is a composition,
not a fourth primitive: WorkQueueCell semantics within a group (competing) and TopicCell
semantics across groups (each group gets the full stream). Model it as a TopicCell whose
subscribers are WorkQueueCells.
Status: the local semantic core, conformance fixtures, and Lean proofs ship in v0.31.0. The consensus-backed distributed storage integration remains in the distributed-queue PRD Phase 3.
WorkQueueCell (competing consumers)
A work queue: N consumers compete for elements from a shared FIFO; each element is delivered to exactly one consumer (exclusive handoff).
Why pure CRDT cannot do this. A queue pop is not idempotent-commutative: two consumers concurrently popping the same head both survive a CRDT merge → duplicate delivery, and there is no “un-pop.” Exclusive handoff therefore needs a single serialization point for the assignment decision — a designated leader assigning each element to one consumer, or a consensus-committed assignment log. This is the queue’s CP nature made concrete.
Safety via quorum intersection. With consensus-committed assignment, “element X → consumer W” is
a majority-committed log entry. Because any two majorities of an n-voter set intersect in ≥1
voter (ReliableSync.majorities_intersect / majorities_overcount), two conflicting assignments of
the same element can never both commit — no double-delivery, ever — and a minority (no quorum) cannot
commit an assignment, so it blocks rather than risking a duplicate.
Three populations — do not conflate.
- Workers/consumers — any count; clients of the queue, not voters. Scale freely.
- Replication peers (per-peer outbox) — any count; transport, not voting.
- Voting replicas (order + commit assignments) — want an odd count:
2f+1replicas tolerateffailures; odd maximizes tolerance-per-node and keeps quorum unambiguous. Make an even data set odd with a witness/arbiter (votes, holds no data).
Partition behavior. Only a partition holding a majority makes progress; the rest block (safe, no double-pop). A 2–2 split of 4 voters gives neither side a majority → both stall (the even-group hazard, fixed by a witness). A 2–2–1 split of 5 voters leaves no side with 3 → all block until partitions heal enough for some side to reach a majority. Odd counts protect two-way splits; they never guarantee a majority under multi-way fragmentation. Halting is correct — safety over liveness.
Delivery & lifecycle (#lzworkqueue):
- Assignment IDs + ack/nack. Each handoff carries a delivery ID; the worker acks (done, remove) or nacks (requeue). Exactly-once commit is the consensus assignment; exactly-once effect needs an idempotent worker or a causal receipt on completion; delivery is at-least-once (redelivery on failure) — so effectively-once = at-least-once + idempotency.
- Visibility-timeout / lease. An assigned-but-unacked element is leased with a TTL; on expiry it reassigns (worker presumed dead) — the per-item analog of the liveness lease.
- Dead-letter queue + poison detection. An element exceeding a max-redelivery count (repeatedly crashing its worker) routes to a DLQ instead of redelivering forever.
- Fairness / dedup. Assignment policy (round-robin, weighted, pull-based) + producer/consumer dedup keys are policy extensions; the portable core uses pull-based FIFO assignment.
Portable local-authority contract. Every binding ships the same in-process
WorkQueueCell state machine. The owning instance is the serialization point; hosts MUST
serialize calls using the binding’s ordinary context/threading boundary. A local instance does
not claim to be a distributed consensus implementation. A cross-process or HA backend MUST put
the claim decision behind a leader or consensus-committed assignment log while preserving these
operations and outcomes:
push(value) -> item_idappends a fresh, monotonically increasing item to the pending FIFO.claim(worker, now) -> delivery?removes the oldest pending item and creates a fresh, monotonically increasingdelivery_id. The returned lease contains the stableitem_id, value, worker, 1-based attempt, anddeadline = now + visibility_timeout. Empty claim is a no-op.ack(worker, delivery_id) -> boolremoves only that worker’s matching in-flight delivery. Unknown, stale, or wrong-worker acknowledgements are no-ops. Re-acking is therefore idempotent.nack(worker, delivery_id) -> boolvalidates ownership, then either appends the item to the pending tail or moves it to the dead-letter list when the configuredmax_deliverieshas been reached.reap_expired(now) -> countexpires leases whosedeadline < now(the deadline itself remains live), in ascending delivery-ID order. Each expired item is requeued at the pending tail or dead-lettered under the same attempt limit. A redelivery receives a new delivery ID while keeping its item ID and value.
visibility_timeout is a positive logical-clock duration and max_deliveries >= 1. Requeue-at-tail
means retries do not block newer work. The portable reactive reader kinds are pending_len,
is_empty (pending only), in_flight_len, and dead_letter_len; a mutation invalidates only readers
whose values may have changed, and these reader-kinds SHOULD be demand-driven (#lzspecdemanddriven,
see § “Demand-driven derivation” above). The canonical fixtures are
conformance/collections/workqueue_competing_delivery.json and
workqueue_lease_deadletter.json.
Assignment-FIFO ≠ processing-FIFO. Competing consumers trade ordering for parallelism: even if
elements are assigned FIFO, N workers process concurrently, so completion order is unordered. A
slow worker holding element 1 does not block element 2 (it routes to another worker) — the point
of the primitive — but a consumer MUST NOT assume total processing order. For total order, use one
consumer (QueueCell) or route through a single worker.
Pull beats push for balancing. A pull model (workers request when ready) is naturally load-balancing and backpressure-friendly — a fast worker pulls more, a saturated worker stops pulling. A push model needs the assigner to track worker capacity. Pull-based competing consumers are the simpler, self-throttling default.
Backpressure & failure — most resilient of the three. A single slow worker does not fill the
queue (its items route to others); the queue backpressures only when all workers are saturated and
depth grows. A worker death does not stall the queue — its in-flight (unacked) items reassign on lease
expiry; throughput degrades, delivery continues. Opposite of QueueCell, whose single consumer dying
stalls until failover.
Status: the portable local-authority state machine, reactive reader kinds, Lean safety model, and cross-language conformance are shipped. Distributed/HA assignment still requires the consensus adapter from the distributed-queue PRD Phase 2; the local shell must not be mistaken for that adapter.
Core surface vs. binding extensions (queue family)
The clauses above are laws. This section says which methods a binding must
expose on QueueCell / TopicCell / WorkQueueCell for those laws to be
observable, and which methods are not spec surface at all. It mirrors
§ “Core surface vs. binding extensions” for ReactiveMap, and exists for the
same reason: without the split, “this binding ships no ordering on two flavors”
and “that binding also ships a convenience helper” both read as divergence and
both score green.
The split is drawn from a survey of the bindings that ship the family. When it was first written that was eight, lazily-cs excepted; lazily-cs has since landed the single-threaded family and satisfies every Core entry below, so the survey is now 9 of 9 — unanimous, therefore law — and the rest are conveniences.
Core — REQUIRED
A conforming binding MUST expose all of the following on every flavor it ships (single-threaded, thread-safe, async), spelled in the binding’s own naming convention:
| primitive | group | methods |
|---|---|---|
QueueCell | construction | construct bound to a context; bounded and unbounded forms |
| mutation | try_push, try_pop, close | |
| reader kinds | head, len, is_empty, is_full, is_closed — all reactive | |
| bound | capacity | |
TopicCell | subscription | subscribe, reconnect, disconnect |
| broadcast | publish | |
| cursor read | read (one element at the cursor), read_stream (the cursor’s tail), advance | |
| retention | cursor/GC observables sufficient to assert the retention law | |
WorkQueueCell | mutation | push, claim, ack, nack, reap_expired |
| reader kinds | pending_len, in_flight_len, dead_letter_len — all reactive |
Three Core entries are load-bearing in ways that are easy to miss:
is_emptyis Core here, unlike onReactiveMap. On the map it is Extended, because it is justlen() == 0. For a queue it is a named reactive observable that § “Threading, permissions, and instrumentation” already requires — “is_emptyandlenare reactive reads dual tois_full; all three MUST be reactive when their respective conditions can change”. The reader-kind independence law is stated over that triple, so dropping one makes the law unassertable.- A reader kind MUST be reactive, and polling a counter is not reactive. A version counter the caller must poll registers no dependency edge, so no reader can be invalidated by it and the independence law cannot be observed. A binding whose reader kinds are polled counters, or which derives them by diffing a snapshot, MUST be recorded as divergent rather than marked green.
try_popMUST distinguish empty from closed. The closure lifecycle law needsClosedto be observably different fromEmpty; apopreturning a bare optional collapses them.
Ordering-style reasoning applies to the whole family: none of the Core surface
touches an entry handle and none of it awaits, so it is neither
thread-coloured nor async-coloured and binds every flavor. Confirmed
empirically — across all nine bindings the queue-family sources contain no
async fn, suspend fun, Future, Task, Promise, or await anywhere. A
flavor is not permitted to colour pop.
Which flavors each binding actually ships, and how that is proved
Auditing this table against the nine bindings found it wrong in both directions at once (#lzcoverageaudit), which is worth stating because the two failure modes hide from different reviewers:
- Understated. lazily-go and lazily-cpp ship and replay all three flavors of
all three primitives, and lazily-py / lazily-kt / lazily-dart ship and replay
the thread-safe and async
WorkQueueCelland the asyncTopicCell. All of it read absent. An understated table is the more dangerous of the two: nobody files a bug about coverage they are told they lack, so a “project this family into the bindings” task reads as unstarted work that is in fact done. - Overstated. lazily-js was marked as shipping the thread-safe and async
QueueCelland the thread-safeTopicCell. It ships none of them — its own flavor ledger records them unshipped and no such type exists insrc/.
A mark is only clearable by replay, never by the type existing. Every
binding’s suite carries a flavor ledger checked against its own sources in both
directions, so a row cannot claim a flavor with no type and a type cannot exist
unreplayed; and the replays are fed by a runtime fixture manifest, so “this
binary opened these bytes” is observed rather than grepped. lazily-zig is the
case that proves the distinction matters: it ships TopicCell and
WorkQueueCell and had tests named after the canonical fixtures whose expected
values were transcribed into the source, with the fixtures themselves listed as
known-uncovered. The types shipped, the corpus did not run, and the honest fix
was to add the replay — not to flip the cell.
Reader-kind derivation — one design, pinned
A conforming binding MUST implement reader kinds as memoized derivation with explicit invalidation: each reader kind is a derived node with no graph dependencies, and a successful op clears exactly those whose value provably changed, in one atomic frontier walk.
Three designs were in the field. This one is pinned because:
- it is what § “Demand-driven derivation” already specifies, and the only one that realises its cost win — a version-cell design charges a graph write on every op even with no subscriber, which is precisely what that clause exists to avoid;
- it is the shape the whole family converged on for
ReactiveMap, where all nine bindings now hold a graph-agnostic ordering/bookkeeping core beside per-flavor version cells minted on each flavor’s own graph. Forking the queue family onto a different reader-kind design would split the family’s own structure for no gain.
A binding MAY keep a version-cell implementation and remain conformant only if reads still register a dependency edge and the invalidation set is exact; it forgoes the demand-driven win. Polled counters and snapshot diffing are not conformant under any reading.
Per-flavor obligation
A flavor MUST preserve the reader-kind independence law, the closure lifecycle,
and the single-frontier-walk atomicity requirement — one op invalidates all
of its changed reader kinds together, so a subscriber never observes len
bumped while is_full has not yet flipped. On top of that it adds exactly its
context-specific guarantee: confluence for thread-safe, eventual
transparency for async.
Two flavor questions are settled here rather than left to each binding:
TopicCellfan-out needs nothing extra under a thread-safe context. Each subscriber owns its cursor and a cursor only advances, so concurrent subscriber reads commute and the retained-until-all-cursors-pass rule is a monotone function of the cursor set. Broadcast is confluent by construction.WorkQueueCelllease expiry stays caller-driven on every flavor.reap_expiredtakes the current time as a parameter; a flavor MUST NOT substitute an owned timer. Time as an input is what makes lease expiry deterministic and fixture-replayable, and an async flavor that grew its own clock would be unreplayable for no gain.
Extended — OPTIONAL, non-normative
The following ship in some bindings and are explicitly not spec surface. A binding MAY offer them, MAY name them differently, and MAY omit them entirely; their absence is not a conformance gap and MUST NOT be scored in the coverage matrix.
push / pop as panicking or asserting variants of try_push / try_pop,
peek, elements (whole-buffer snapshot), base_offset, snapshot /
from_snapshot, restart, gc as a caller-visible entry point when retention
is already automatic, reader_handle accessors for the reader-kind nodes
themselves, and any binding-local accessors for the version cells.
RelayCell — Algebra-Backed Backpressure
RelayCell is the algebra-typed conflating relay: a stream transform that
sits on an edge, adapts a fast ingress to a bounded/slow egress by an
algebra-typed merge, a reactive backpressure policy, and an optional
paged durable spill. This chapter is the normative surface of the phased plan
in relaycell-backpressure-analysis.md; the
merge algebra
(Phase 1) is its foundation and the
lazily-formal Lean model
(LazilyFormal.Merge, LazilyFormal.Relay) is the executable reference for the
invariants below.
Status. Phases 1–8 are complete across all eight bindings. The lazily-rs reference implementation and Lean pins validate the API shapes: Phase 1–4 (merge algebra, RelayCell core, SpillStore, Transport —
merge.rs,relay.rs,spill.rs,relay_transport.rs), Phase 5 (Inbox/Outbox role facades —relay_roles.rs), Phase 6 (the extraRate/Window/Expiry/Priority/keyed policies —relay_policy.rs), and Phase 7 (example systems as integration tests —relay_examples.rs). The Phase 6 policies are formally corollaries of the core theorems (window → flush-grouping, priority/sharding → reorder), and every binding exercises the Phase 2–6 behavior in its relay tests.
Invariant above all else (§9). Policy and transport are local mechanism; the converged egress state is independent of binding and mechanism whenever the merge
⊕is associative. Every RelayCell fixture asserts this, and the Leanrelay_convergestheorem pins it. Break it — a non-associative merge, or a non-lattice merge on a reordering transport — and portability, determinism, and the formal pin all fail together.
1. Not a new node — a composite (Phase 2)
RelayCell<T, M> is not a new Axis-1 delivery primitive and not a new
reactive node category. It decomposes into the reactive family:
- the hot head is a
SourceCell<T, M>(write =⊕under policyM); - its reactive reads (
depth/bytes/pending_keys/is_full/is_spilling/is_draining/lag) areFormulaCells — demand-driven, so an unobserved relay costsN·⊕and nothing more (the merge cost law); - an ingress Effect drives the merge from the transport.
It composes onto QueueCell / TopicCell / WorkQueueCell edges and
subsumes today’s scattered backpressure logic:
DurableOutbox.coalesce_to_snapshot is a RelayCell<_, CrdtJoin<Lww>> + a
SpillStore; per-subscriber TopicCell conflation is a per-sub RelayCell;
op-log outbox fusion is a RelayCell<_, RawFifo> + Spill (no conflate).
ingress ┌──────────── RelayCell ─────────────┐ egress
───(Transport)───────►│ hot head : accumulating merge (⊕) │───────►(Transport)──►
ops arrive fast │ BackpressurePolicy (watermarks) │ drained on egress
│ overflow: Block|Drop|Conflate|Spill │ readiness (credit)
│ SpillStore (paged durable tail) │
│ reactive reads: depth/bytes/keys │
└─────────────────────────────────────┘
2. BackpressurePolicy — reactive limits (Phase 2)
Every limit is a reactive cell, so an operator or an adaptive controller retunes it live and every dependent relay reacts:
| Field | Type | Meaning |
|---|---|---|
dimension | Cell<BoundDim> | Count | Bytes | Keys | Age — what the bound measures |
high_water | Cell<u64> | gate ingress at/above this level |
low_water | Cell<u64> | re-open ingress at/below this level |
overflow | Cell<Overflow> | Block | DropOldest | DropNewest | Conflate | Spill |
Hysteresis is required: high_water ≠ low_water so a relay riding the bound
does not flap open/closed. An adaptive controller is an ordinary formula
cell that observes depth/lag/downstream latency and drives high_water — the
control loop is reactive policy driving reactive limits, not a config struct.
3. Overflow actions and policy-flag validation (Phase 2)
The merge algebra, not the relay, decides which overflow is sound. A
RelayCell MUST reject an (overflow, transport) pair the policy’s flags forbid,
at construction:
| Overflow | Loss | Requires | Rejected when |
|---|---|---|---|
Block | lossless | — (propagates backpressure via is_full) | never |
Conflate | lossless for converged state | ⊕ associative (always) | RawFifo (order/multiplicity are meaning) |
Spill | lossless | ⊕ idempotent or dedup keys (crash-replay is at-least-once) | non-idempotent ⊕ without spill dedup |
DropOldest/DropNewest | lossy | caller opts into loss | a lossless policy (e.g. Sum, GCounter) declares Drop forbidden |
A relay MUST additionally reject reordered pages / shards for a
non-commutative policy (the reordering tax), and MUST log its overflow
action and expose dropped / conflated counters — no silent truncation.
Merge swap (can_reconfigure): M MAY be swapped only when the hot head is
empty (changing ⊕ mid-stream changes meaning); the relay exposes
can_reconfigure (true iff head empty), and re-validates overflow/transport
against the new policy’s flags at swap time.
4. SpillStore — paged durable tail (Phase 3)
SpillStore generalizes DurableOutbox: a hot page in RAM
(actively merged) plus immutable cold pages on durable store, a bounded
manifest (page_id → location, watermark, bytes), an egress cursor, and
ack-before-reclaim. Memory is O(hot page) + O(manifest) for any algebra.
Retrieval is pull-metered by egress credit; associativity lets a lagging consumer
receive a conflated catch-up. Reconstruction (cold pages in order, then hot
head) reproduces the flat fold — paged spill loses nothing (Lean
spill_lossless). Crash recovery replays the last unacked page; because a page is
one coalesced summary op at the egress, replay is a no-op for an idempotent policy
(Lean spill_replay_idempotent) — at-least-once delivery converges.
SpillPolicy (all reactive): page_size; mode = CompactOnWrite (keep-latest,
minimizes disk) | AppendCompact (LSM-style, preserves increments for
accumulating semilattices); retention = BoundedDisk | UnboundedLog |
Ttl; rehydrate = Sequential | Parallel. Parallel rehydrate requires a
commutative merge (pages merge out of order).
5. Transport seam (Phase 4)
Transport abstracts ingress/egress delivery so the mechanism is pluggable and
per-binding: InProc (direct) | CrossThread (native mpsc or shared
ThreadSafeContext) | IpcTransport | WsTransport. RelayCell is written
once against Transport; a binding drives it on a goroutine, an async task, or an
evented IO interface. The merge algebra — not the transport — guarantees
converged state (Lean transport_independent), so transports may differ across
bindings and still converge to the same egress.
A channel is a Transport, not a QueueStorage. With peek optional
(QueueCell) a native channel satisfies
the minimal storage contract directly, but its real role is cross-thread
delivery: the idiomatic form is CrossThread, where a worker owns the channel
(its mailbox), drains it, and merges into the hot-head SourceCell (the push-fed
regime — the pop message carries the value). Crossing a thread boundary requires
the thread-safe context.
6. Inbox / Outbox — directional roles (Phase 5)
RelayCell is direction-neutral. Inbox and Outbox are role facades (typed
constructors with direction-appropriate defaults), not reimplementations —
mirroring “MPSC is a usage of QueueCell, not a subtype”. They earn names
because they differ in the backpressure-propagation contract — who you can
backpressure and how:
| Role | Edge | Backpressure target | Default overflow |
|---|---|---|---|
Outbox | app → transport (send) | the local producer (directly blockable via is_full) | Conflate(state) / Spill(event) |
Inbox | transport → app (receive) | the remote peer — only via transport flow control (withhold credits/acks, TCP window) | Conflate(inbound) / Drop / Credit-meter |
A network link is Outbox → Transport → Inbox, and end-to-end backpressure is
a chain of relays: the local producer’s is_full ← Outbox fullness ←
(credits / TCP window) ← remote Inbox fullness ← remote app’s consumption. Both
ends share one RelayCell core so the signal propagates through the link as one
continuous reactive edge. Fan-out reuse: per-subscriber RelayCells wire into
TopicCell (the state-vs-event choice
becomes a MergePolicy choice); WorkQueueCell stays as-is (competing
consumers, no conflate).
7. Extra reactive policies (Phase 6)
Optional reactive stages composed onto the relay egress, each covering a row of the backpressure case matrix:
| Policy | Case | Behavior | Soundness |
|---|---|---|---|
WindowPolicy | 8 (debounce/throttle) | coalesce on a time window, not just fullness | associativity (a window is a flush group — flushGroupingIrrelevant) |
RatePolicy | 9 (token bucket) | pace egress; ingress backpressures | pacing re-chunks flushes; converged state unchanged |
ExpiryPolicy | 10 (TTL) | drop elements older than a deadline | lossy-by-age (explicit) |
PriorityStorage | 11 | egress by priority, not arrival | reordering — requires Commutative (Lean reorder_adjacent) |
| keyed sharding | 18 | N relays by key; merge across shards | commutative merge across shards |
WindowPolicy and RatePolicy only change where the relay flushes, so the
converged state is invariant (relay_converges). PriorityStorage and keyed
sharding reorder ops, so they are sound exactly when the policy is commutative.
8. Conformance
A RelayCell implementation conforms when:
- Composite, not a new node. The relay is built from a
SourceCellhot head, demand-drivenFormulaCellreads, and an ingress Effect — no new node category. - Converged-egress invariance (§9). For an associative
M, the egress state after any flush schedule equals the lossless flat fold of the delivered ops. Pinned per-binding by replayingmergecell_algebra.jsonthrough a relay and by the Leanrelay_converges/transport_independenttheorems. - Overflow validation. An
(overflow, transport)pair the policy’s flags forbid is rejected at construction;ConflateonRawFifo,Dropon a lossless policy, and reordered pages/shards on a non-commutative policy all fail fast. Lossy actionslogand incrementdropped/conflatedcounters. - Spill losslessness + idempotent replay. Reconstruction from cold pages +
hot head reproduces the flat fold; crash-replay of the last unacked page
converges (idempotent policy or spill dedup). Lean
spill_lossless/spill_replay_idempotent. - Reordering tax. A commutative policy is invariant under reordering
(priority egress, keyed sharding, out-of-order pages); a non-commutative one
preserves arrival order. Lean
reorder_adjacent. - Reactive policy.
BackpressurePolicy/SpillPolicylimits are reactive cells with hysteresis; retuning a limit live re-drives dependent relays.
Verification form. The converged-state and algebra invariants are pinned by the Lean model (
LazilyFormal.Relay) and themergecell_algebra.jsoncompute fixture; per-binding behavior (overflow validation, spill, transport) is exercised by each binding’s relay tests. Concurrency/transport interleaving is not a deterministic replay, so — like the thread-safe context — it is verified by each binding’s model checker / stress harness, not a portable fixture.
State Machine
A state machine is the flat finite-state-machine primitive — a reactive cell
holding the current state plus a pure transition function. It is the kernel a
single-region State Chart compiles down to, and the Lean
formal model in lazily-formal
(LazilyFormal/StateMachine.lean) fixes its semantics normatively.
A state machine is compute, not protocol. Like a chart, it is never
serialized as a distinct wire kind — only its current state crosses
IPC/FFI as an ordinary cell Payload (a CellSet op on the active-state cell).
Each binding implements it natively; this chapter fixes the behavior so a
machine defined in one language means the same thing in another.
The kernel
The pure transition core is:
Transition : State -> Event -> Option State
Machine = { current: State, transition: Transition }
send(m, e) = match m.transition(m.current, e) of
| Some next => { current: next, ... } // accepted
| None => m // rejected (guard)
A binding wraps the current State in a reactive cell and exposes send as the
mutator. The transition function is pure: given the current state and an event
it returns the next state (Some) or rejects the event (None, a guard).
API surface
| Method | Description |
|---|---|
new(ctx, initial, transition_fn) | Create with an initial state and a pure transition function |
send(ctx, event) -> bool | Evaluate the transition; true if accepted, false if rejected (None) |
state(ctx) -> State | Read the current state |
state_handle() -> SourceCell<State> | The underlying active-state cell, for reactive dependencies |
on_transition(ctx, old_new_callback) -> EffectHandle | Observer firing on each state change with (old, new) |
state_is(ctx, target) -> FormulaCell<bool> | Eager (driven) formula: true while in target |
Semantics
- PartialEq guard (no-op suppression): a transition to an equal state is
accepted (
sendreturnstrue) but does not invalidate dependents — the active-state cell’s equality guard suppresses the no-op update. This is the flat analogue of the chart self-transition rule. To force re-entry, clear the cell’s dependents beforesend. - Reactive integration: any formula, driven formula, or effect that reads
state_handle()automatically recomputes or reruns on a real transition. - On-enter / on-exit: model with an effect that has a cleanup closure — the
body is on-enter, the returned cleanup is on-exit (runs before the next rerun).
on_transitionprovides a single(old, new)observer instead. - Batch atomicity: a batch coalesces multiple
sendcalls — effects fire once after the batch settles. - Deterministic transition function: the transition function MUST be pure and
deterministic.
sendnever changes the transition function — a machine is fully described by its transition function and current state.
Proven invariants
The Lean kernel (lazily-formal/LazilyFormal/StateMachine.lean) proves the
properties that are easiest to blur across bindings:
- Guard rejection preserves state — a rejected event (
None) leavescurrentunchanged. - Accepted transitions advance state — an accepted
Some(next)setscurrent = next. - Self-transitions are no-ops —
Some(current)leavescurrentunchanged andsendsreturnsfalse. - Changed transitions send
true—Some(next)withnext != currentreports a real change. sendpreserves the transition function.
Relationship to state charts
A single-region chart refines this send: the chart’s active leaf is the
State, the chart event is the Event, and the chart’s walk-up + LCA +
descend-initial logic is a pure function returning Some(new_leaf) or None.
The confluence/determinism of a single-region chart is inherited from this kernel
(see single_region_refines_flat_machine in
lazily-formal/LazilyFormal/StateChart.lean). A flat StateMachine is the
degenerate chart with no nesting.
Context layers
The flat kernel is context-agnostic. A binding offers it over each reactive context layer it implements, with identical semantics:
- Single-threaded — backed by the single-threaded reactive context.
- Thread-safe — a lock-backed counterpart over the thread-safe context; the
transition function and state are
Send + Sync, the machine is a clonable handle to the same state cell, and observers fire synchronously within the invalidatingsend/batchcall preserving glitch-free pull-based ordering. - Async — backed by the async context;
sendandstatestay synchronous (cells are the synchronous input layer), while reactive observers use the async effect/signal APIs and settle on the runtime rather than withinsend.
Implementation status
The flat machine is required of all bindings that ship a reactive graph. The
thread-safe and async counterparts are required of any binding whose platform
supports them (see Wire Protocol § Concurrency layers are required);
a binding MAY omit one only when it has declared the matching thread_safe /
async capability as none. In every case the single-threaded machine and its
PartialEq no-op suppression MUST be present wherever a chart is supported (a
chart depends on this kernel). lazily-rs, lazily-zig, and lazily-py implement
it; the Lean model is the executable reference.
State Charts
A state chart is a Harel/SCXML hierarchical state machine. This chapter
specifies the full cross-language subset all lazily bindings implement:
compound (nested) states, orthogonal (parallel / AND) regions, history
states (shallow and deep), entry/exit/run actions, internal/external
transitions, guards, and extended state (context). It also fixes the
declarative chart form the conformance fixtures use and how a chart relates to
the reactive Cell Model and the flat FSM kernel in the Lean
Formal Model.
A state chart is compute, not protocol. It is never serialized over the wire
as a distinct kind — only its converged active configuration crosses
IPC/FFI as an ordinary cell Payload. Each binding implements the chart
natively; this chapter fixes the behavior so a chart defined in one language
means the same thing in another (validated by the conformance fixtures).
The declarative chart form is normatively defined by
schemas/statechart.json (JSON Schema, Draft
2020-12). The prose below fixes its semantics.
Why a chart reduces to existing machinery
The Lean kernel (formal/lean/LazilyFormal/StateMachine.lean) defines a pure
transition State → Event → Option State. A single-region chart compiles to
exactly this kernel: the chart’s active leaf is the State, the event
is the Event, and the chart’s transition logic (walk-up + LCA +
descend-initial) is a pure function returning Some(new_leaf) or None.
Orthogonal regions generalize State from a single leaf to an active
configuration (a set of leaves, one per region). The transition is still a
pure Configuration → Event → Option Configuration: enabled transitions,
conflict resolution, and exit/enter-set computation are all deterministic
functions of the chart definition and the current configuration. Because of
this, a chart cell is single-writer by the Cell Model
rules — it is either a direct cell (reactive bindings) or a derived cell keyed
off event cells. It is therefore never multi-write: replicas converge on
the event stream, and each replica’s chart recomputes deterministically. No
merge: mechanism is defined or needed for charts.
Reactive vs. projection bindings
| Binding | Chart backing | Why |
|---|---|---|
| lazily-rs, lazily-py, lazily-zig, lazily-kt, lazily-js, lazily-dart | reactive Cell (active configuration) | these run their own reactive graph; the chart composes with slots/signals/effects |
A chart is never exposed over FFI. Every binding implements the transition
as pure logic with zero system dependencies; routing it through JNA/koffi to a
Rust Context would be circular (the only thing FFI buys a consumer is an
authoritative projection, which a local chart is not).
Declarative chart form
Conformance fixtures and cross-language chart definitions use a flat id map.
States form a tree via parent. The full form:
{
"initial": "<state-id>",
"context"?: { … },
"states": {
"<id>": {
"parent"?: "<id>",
"kind"?: "atomic" | "compound" | "parallel" | "history" | "final",
"initial"?: "<child-id>", // compound only
"parallel"?: true, // AND-state: children are concurrent regions
"history"?: "shallow" | "deep", // history pseudo-state
"default"?: "<target-id>", // history: resume target before any recording
"on"?: { "<event>": <transition>, … },
"entry"?: [<action>, …],
"exit"?: [<action>, …],
"run"?: [<action>, …]
}
}
}
kind is optional and inferred: history when history is set; parallel
when parallel is true; compound when the state has children; otherwise
atomic. When kind is present it is authoritative, not an annotation: it
MUST be one of the five values above and MUST agree with the structural fields
and child relation. final is the only kind that cannot be inferred; it MUST
have no children, initial, parallel, or history. A contradictory or
unknown declared kind makes the entire chart malformed.
Every id-valued reference MUST name a state declared in the same states map
before the chart can execute. This includes parent, compound initial,
history default, transition target, and the chart-level initial.
Implementations MUST reject an unresolved reference while building the chart;
they MUST NOT infer an undeclared id as an atomic leaf or allow it into the
active configuration.
A <transition> is a bare target-id string (shorthand for
{"target": id}) or an object:
{ "target": "<id>", "guard"?: <guard>, "action"?: [<action>], "internal"?: false }
Structural rules
- Exactly one state has no
parent— the root. - A compound state (one with children) MUST declare
initial, which MUST resolve to a leaf by descending compoundinitials. - A parallel state MUST NOT declare
initial; its children are the concurrent regions, and all of them are active whenever the parallel state is. - A history state MUST NOT declare
initialorparallel; it MUST declarehistoryand SHOULD declaredefault. Itsparentis the region whose configuration it records. - An atomic state has no children, no
initial, and noparallel. - A
finalstate signals completion of its parent region (see Completion).
Active configuration
The active configuration is the set of states that are currently active:
- A state is active if any of its children is active (compound), or all of its region children are active (parallel), or it is an active leaf.
- For a single-region chart, the configuration is the path root → active leaf and contains exactly one leaf.
- For a chart with parallel regions, the configuration contains one leaf per region plus all their ancestors (including the parallel state itself).
The set of active leaves is the subset of atomic active states.
Transition selection
send(event) runs run-to-completion: it computes the next configuration
from the current one with no interleaving of external events.
1. Enabled transitions
For each active leaf, walk up its ancestor chain. At each ancestor,
collect every transition whose on[event] matches event and whose guard
passes. A guard passes when it is absent, resolves true (see Guards),
or — for a context-expression guard — evaluates true against context. A
single event may enable transitions in more than one region (parallel
charts fire concurrently).
2. Conflict resolution
Two enabled transitions conflict if their exit sets (computed next) intersect — i.e. one would exit a state the other needs. Resolve conflicts by:
- If one transition’s source is a descendant of the other’s source, keep the descendant (innermost wins).
- Otherwise keep the one that appears first in document order (the order
states and their
onentries are declared).
The surviving set is taken atomically.
3. Exit and enter sets
For each taken transition with source s and target t:
lca= the lowest state that is an ancestor of both the active leaf ins’s region andt. (For a transition internal to one parallel region, thelcastays inside that region and the sibling regions are untouched.)- Exit set = active states strictly below
lca, restricted tos’s region subtree. - Enter set = the path
lca → t, then:- if
tis compound, descend viainitialto a leaf; - if
tis parallel, enter every region child and descend each; - if entering a region whose recorded history exists, descend via the
recorded configuration instead of
initial(see History); - if
tis a history state, resume its parent region per its recorded configuration (ordefaultif none).
- if
The total exit set is the union over taken transitions; the total enter set is
the union, plus any parallel siblings forced active by entering a parallel
state. Exit actions run innermost-first (reverse document order within the
exit set); then transition actions run; then entry actions run
outermost-first (document order within the enter set).
Internal vs external. A transition with internal: true whose target is
the source or a descendant of it does not exit/re-enter the source — only
its action runs and descendants below the target are reconfigured. The
default is external (exit/re-enter the source).
4. Apply
The new configuration is the old one minus the total exit set plus the total
enter set. If no transition was enabled, the event is rejected: return
false, state unchanged.
Single-region specialization
With no parallel regions the algorithm collapses to the SCXML single-region
rule: walk up from the one active leaf, take the first passing transition, and
compute exit/enter through the lca. lca makes sibling-substate transitions
(e.g. a.x → a.y) cheap (exit only x, enter y) and cross-subtree
transitions (a.x → b.y) correct (exit a.x…, enter b.y…).
Queries
active()→ the active leaf id. Defined for single-region charts; for charts with parallel regions it is undefined (useconfiguration()).configuration()→ the full set of active state ids (leaves plus all active ancestors).activeLeaves()→ the set of active atomic state ids (one per region).matches(id)→trueiffidis inconfiguration()(the hierarchical “state-in” predicate).
History
A history pseudo-state records its parent region’s active configuration whenever that region is exited, and resumes it when the region is re-entered by a transition targeting the history state.
- Shallow (
"history": "shallow"): records/restores the direct child of the parent region that was active. Nested configuration below that child is re-derived from the child’s owninitial. - Deep (
"history": "deep"): records/restores the full nested leaf configuration of the parent region, across all descendant compound and parallel levels. - First entry: when a region with a history child has never been exited
(no recording), a transition targeting the history state enters
defaultinstead.defaultSHOULD be declared; if absent, the region’sinitialis used.
A region records history on every exit, including exits caused by a transition that leaves the region without targeting its history, so a later re-entry via history resumes the most recent configuration.
Actions
Actions are host-resolved side effects, never part of the configuration.
A binding accepts an action handler name → effect; for conformance replay,
each step asserts the ordered trace of action names that fired.
entry— fired when the state enters, after its ancestors’ entries.exit— fired when the state exits, before its ancestors’ exits.run— ongoing (do) actions: started on entry, cancelled on exit. Host-managed; not part of conformance replay.- transition
action— fired after the exit set and before the enter set.
Order, restated: exit actions (innermost-first) → transition action → entry actions (outermost-first), per the exit/enter set computation.
Guards
A transition may name a guard. A guard is either:
- a bare string — a named guard resolved by the caller’s guard resolver
(
name → bool); when no resolver is supplied or the name is unknown it is treated asfalse(fail-closed); or - an object
{"expr": "…"}— an extended-state expression the host evaluates againstcontext.
Guards are pure predicates over caller-supplied state, never over the chart’s own configuration. For conformance replay, each step supplies its guard outcomes explicitly by name, so every binding reproduces identical behavior without shipping a guard evaluator.
Extended state
context is optional host-resolved caller state over which context-expression
guards evaluate. It is never serialized as part of the active configuration; it
lives outside the chart. A binding that supports context exposes it to guard
expressions and transition/entry actions; a binding that does not MUST reject
charts that use {"expr": …} guards explicitly rather than silently treating
them as passing.
Completion
A final state marks its parent region complete. When every region of a
parallel state reaches a final child, the parallel state itself is complete
and a completion (done) event is raised for the parent. This is the SCXML
automatic transition on completion. (Bindings MAY defer final/completion to a
later revision, but MUST reject final explicitly if unsupported.)
Reactive binding
In reactive bindings the active configuration lives in a Cell. On
send(event) the pure transition produces a new configuration; the cell’s
!= (PartialEq) guard suppresses downstream invalidation when the
configuration is unchanged (e.g. a no-op self-transition or a parallel-region
transition that doesn’t change the leaf set). Any Slot / Signal /
subscriber reading active(), configuration(), or matches() is
invalidated on a real transition.
Context-layer variants (non-normative)
A binding MAY expose the chart over each concurrency layer its context stack
supports, exactly as it does for the flat StateMachine — one chart type per
context layer, sharing a single context-free transition engine so the Harel
semantics (and hence conformance) are identical across layers. lazily-rs ships
StateChart (single-threaded Context), ThreadSafeStateChart
(ThreadSafeContext), and AsyncStateChart (AsyncContext); lazily-kt ships
StateChart + ThreadSafeStateChart (it has no async state machine);
projection-only bindings without a reactive graph (lazily-js) expose a single
plain-Set chart with no thread-safe/async variant. Which layers exist is a
binding property; the transition function they compute is the same one this
spec fixes.
Typed builder (non-normative)
fromChart (the declarative JSON form above) is the normative, conformance-
tested definition path. A binding MAY additionally offer a typed native builder
that constructs the same ChartDef — e.g. lazily-rs ChartBuilder, lazily-kt
ChartBuilder, lazily-js ChartBuilder. A chart assembled by the builder MUST
be behaviourally identical to the same chart parsed from JSON (both derive the
children / root / depth structure through one shared assembly step); the
builder is a source-ergonomic convenience, not a distinct semantics.
Self-transitions
A transition whose resulting configuration equals the current one is a no-op:
accepted (true) but the cell’s PartialEq guard suppresses downstream
invalidation, identical to the flat StateMachine self-transition rule. (For
an external self-transition that must re-run entry/exit, the configuration
object is still replaced; whether that invalidates dependents is the binding’s
documented choice — conformance asserts the action trace, not invalidation
counts.)
Conformance fixtures
Canonical charts live in
conformance/statechart/. Each binding loads
them, asserts initial_active (and initial_actions when present), replays
the steps, and asserts accepted, active, matches, and (when present)
actions after each step. For parallel charts active is an array of active
leaves (sorted); for single-region charts it is a single leaf id.
Fixture schema
{
"description": "…",
"kind": "StateChart",
"initial_active": "leaf" | ["leaf", …],
"initial_actions"?: ["action", …],
"chart": { …per schemas/statechart.json… },
"steps": [
{
"event": "START",
"guards"?: { "name": true },
"accepted": true,
"active": "leaf" | ["leaf", …],
"matches"?: { "state-id": true },
"actions"?: ["action", …]
}
]
}
initial_active— expected active leaf (or leaves for parallel) after descendingchart.initial, asserted once before any step.initial_actions(optional) — ordered action names fired during initial entry.event— the event sent.guards(optional) — per-step named guard outcomes for this send.accepted— expectedsendreturn value.active— expected active leaf (single) or active-leaf set (parallel) after the step.matches(optional) —{ state-id: bool }expectations formatches().actions(optional) — ordered action names fired during the step (exit → transition → entry).
Current fixtures:
| Fixture | Covers |
|---|---|
flat_cycle.json | flat (single-level) transitions, rejection, cycle |
hierarchical_player.json | nesting, walk-up transition resolution, LCA across levels, matches() |
guarded_door.json | named guards, fail-closed rejection, guard pass |
parallel_regions.json | orthogonal (AND) regions: per-region transitions, matches() across regions, exiting all regions |
history_shallow.json | shallow history: resume last direct child on re-entry; first-entry default |
history_deep.json | deep history: resume full nested leaf configuration; sticky across cycles |
entry_exit_actions.json | entry/exit/transition action ordering across LCA boundaries |
Implementation status
The single-region subset (compound states, walk-up resolution, LCA, guards,
self-transitions) is required of all bindings. Orthogonal regions, history,
actions, and extended state are specified here in full; a binding MAY implement
a subset, but MUST reject any feature it does not implement explicitly
(never silently ignore a parallel, history, entry/exit/run, or
{"expr": …} guard). Each binding’s conformance run selects the fixtures that
match its implemented subset.
Async Reactive Context
An async context is a separate reactive surface for computations whose
values are produced by async/future-returning functions. It is not an
overload of the synchronous or thread-safe context; it is a distinct graph with
its own handles, because futures introduce in-flight state, cancellation, stale
completion, and dependency tracking across suspension points that the
synchronous graph does not have.
This chapter fixes the cross-language contract. An async context is compute, not protocol — only resolved slot values cross IPC/FFI as ordinary cell payloads, exactly like the synchronous graph.
Why a separate surface
A synchronous reactive context tracks dependencies through a thread-local stack
touched on every read, and a slot’s value is either present or unset. An async
computation can be in-flight (suspended at an .await) when its inputs
change, can complete after those inputs are gone, and can be canceled mid-flight.
Those states require:
- an explicit per-slot state machine (not just present/absent),
- revision tracking so a stale completion is discarded,
- dependency edges registered before the read is awaited, and
- a cancellation contract that is safe under waiter drop, supersession, and context disposal.
A binding gates this surface behind a separate feature flag so downstream users do not accidentally accept the larger semantic surface.
Handles
An async context exposes its own copyable, id-only handles, distinct from the synchronous handles:
| Handle | Wraps |
|---|---|
AsyncSource<T> | A mutable input cell (the synchronous input layer) |
AsyncComputed<T> | A computed/memoized async slot |
AsyncEffectHandle | An async effect |
Handles are id-only and copyable; they are usable only with the owning async context.
AsyncSource and AsyncComputed are the canonical public value kinds in every
binding. Published pre-v2 spellings such as AsyncCellHandle and
AsyncSlotHandle may remain only as deprecated aliases of those exact types.
Constructing through an old spelling must still return the canonical type.
API surface
| Method | Description |
|---|---|
source(value) | Create a mutable AsyncSource (value type equality-comparable, cloneable, Send + Sync) |
get(source) | Read an AsyncSource value synchronously |
set(source, value) | Update an AsyncSource and invalidate dependents |
computed / computed_async(compute) | Create an AsyncComputed; the exact spelling follows the binding’s async conventions |
computed_with_equals(compute, equals) | Create a guarded AsyncComputed with an explicit equality policy |
get(computed) -> Option<T> | Read an AsyncComputed cache synchronously; Some(T) if resolved, None otherwise (warm-path fast path) |
get_async(computed) -> T | Await an AsyncComputed; uses get() for resolved values, otherwise spawns async compute |
memo / memo_async(compute, equals) | Deprecated compatibility constructor forwarding to guarded computed; memo is not a third node kind |
effect_async(effect) | Create an async effect with an async cleanup |
dispose_async_effect(handle) | Dispose an async effect and await its cleanup |
batch(run) | Synchronous batch boundary; schedules async reruns at batch exit |
Sources are the synchronous input layer: source, get, and set are
synchronous. Only computed evaluation and effects are async. The constructor and
read/write vocabulary is deliberately the same as the local and thread-safe
contexts; only the async handle types and get_async are execution-model-specific.
The public state projection is AsyncComputedState. The formal theorem/module
name remains AsyncSlotState, because it describes the storage state machine,
not a public value-handle kind. Bindings with a published AsyncSlotState API
keep it as a deprecated alias of AsyncComputedState; formal references are not
renamed mechanically.
Async computed state machine (formal AsyncSlotState)
Each async slot tracks its state through a finite state machine:
first get_async / future Ok, dependency
Empty ──────────────────► Computing ──────► Resolved ──────► Computing
▲ (spawn) │ │ │
│ │ │ future Err │
│ │ ▼ │
│ │ Error ──────────────────────────┘
│ │ retry get_async
│ ▼
│ dependency invalidation revision mismatch on
└──── during in-flight compute ──► (stale) Computing ──► complete discards;
hard clear new future spawned
| State | Meaning |
|---|---|
Empty | No cached value, no in-flight computation. Entered on creation and after a hard clear. |
Computing | A handle tracks the in-flight future for the current revision. Concurrent get_async callers attach as waiters to the same in-flight result instead of spawning duplicate futures. |
Resolved | The cached value is fresh, until dependency invalidation transitions back to Computing. |
Error | The last computation failed. Every caller waiting on that attempt receives its error. The error is not a cached value: the next get_async MUST re-spawn (Error → Computing) rather than replay the stored error. |
Revision tracking is load-bearing: a computation records the slot revision at start; at publish time the graph accepts the value only if the revision is still current. This is what makes stale completion safe.
Transitions:
Empty → Computing— firstget_asyncor invalidation with no cached value.Computing → Resolved— future completesOkand the recorded revision still matches.Computing → Error— future completesErrand the recorded revision still matches.Computing → Computing (stale)— invalidation advances the slot revision during an in-flight computation. The completing future finds its revision no longer matches and discards the result; a new future is spawned for the updated revision.Resolved → Computing— invalidation marks the cached value stale and spawns a new computation.Error → Computing—get_asyncretry after an error. This transition is mandatory, not optional: a slot inErrorholds no cached result, so the next read re-spawns for the current revision. A binding that replays the stored error to every later reader is non-conforming — it makes a transient failure (a timed-out fetch, a disconnected peer) permanent for the lifetime of the slot, with no read path that can recover it.invalidateon anErrorslot is therefore a no-op rather than a repair: the retry is owned by the read, not by a dependency change. The executable form isLazilyFormal.AsyncSlotState.step … SlotEvent.retry, which mapserror → computingand is a no-op from every other state.
Cancellation contract
A conforming async context MUST honor all five of:
-
Waiter cancellation is safe. Dropping one
get_asyncfuture does not cancel the shared in-flight computation while other waiters still need it. Each waiter holds a shared handle; dropping a receiver does not abort the in-flight task. -
Stale completion is discarded, not published. When invalidation advances the slot revision during an in-flight computation, the completing future finds its recorded revision no longer matches and discards the result. Waiting callers are retried against the new revision or attached to the newly spawned future.
-
Explicit cancellation. A hard clear, invalidation, or context disposal may mark the in-flight revision as canceled; if an abort handle is available, the task is aborted. User futures MUST be cancellation-safe, because aborting drops them at an
.awaitboundary. -
Context disposal. Dropping the async context cancels all in-flight computations via their abort handles and awaits completion of all active cleanup futures before returning.
-
Effect cleanup is triggered by rerun or dispose, and completes before the next body. An effect’s cleanup future MUST run only when the effect reruns or is disposed — the trigger fixed by
reactive-graph.md§ Conformance (“cleanup runs before each rerun and on dispose”) — and when it runs on a rerun it MUST complete before the next body starts. Disposal removes pending reruns before awaiting cleanup.A binding MUST NOT run cleanup at the end of the flush that ran the body. This clause previously stated only the ordering (“cleanup completes before the next body starts”) and left the trigger to
reactive-graph.md, which turned out to be enough rope:lazily-py’s async effect awaited cleanup at flush end whenever no rerun was queued, and satisfied the ordering vacuously because there was no next body to be ordered against. Found 2026-07-19 bydisarm_disposes_nothing, which assertscleanup_order: []at a step where nothing was disposed or invalidated, and observed a cleanup.The reason the trigger matters rather than just the ordering: the canonical effect acquires a resource in the body and releases it in the cleanup. Running cleanup at flush end releases the resource while the effect is still live and will rerun later — an effect that subscribes and returns an unsubscribe would unsubscribe itself immediately.
lazily-goandlazily-dartboth retain the cleanup between runs and execute it at the start of the next one; that is the family behavior.
get_async re-resolve contract
get_async MUST treat the slot state as authoritative and re-resolve rather
than assert, because the slot can change between its lock acquisitions and a
notifier can close under it. It runs an outer loop that, each pass, re-reads the
slot via the get() fast path and then re-locks to attach to or spawn a
computation. Two concurrency windows are benign (not data inconsistencies — the
published value is always correct):
- Resolved-since-
get(): the slot can transitionComputing → Resolvedbetween the fast-pathget()(which releases the lock) and the re-lock. ObservingResolvedat the re-lock is expected; the cached value is read directly. It is not an unreachable state. - Notifier dropped: the per-computation waiters can all close without a
final
Resolvedsignal when an in-flight compute is superseded by a newer revision (the staleComputing → Computingtransition early-returns) or the slot is invalidated. A “the world changed” error means re-resolve from current slot state — return the now-published value, attach to the new in-flight compute, or respawn. It MUST NOT panic.
Dependency tracking
Async compute and effect callbacks do not use a thread-local tracking stack
(a thread-local does not survive executor thread migration or suspension/resume
across .await). Instead each callback receives a compute context:
| Method | Tracking |
|---|---|
get_async(computed) | Records the accessed computed as a dependency before awaiting its value |
get(source) | Records the accessed source as a dependency synchronously |
Edges register immediately, so source invalidation while the future is suspended can cancel or supersede the in-flight computation before it publishes stale data. On rerun, stale dependencies are removed and new ones registered; the dependency set is carried by the compute context, not a thread-local.
Async effects
An async effect runs an effect body returning an optional async cleanup:
- Serialized reruns: reruns are serialized per effect — a rerun does not start until the previous cleanup future completes.
- Cleanup trigger and ordering: cleanup runs on rerun or dispose and at no other time — not at the end of the flush that ran the body. When it runs on a rerun, the previous run’s cleanup completes before the next body starts; disposal awaits the current cleanup before removing the node. The cleanup is therefore retained between runs rather than executed eagerly. See § “Conformance” item 5 for why the trigger is normative and not merely the ordering.
- Auto-tracking: the body receives a compute context and tracks dependencies
through
get_async/get. - Scheduled, not inline: dependency invalidation schedules an async rerun
after the current invalidation pass; the rerun runs on the runtime executor,
not inline within
send/batch. - Disposal: removes pending scheduled reruns, awaits the current cleanup future, and unsubscribes dependency edges.
Batch support
batch(run) is a synchronous boundary. Cell updates queue invalidation
roots; at batch exit, queued roots trigger propagation. Async slots and effects
are scheduled for rerun but do not execute inside the batch callback — async
reruns execute after the batch returns, on the runtime executor. Mutation
semantics stay synchronous at the graph boundary: invalidations schedule async
reruns only after the outermost batch exits.
In-flight deduplication & fast path
- One in-flight computation per revision: each async slot has one published
cache and at most one in-flight computation for the current revision.
Concurrent
get_asynccallers await the same in-flight result instead of spawning duplicate futures. - Synchronous fast path:
get()returns the cached value synchronously when the slot isResolved, avoiding async overhead.get_async()callsget()first; only unresolved or dirty slots enter the async spawn path.
Async state machine
A flat State Machine over the async context keeps send and
state synchronous (sources are the synchronous input layer) while reactive
observers use the async APIs: on_transition returns an async effect handle and
state_is returns an async signal handle. Because resolution is asynchronous,
eager recomputation settles on the runtime rather than synchronously within
send.
Conformance
An async context conforms when:
- The slot state machine (
Empty/Computing/Resolved/Error) and its transitions, including the staleComputing → Computingdiscard, are implemented exactly. - Revision tracking discards every stale completion; a stale value is never published.
- All five cancellation properties hold, and disposal awaits cleanup.
get_asyncre-resolves through both benign-race windows without panicking.- Dependencies are tracked through the compute context (not a thread-local) and registered before the awaited read.
- Async effect reruns are serialized, cleanup-before-body ordered, and executor-scheduled rather than inline.
- Batching is synchronous at the mutation boundary; async reruns fire only after the outermost batch exits.
Implementation status
The async surface is required of any binding whose platform exposes an
async/future runtime (async/await, promises, coroutines, a suspending
executor) — see Wire Protocol § Concurrency layers are required.
A binding on a platform with no notion of suspendable async computation declares
the async capability as none and advertises it, never silently. A binding
that ships the async context MUST honor the full cancellation and re-resolve
contract above. API-shape checks for the canonical async-v2 names are
informational only: they do not admit a peer to transport, CRDT, queue, or sync
feature groups. Network-suite membership continues to follow the binding’s
advertised production capabilities, channels, codecs, and variants.
Concurrency-window coverage is pinned by targeted deterministic tests rather than
exhaustive interleaving exploration, because the async resolve loop runs on a
real async executor whose primitives a synchronization-model checker cannot shim.
Lossless Tree CRDT
LosslessTreeCrdt is a single rooted concrete-syntax tree whose leaves own
every rendered byte, so the tree itself is the lossless wire authority — no
separate flat text CRDT floor is required for capable sessions. The defining
invariant is losslessness:
render(tree) == source_text
for valid, invalid, and unknown source alike. This chapter specifies the M1
syntax-agnostic core: the tree state model, node/op id formats, the leaf and
element node variants, the dotted non-contiguous version frontier, the op
vocabulary, the byte-offset policy, and the compute-fixture contract every binding
replays. The Lean model of the render algebra, convergence, and frontier soundness
lives in lazily-formal (LazilyFormal.LosslessTree,
LazilyFormal.LosslessTreeSync).
M1 scope. Create / tombstone / intra-parent reorder / leaf-edit / split-leaf / merge-adjacent-leaves, plus op-based delta sync, plus the op-delta wire schema (see Wire schemas). Deferred to later milestones: cross-parent move (single-parent + acyclicity enforcement), metadata/kind mutation, subtree replace, snapshot/GC and its wire schema, and full
SessionHandshakecapability negotiation (the capability name is reserved below). The Kotlin and JS bindings land in M2. In the create-only + intra-parent-reorder algebra, single-parent and acyclicity hold by construction.
Tree state model
The document is one rooted tree. Every node has a stable identity and belongs to exactly one parent’s ordered child list.
DocumentRoot— the sentinel root element (TreeNodeId{counter: 0, peer: 0}).ElementNode— an internal semantic node with akindand an ordered list of children. Owns structure only, never text.Leaf— owns one exact source span, classified by aLeafKind:Token— a syntax delimiter or marker;Trivia— whitespace, blank lines, indentation, comments, separators;Raw— valid text the adapter deliberately keeps opaque;Error— invalid or ambiguous text that must still round-trip exactly.
render is a depth-first concatenation of the live leaves’ text in child order.
Because every rendered byte belongs to exactly one live leaf and elements own no
text, unknown or invalid spans round-trip exactly as Raw/Error leaves rather
than being discarded — the requirement that keeps a semantic AST from being
mistaken for a lossless tree.
Identity and clock
TreeOpId{counter, peer}— a dotted operation id: a Lamport counter tiebroken by peer, totally ordered by(counter, peer). The counter advances past every observed op, so a causally-later write wins last-writer-wins and concurrent ops tiebreak deterministically by peer. (No HLC is needed; tree anti-entropy is op-based.)TreeNodeId— a node’s identity is the id of the op that created it, so a node keeps its id through reorder, edit, and (future) move.SortKey{frac, peer}— a fractional-index child position: orderable bytes tiebroken by the minting peer, so concurrent inserts into the same gap get a deterministic total order. Positions travel inside create/reorder ops, so both replicas store byte-identical keys.
Operation vocabulary
Every mutation is one op carrying everything a remote replica needs to converge deterministically (positions and seed text travel inside the op):
| Op | Effect |
|---|---|
CreateNode {id, parent, sort, seed} | materialize an element shell or a text leaf seeded from exact text |
Tombstone {node} | tombstone a node (sticky; smaller op id wins concurrently) |
Reorder {node, sort} | LWW position reassignment within the parent (identity + payload preserved) |
LeafEdit {node, prev, ops} | apply an embedded text-CRDT delta to one leaf |
SplitLeaf {node, new, sort, at_char, prev} | split a leaf at a char boundary into two adjacent leaves of the same kind |
MergeLeaves {left, right, prev_left, prev_right} | merge two adjacent leaf siblings; total text unchanged |
Split and merge reseed a leaf’s text destructively, so per-leaf text ops form a
causal chain: each LeafEdit/SplitLeaf/MergeLeaves carries the prior
text-op id (prev) and is buffered until it arrives, keeping out-of-order delivery
convergent.
Dotted version frontier
The frontier summarizing “which ops do I hold” is a dot set — per peer, a contiguous prefix plus out-of-order holes — never a per-peer max counter:
frontier[peer] = { contiguous, sparse: {…} }
diff(their_frontier) returns the ops a replica holds that the partner’s frontier
lacks (a true set difference over dots), ordered by dotted id. apply_update is
idempotent (already-held ops are skipped) and buffers ops whose parent/target or
prev has not arrived yet.
A per-peer max would record only the highest delivered dot and imply every lower
dot is held; delivering dot 3 while dot 2 is missing would make the partner believe
it holds 2, so its diff would omit an op it genuinely lacks and the replicas would
never converge. The dot set keeps the hole representable and re-requestable. This
is proven in LazilyFormal.LosslessTreeSync (frontier_no_skip,
perPeerMax_skips) and exercised by the
non_contiguous_anti_entropy fixture.
Offset policy
Wire and API text offsets are UTF-8 byte offsets, leaf-local. The embedded text
CRDT is char-indexed, so the byte→char conversion happens only at the two
byte-taking mutators (edit_leaf, split_leaf), against that leaf’s current text.
Offsets must land on a UTF-8 char boundary and within the leaf, else the mutation
is rejected. No binding may treat UTF-16 code units as wire offsets; the Kotlin/JS
bindings (later milestones) must convert.
Conformance
The M1 fixtures live in conformance/lossless-tree/ as compute fixtures — each
builds an initial tree on replica a, runs a schedule of ops / forks /
anti-entropy syncs across named replicas, and asserts exact rendered text,
live-node counts, and convergence across delivery orders:
exact_roundtrip— Token/Trivia/Raw/Error leaves incl. an invalid span and multi-byte text;render == source;one_leaf_edit_delta— a one-leaf edit at a byte offset into multi-byte text, delivered by anti-entropy;split_merge— split then merge, render preserved, live-node count grows then restores;concurrent_insert_same_parent— two replicas insert into the same gap; both survive, deterministic order;concurrent_reorder_and_leaf_edit— a concurrent move + text edit both apply;non_contiguous_anti_entropy— a delivery hole is re-requested and converges;token_trivia_preservation— a leaf edit leaves adjacent Token/Trivia leaves byte-for-byte unchanged;invalid_source_roundtrip— unclosed fence / comment carried as Error leaves round-trips exactly, and editing an adjacent Raw leaf keeps the Error spans;concurrent_conflict_preserves_text— incompatible concurrent shapes (element wrap vs bare leaf) both survive with no bytes dropped (text preservation wins over semantic shape; adapter-level raw/error degradation layers above the core);apply_update_advances_counter— the counter advance above, observed: a replica reorders AFTER syncing in six remote reorders, and its new op must outrank the stamp it just ingested. The failure is symmetric (both replicas converge on the wrong text), sorender_onis what sees it, notconverged;out_of_order_delivery_buffers— a three-op batch delivered in reversed order (deliver.order) must drain through the dependency buffer; a binding that drops what it cannot yet apply still records the dot, so the following full sync returns nothing and the loss is permanent.
See Conformance Fixtures for the fixture format and the binding replay contract.
Wire schemas
The op-delta wire form is pinned by two JSON Schemas, derived from the lazily-rs reference’s serde output (which is the normative form):
schemas/lossless-tree.json— the shared vocabulary$defs:OpId,SortKey(fracis a u8 array, never base64),LeafKind(PascalCase on the wire),NodeSeed,TextOp, the externally-taggedTreeOpKind/TreeOp, and the dottedTreeVersionFrontier/DotRange.schemas/lossless-tree-delta.json— theTreeUpdatemessage:{ "ops": [TreeOp, …] }, the output ofdiffand the input toapply_update.
The reference validates its own TreeUpdate and frontier serde output against
these schemas (lazily-rs/tests/lossless_tree_schema.rs), and the M2 Kotlin/JS
ports validate their emitted frames against the same files. The snapshot/checkpoint
wire schema is deferred with snapshot/GC.
Capability
A binding that implements this chapter advertises the capability name
lossless_tree_crdt_v1 (reserved here; wired into SessionHandshake capability
negotiation in a later milestone). A peer that lacks it falls back to the flat
text CRDT floor.
CrdtTree Contract (#lzcrdttree)
CrdtTree is the shared lossless document-CRDT seam used by snapshots, relay
canonicals, and document-op replication. An implementation exposes:
version_vector()— a compact distributed frontier;delta_since(frontier)— every operation the frontier has not observed;apply_delta(delta)— an identity-preserving, idempotent fold;text()/value()— the visible lossless projection; andmerge_from(other)— the state join.
The join and delta fold are commutative, associative, and idempotent. A full
snapshot is exactly delta_since(empty_frontier); applying it to an empty replica
must preserve operation identities, not reparse the visible text and mint a new
lineage. Applying delta_since(version_vector()) is a no-op.
The compute fixture at
conformance/crdt-tree/algebra.json
pins merge order independence, idempotence, snapshot equivalence, and incremental
round-trip behavior. The Lean model in LazilyFormal.Replication pins the three
join laws.
Durable Outbox Stores (#lzdurableoutbox)
Reliable sync separates protocol from persistence:
OutboxStore (ordered byte CRUD) → Outbox<S> (ack/prune/replay protocol) → SyncDriver
OutboxStore owns only put, delete_through, scan_after, load_cursor, and
save_cursor. Outbox<S> owns serialization and the invariants:
- append before send;
- retain every frame until the peer acknowledges its epoch;
- keep the acknowledgment cursor monotone, including when a stale storage handle writes after a newer handle;
- prune only epochs at or below that cursor; and
- replay epochs above the cursor in ascending order after restart.
Persistent save_cursor(epoch) implementations MUST serialize
max(stored_cursor, epoch) atomically. A process-local maximum is insufficient:
two handles can both open at cursor zero, then write 9 and 3 in that order. The
serialized result must remain 9, and a subsequent protocol read must observe 9.
Bindings may provide platform stores without duplicating this protocol. Rust
ships InMemoryStore and feature-gated SqliteStore; Kotlin ships a
SQLite/Room-shaped store; browser JS ships IndexedDbStore; other native
bindings ship append-only file journals. SQLite is never a default/WASM feature.
conformance/reliable-sync/outbox_store_protocol.json
pins the storage-independent behavior. LazilyFormal.Replication proves cursor
monotonicity and that replay never contains a pruned epoch.
An append-only journal is a versioned durable wire surface: a later process, and possibly a different build of the same binding, reads records written earlier. Readers MUST reject a complete record whose opcode they do not recognize. They MUST NOT skip it as though it were a cursor marker, because an ignored pruning operation can resurrect acknowledged frames and redeliver them while reporting a successful scan.
The sole recovery exception is a torn trailing record. A crash may interrupt the
last append, so a reader MUST forgive one incomplete final record and preserve
every complete record before it. The same malformed bytes in an interior record
are corruption and MUST be rejected. The logical, encoding-neutral cases are
pinned by
conformance/reliable-sync/outbox_journal_decode.json;
bindings encode them into their native JSON-lines or binary journal format before
replay.
Durable Effect Sinks (#lzdurablesink)
Lazily fixes one direction of authority between live reactive state and durable storage:
cold durable state ──hydrate once──▶ live Lazily state
│
▼
computed projection or ordered fact stream
│
▼
Effect / AsyncEffect
│
▼
write-only durable sink
│
ack / failure
└────────▶ live Lazily state
Invariant. While a Lazily runtime is live, transitions are decided from Lazily state. Durable storage receives a projection or an ordered fact as an effect. A sink MUST NOT reload storage and use it to arbitrate the transition it is currently persisting.
This makes durable storage an effect sink. It does not claim that every effect is durable, nor that durable storage is always eventually consistent. Lazily is not a database framework: applications continue to own serialization, transactions, schema migration, storage selection, and startup recovery. Lazily ships no SQLite, filesystem, or cloud-store adapter as part of this decision — the sink is an application-owned trait (see examples below).
Authority rule
- Live coordination, compare-and-swap, ownership, deduplication, leases, and transition selection happen in Lazily state.
- Durable I/O runs from an
Effect/AsyncEffect, or from aTopicCell/DurableOutboxdrain when history must be lossless. - A runtime sink is write-only with respect to transition authority. Loading and migration belong to a separate startup hydrator that runs once before the runtime is live, not on the decision seam.
Computedvalues and transition reducers stay pure — a reducer MUST NOT perform I/O or read storage to decide a transition.- Success advances a monotone acknowledgement such as
durable_through(epoch). - Failure stays represented in live state as
pending/retrying/backpressured. It MUST NOT trigger a database reload at the decision seam. - Values on the existing
Ephemeralplane MUST NOT enter a durable sink. Reuse the existingDurablemarker — do not invent a second marker hierarchy. (TheEphemeral-never-Durableseparation is already pinned byPresence.ephemeral_never_durable.)
Projection vs history — pick the shape before the API
Ordinary Effect / AsyncEffect observe settled reactive state. Lazily
already coalesces effect reruns across a batch, so a batch containing
A → B → C may persist only C. That is correct for a durable projection
(current/recoverable state). Durable history has a different contract: every
accepted fact must survive and remain ordered. It uses the existing
TopicCell / DurableOutbox
family — stable cursor, replay, idempotency key, and monotone acknowledgement —
not ordinary effects. Do not modify ordinary effects to retain intermediate
values; that would weaken their coalescing semantics and duplicate the
ordered-stream primitives.
Before choosing an API, answer two questions:
- Is this sink persisting the latest projection or an ordered history?
- Must the transition be durable before it is externally visible?
Those two answers fix the shape:
| Need | Lazily source | Sink contract |
|---|---|---|
| Latest recoverable state | Computed read by Effect / AsyncEffect | Idempotent upsert of the latest epoch |
| Every accepted fact | TopicCell or existing DurableOutbox | Append / replay / ack with a stable cursor |
| Durable before visible | Ordered fact + application acknowledgement cell | Visibility waits for monotone durable_through |
| Ephemeral state | Presence / Ephemeral primitives | Persistence rejected |
Application-owned sink trait
The sink is a narrow, write-only trait owned by the application. The reactive graph holds no reference to a query/hydration interface — those live in a separate module used only by the startup hydrator, never passed into the runtime effect. (Phase 3 adds architecture tests that fail if a hot-path actor module imports the persistence query/CAS or file-lock modules.)
The two examples below sketch the trait shape in Rust-flavoured pseudocode. The
store API is deliberately minimal — upsert_latest for a projection,
append_fact for history — and always carries the epoch so success can advance a
monotone durable_through.
Example 1 — coalesced current-state projection
A Computed projects the actor’s current state. An Effect reads it and
upserts only the settled value; intermediate batch values are coalesced away by
Lazily’s existing effect-batch dedup, so the sink sees one write per batch.
Acknowledgement advances durable_through; a sink failure flips the live actor
to retrying without reloading storage.
#![allow(unused)]
fn main() {
// Application-owned, write-only. No read/CAS surface reaches the runtime effect.
trait ProjectionSink {
fn upsert_latest(&mut self, epoch: u64, state: &ActorState) -> Result<(), SinkErr>;
}
// Live state. Pure reducer; no I/O.
let live_state: Source<ActorState> = ctx.source(initial);
let epoch: Source<u64> = ctx.source(0);
let projected: Computed<ActorState> = ctx.computed(|c| c.get(live_state).clone());
// Durable I/O runs from an Effect — never reads storage to decide a transition.
ctx.effect(|c| {
let s = c.get(projected);
let e = c.get(epoch);
match sink.upsert_latest(e, s) {
Ok(()) => c.set(durable_through, e), // monotone ack
Err(_) => c.set(status, Status::Retrying), // failure stays live
}
});
}
Example 2 — lossless ordered fact sink
Every accepted fact must survive and stay ordered, so the source is a
TopicCell (or the existing DurableOutbox), drained by an effect into the
application store. Replay from a stable cursor covers every epoch after the
durable frontier; duplicate delivery is idempotent by the event_id key. This is
the DurableOutbox append-before-send
/ replay-from-cursor / ack_through contract, applied as a sink.
#![allow(unused)]
fn main() {
trait HistorySink {
fn append_fact(&mut self, epoch: u64, fact: &Fact) -> Result<(), SinkErr>;
fn ack_through(&mut self, epoch: u64);
}
let facts: TopicCell<Fact> = ctx.topic();
// Drain: every unacked fact is appended in order; ack advances the cursor.
ctx.effect(|c| {
while let Some(fact) = facts.read_after(cursor) {
sink.append_fact(fact.epoch, &fact)?;
sink.ack_through(fact.epoch); // cursor advances; GC-safe below it
cursor = fact.epoch;
}
});
}
Visibility policy (caller-chosen)
A caller picks an explicit policy per transition; Lazily mandates none:
eventual_projection— live state is immediately authoritative; the sink persists in the background and may lag.durable_before_applied— external visibility waits for thedurable_through(epoch)acknowledgement before the transition is observed as applied (an ordered fact plus an application acknowledgement cell).ephemeral— no durable sink; the value lives on theEphemeralplane.
Cold restart
Restart recovery is the hydrator’s job, run once at startup before the runtime is live: load the last acknowledged projection (or replay the ordered fact log up to the durable cursor) into live state, then resume. After hydration, authority is live-only — a sink failure during a live transition never rolls authority backward by rehydrating storage at the decision seam.
Formal backstop
lazily-formal/LazilyFormal/DurableSink.lean pins the load-bearing invariants:
durable_through is monotone; a batched projection persists only the settled
epoch (coalescing); an ordered history replays every epoch past the durable
cursor; and a sink failure leaves live authority unchanged (no rehydrate-at-
decision-seam). The Ephemeral-never-Durable separation is already proven in
Presence.ephemeral_never_durable.
Transport-agnostic reactive egress (#lzegress)
Egress is the send-side mirror of
reactive ingress. The pure EgressCore<T> owns delivery
authority:
- monotone sequence assignment;
- the pending queue and bounded unacknowledged window;
- the monotone cumulative acknowledgement watermark;
- retry attempts, bounded exponential backoff, and exhaustion; and
- the producer-generation fence.
The graph shells expose that state as Computeds. Transport I/O lives in exactly one Effect per attachment:
EgressCore<T> sequence, window, watermark, retry, fence
├── EgressCell<T> Context shell
├── ThreadSafeEgressCell<T> ThreadSafeContext shell
└── AsyncEgressCell<T> AsyncContext shell
│
└── one attached Effect ──send──▶ transport
An Effect is not an egress family by itself: it has no retained delivery state.
The Effect observes the pending and inflight projections, asks the core to
claim eligible envelopes, and sends those envelopes. The core decides whether a
claim, acknowledgement, failure, retry, or reconnect is admissible.
This is reactive projection, not a request/ACK protocol. Enqueue invalidates
pending; claiming moves an envelope to inflight; an acknowledgement
invalidates inflight and re-runs the same attachment when it reopens a bounded
send window. The acknowledgement is domain delivery input to the core, not an
acknowledgement of a projection request.
Envelope identity and ordering
Each enqueue assigns the next sequence in the current sequence space:
enqueue(payload) =
pending.push({ generation, sequence: next_sequence, attempt: 0, payload })
next_sequence += 1
(generation, sequence) is the transport idempotency identity. Pending records
are claimed oldest-sequence first. A successful claim increments attempt and
moves the record into the unacknowledged window. A claim is refused when that
window reaches inflight_limit.
An acknowledgement is cumulative. ack(generation, through) advances
acked_through only when through is greater than the current watermark, and
removes all pending, in-flight, or retry-parked records at or below it.
Send-side generation fence
Every transport Effect captures the generation current when it attaches. Every send-side transition supplies that generation to the core. A mismatched generation is rejected without changing state or invalidating a reader.
reconnect(new_generation) must strictly advance the generation. It:
- fences the old Effect;
- preserves
next_sequenceandacked_through; - moves every unacknowledged in-flight or retry-parked record back to pending in sequence order; and
- rewrites all pending envelopes to the new generation.
The old Effect remains attached but inert. The replacement transport receives one new Effect, which replays the retained pending projection under the new incarnation.
Retry
retry_budget counts retries after the first send attempt. A failed in-flight
record is parked until the derived backoff elapses:
backoff(attempt) =
min(retry_ceiling, retry_base * 2^(attempt - 1))
While budget remains, retry_now returns the parked record to pending in
sequence order. Once exhausted, the record is terminal and cannot be scheduled
again. Parking is explicit so a failed send cannot cause the attachment Effect
to spin in an immediate reactive retry loop.
Reader kinds and invalidation
| Reader | Type | Invalidated by |
|---|---|---|
pending | Vec<EgressEnvelope<T>> | enqueue, claim, retry scheduling, replay, cumulative ACK pruning |
inflight | Vec<EgressEnvelope<T>> | claim, failure, replay, cumulative ACK pruning |
acked_through | Option<u64> | advancing ACK only |
retry | Option<EgressRetry> | failure, retry claim/scheduling, replay, ACK pruning |
No-op and stale-generation transitions invalidate nothing. The async shell uses synchronous Computeds for the delivery state; only the transport attachment is async-coloured.
Composition boundaries
Egress does not duplicate existing storage or flow-control primitives:
- use
RelayCell/Outboxahead of enqueue for conflation and backpressure; - use
SpillStorefor overflow-to-storage; - use
DurableOutboxfor durable recovery; and - use transport framing outside the core.
Those adapters feed or persist the egress state machine. They do not move sequence assignment, acknowledgement authority, or the generation fence into an Effect.
Conformance and formal obligations
The canonical corpus in conformance/egress/ covers ordered cumulative ACK,
window reopening, bounded retry, and reconnect fencing. Binding runners must
assert both returned outcomes and the full projected state after every step.
The Lean model proves:
- sequence assignment is monotone;
- acknowledgement progress is monotone;
- stale-generation send-side transitions preserve state;
- reconnect strictly advances generation while preserving the ACK watermark; and
- permitted retry attempts are bounded by the configured budget.
Cross-Language Feature Coverage
This is the canonical feature-coverage matrix for the lazily family. Each binding’s README reproduces it; this page is the source of truth. It is a status view (what each port ships today), distinct from the normative Binding Conformance Matrix, which fixes what every binding must eventually provide.
Legend: ✅ shipped · ~ partial · — absent · ⊘ not applicable (see notes).
The table below is generated from
coverage.jsonbyscripts/sync-coverage.mjs. Editcoverage.jsonand runmake coverage-sync(ornode scripts/sync-coverage.mjs) to update this table and every binding README in one shot;make coverage-checkguards drift in CI.
Summary — family × language
| Family | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reactive graph | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ~ |
| Materialization | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Family sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Statecharts | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Keyed collections | ✅ | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | — |
| Reactive queue | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Broadcast topic | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Work queue | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| CRDT data types | ✅ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ✅ | — |
| Lossless tree | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Egress | ✅ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | — |
| Ingress | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Wire codec | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | ✅ | — |
| Transport & FFI | ✅ | ✅ | ✅ | ~ | ~ | ✅ | ✅ | ~ | ✅ | — |
| Message passing | ✅ | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | — |
| Reliable sync | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | — |
| Distributed plane | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Causal receipts | ~ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Security boundary | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Membership | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Coordination | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Presence | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Temporal | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Rate shaping | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Windowing | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Resilience | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Portable stdlib | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Service plane | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Instrumentation | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Roll-up rule: a family cell is ✅ only when every required row in that family is ✅; ~ when the family is mixed (some shipped or partial); — when no required row is shipped or partial; ⊘ only when every required row in the family is not applicable. Rows the spec marks MAY (optional, shown as opt below) are excluded from the roll-up — declining an optional feature is not a gap.
Reactive graph
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reactive graph 1 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ~ |
| Thread-safe context 2 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Async reactive context 3 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Merge algebra 4 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ~ |
Materialization
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Keyed-map materialization 5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Thread-safe keyed map 6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Async keyed map 7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Family sync
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Keyed-map sync 8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Statecharts
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Flat state machine 9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Harel state charts 10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Keyed collections
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Keyed reactive maps 11 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| ReactiveMap core — single-threaded 12 | ✅ | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | — |
| ReactiveMap core — thread-safe 13 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| ReactiveMap core — async 14 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Exact-key dependency availability 15 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Atomic ordered move 16 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Memoized semantic tree 17 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Stable-id alignment 18 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Reactive queue
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reactive queue core — single-threaded 19 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Reactive queue core — thread-safe 20 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Reactive queue core — async 21 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Broadcast topic
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Broadcast topic core — single-threaded 22 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Broadcast topic core — thread-safe 23 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Broadcast topic core — async 24 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Work queue
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Work queue core — single-threaded 25 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Work queue core — thread-safe 26 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Work queue core — async 27 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
CRDT data types
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Free-text character CRDT 28 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| TextCrdt delta sync 29 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| CrdtTree lossless document 30 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Move-aware sequence CRDT 31 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Registers (LWW/MV) + PnCounter 32 | ✅ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ✅ | — |
Lossless tree
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Lossless tree CRDT core 33 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Lossless tree — anti-entropy 34 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Lossless tree — merge convergence 35 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Egress
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reactive egress 36 | ✅ | — | — | — | — | — | — | — | — | — |
| Egress — thread-safe 37 | ✅ | — | — | — | — | — | — | — | — | — |
| Egress — async 38 | ✅ | — | — | — | — | — | — | — | — | — |
| RelayCell 39 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Ingress
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reactive ingress 40 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Ingress — thread-safe 41 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Ingress — async 42 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Boundary-ingress adapter 43 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Wire codec
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| IPC wire — Snapshot/Delta/CrdtSync 44 | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | ✅ | — |
| Frame codec — json 45 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Frame codec — msgpack 46 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Frame codec — postcard (opt) 47 | ✅ | — | — | — | — | — | — | — | — | — |
| NodeId/PeerId exact-representation 48 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| NodeKey null-leniency 49 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Capability negotiation 50 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Transport & FFI
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Shared-memory blob path 51 | ✅ | ✅ | ✅ | ~ | ~ | ✅ | ✅ | ~ | ✅ | — |
| Cross-process zero-copy transport 52 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| C-ABI FFI boundary 53 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Message passing
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Message-passing + RPC command plane 54 | ✅ | ✅ | ✅ | ✅ | ✅ | ~ | ✅ | ✅ | ✅ | — |
Reliable sync
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Reliable sync 55 | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | ~ | — |
| Storage-independent durable outbox 56 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Reliable-sync transport seam 57 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Distributed plane
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Distributed CRDT plane 58 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Distributed plane — WebRTC 59 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| State projection / mirror 60 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Causal receipts
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Causal receipts 61 | ~ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Security boundary
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Permission boundary 62 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Membership
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Membership + failure detection 63 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Coordination
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Distributed coordination 64 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Presence
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Presence + ephemeral plane 65 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Temporal
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Temporal sources 66 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Rate shaping
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Rate-shaping operators 67 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Windowing
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Stream windowing 68 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Resilience
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Fault tolerance 69 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Portable stdlib
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Portable stdlib Timer 70 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Portable stdlib Timeout 71 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
| Portable stdlib RevisionBarrier 72 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Service plane
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Embedded-service plane 73 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Instrumentation
| Feature | Rust | Python | Kotlin | JS | Dart | Zig | Go | C++ | C# | GDScript |
|---|---|---|---|---|---|---|---|---|---|---|
| Instrumentation / benchmarks 74 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — |
Convergence and the wire protocol are pinned by the shared conformance fixtures and JSON Schemas in this repo and the Lean models in lazily-formal.
Notes
- ᵃ Python reactive graph:
Cell/Slot/Signal/Effect(sync) and the top-levelbatch(run)boundary ship; the!=PartialEq memo guard applies to cells, slots, and signals. The async counterpart (AsyncEffect) queues reruns at the batch boundary forasyncioreactors. - ᵇ Dart reactive graph:
Context/Slot/Cell/Signalship; there is no standaloneEffecttype (observers subscribe on cells) andbatchis scoped to the async context. - ᶜ Zig reactive graph:
Cell/Slot/Signal/Effectand the publicbatch(run)boundary ship (context.zigcoalesces the eager-recompute drain at the outermost batch exit). - ᵈ Serialized context (JS / Dart) — decision: both are meaningful as
serialized, realm-local execution flavors, not asshared-graphcontexts. They are scored when their own surfaces replay the portable Core fixtures, and are excluded from cross-thread shared-graph stress/model-check tests. A duplicate wrapper is not sufficient by itself: unsupported Core features remain staged, and runner-local locks/dictionaries cannot stand in for them. Either binding may remove the flavor and declare it absent; while it advertises the flavor, it must join every feature peer group it actually supports. See protocol.md § Concurrency layers are required for the conditional layer requirement and reactive-graph.md § Declared context capabilities for theserialized/shared-graphdistinction. - ᵉ Zig async context: Zig removed language
asyncand has no suspendable executor, so the layer is a task-queue +settle()drain surface — the synchronous graph’spending_recompute/drainPendingRecomputegeneralized with revision tracking and the 4-state slot machine (async_context.zig). - ᶠ Zig collections:
SourceMap/ComputedMapwith atomic move,SourceTree(per-level membership/order reactivity, atomic child move), and the LIS-move-minimized reconcile op-set all ship (collection.zig,cell_tree.zig,reconcile.zig). - ᵍ Shared-memory blob path (JS / Dart): carry
ShmBlobRefwire references but no host-sideShmBlobArena— the I/O-channel fallback of the shared-memory carve-out (see protocol.md § Shared-memory payload path is required). - ʰ Dart distributed CRDT plane: the
CrdtPlaneengine (HLC / stamp frontier / stability watermark) ships, but is not yet wired to livemerge: crdtroot cells. - ⁱ C-ABI FFI (JS): platform carve-out
ffi = none— browser/Worker JS has no shared in-process C ABI. The full state plane (includingCrdtSync) still flows over IPC / WebSocket / WebRTC (see protocol.md § C-ABI FFI is required). - Distributed plane — WebRTC transport + signaling (Rust / Kotlin / JS / Zig): the
portable stack (signaling protocol + client, the
DataChannelseam, permission- filtering sink/source, in-memory loopback, and the CRDT plane runtime) ships and is conformance-tested; the concrete native WebRTC backend is a platform adapter (str0m in Rust; the browserRTCPeerConnectionin JS; a consumer-provided seam in Kotlin and Zig), matching the reference design where the heavy transport is optional behind the seam.
-
Reactive graph — two cell kinds (nodes
SourceCell/ComputedCell; handlesSource<T, M>/Computed<T>) +Effectsink + eagerComputed(computed().eager()) / all cells guarded / batch ↩ -
Thread-safe context (lock-backed) ↩
-
Async reactive context ↩
-
Merge algebra +
Source<T, M>— associativeMergePolicy(KeepLatest/Sum/Max/SetUnion/RawFifo),Cell ≡ Source<KeepLatest>, read-any-cell/write-Sourcesplit (#relaycell) ↩ -
Keyed-map materialization (
ComputedMap) — mint-on-access derived slots: transparency + deferral (#lzmatmode) ↩ -
Thread-safe keyed map (
ThreadSafeComputedMap) —Send + Sync+ materialization confluence (#lzmatmode) ↩ -
Async keyed map (
AsyncComputedMap) — eventual transparency (#lzmatmode) ↩ -
Keyed-map sync — membership propagation + materialize-on-ingest + derived-aggregate transparency (
#lzfamilysync) ↩ -
Flat state machine ↩
-
Harel state charts ↩
-
Keyed reactive maps (
ReactiveMap:SourceMap/ComputedMap) +SourceTree+ reconcile ↩ -
ReactiveMapCore surface — single-threaded flavor (cell-model.md § Core surface vs. binding extensions) ↩ -
ReactiveMapCore surface — thread-safe flavor (ordering + membership reactivity) ↩ -
ReactiveMapCore surface — async flavor (ordering + membership reactivity) ↩ -
Exact-key dependency availability (
DependencyMap: observe before publish, unrelated-key isolation, stable identity;#lzdependencyavailability) ↩ -
Atomic ordered move replayed against all three flavors (
cellmap_atomic_move+cellmap_independence) ↩ -
Memoized semantic tree (
SemTree) ↩ -
Stable-id alignment (manufactured identity) ↩
-
Reactive queue (
QueueCellSPSC/MPSC +QueueStorageadapter) Core surface — single-threaded flavor ↩ -
Reactive queue (
QueueCellSPSC/MPSC +QueueStorageadapter) Core surface — thread-safe flavor (reader kinds + closure lifecycle) ↩ -
Reactive queue (
QueueCellSPSC/MPSC +QueueStorageadapter) Core surface — async flavor (reader kinds + eventual transparency) ↩ -
Broadcast topic (
TopicCell) Core surface — single-threaded flavor — independent cursors + durable replay + safe GC (#lztopiccell) ↩ -
Broadcast topic (
TopicCell) Core surface — thread-safe flavor (reader kinds + closure lifecycle) ↩ -
Broadcast topic (
TopicCell) Core surface — async flavor (reader kinds + eventual transparency) ↩ -
Competing-consumer work queue (
WorkQueueCell) Core surface — single-threaded flavor — exclusive leases + ack/nack + redelivery + DLQ (#lzworkqueue) ↩ -
Competing-consumer work queue (
WorkQueueCell) Core surface — thread-safe flavor (reader kinds + closure lifecycle) ↩ -
Competing-consumer work queue (
WorkQueueCell) Core surface — async flavor (reader kinds + eventual transparency) ↩ -
Free-text character CRDT (
TextCrdt) ↩ -
TextCrdtdelta sync (version_vector/delta_since/apply_delta) ↩ -
CrdtTreelossless document contract (#lzcrdttree) ↩ -
Move-aware sequence CRDT (
SeqCrdt) ↩ -
Registers (LWW / MV) +
PnCounter+CellCrdt↩ -
Lossless tree CRDT core (
LosslessTreeCrdt, M1) ↩ -
Lossless tree — dotted-frontier anti-entropy ↩
-
Lossless tree — concurrent merge convergence ↩
-
Transport-agnostic reactive egress (
EgressCore) — monotone sequence assignment, bounded unacknowledged window, cumulative ACK watermark, bounded retry/backoff/exhaustion, producer-generation fence (#lzegress) ↩ -
Egress family —
Send + Syncflavor (ThreadSafeEgressCell): delivery authority stays in the shared core, one attached transport Effect per incarnation (#lzegress) ↩ -
Egress family — async flavor (
AsyncEgressCell): the delivery-state readers stay synchronous Computeds; only the transport attachment is async-coloured (#lzegress) ↩ -
RelayCell — conflating relay +
BackpressurePolicy+SpillStore+Transport+ Inbox/Outbox + Rate/Window/Expiry/Priority/keyed policies (#relaycell) ↩ -
Transport-agnostic reactive ingress (
IngressCell) — keyed lifecycle scopes, generation/sequence/freshness envelopes, reorder buffer, accepted/dropped/error receipt readers (#designimplementtransport) ↩ -
Ingress family —
Send + Syncflavor (ThreadSafeIngressCell): one frontier walk per admission (#designimplementtransport) ↩ -
Ingress family — async flavor (
AsyncIngressCell): admission is not async-coloured (#designimplementtransport) ↩ -
Boundary-ingress adapter (
BoundaryIngressAdapter) — the non-reactive/reactive seam ahead ofIngressCell: one monotone channel cursor per producer generation, snapshot-plus-event bootstrap applied as one batch, derivedReplayRequired(from)on a cursor gap, generation fence + hot resubscribe (#designimplementtransport) ↩ -
IPC wire —
Snapshot+Delta+CrdtSync↩ -
Frame codec —
jsonreference codec: dependency-free interop floor, FFI baseline form, byte-canonical (MUST) — executable round-trip obligation (conformance/codec/frame_roundtrip_json.json,#lzmsgpackparity) ↩ -
Frame codec —
msgpackcross-language binary default: externally-tagged frame over named-field maps, semantic (not byte-identical) round-trip (MUST) — executable round-trip obligation (conformance/codec/frame_roundtrip_msgpack.json,#lzmsgpackparity). Shipping a MessagePack codec does not earn this mark: lazily-cpp read~here while its private internally-tagged framing wore the token, and only flipped once it shipped the spec wire (#lzcppmsgpackwire) ↩ -
Frame codec —
postcardpositional same-schema fast path: smallest + byte-canonical, not cross-language (MAY) ↩ -
NodeId/PeerIdexact-representation bound (MUST) — a decoder that cannot represent a received identifier exactly rejects the frame rather than rounding it (conformance/codec/nodeid_exact_range.json,#lzspecdecoderbound). A binding’s exact range MAY be narrower than theu64wire type; ✅ means it refuses outside that range instead of substituting a neighbouring id, not that it carries the fullu64. Exact ranges: fullu64in Rust / Zig / C#, unbounded in Python,[0, 2^63)in Kotlin / Go / C++,[0, 2^53)in JS, and platform-split in Dart (63-bit on the VM, 53-bit on web). protocol.md stated only the PRODUCER half until this audit, and two C++ decoders were substituting rather than refusing. ↩ -
NodeKeynull-leniency on decode (MUST) — omit-when-absent binds the ENCODER; a decoder reads both an omittedkeyand an explicitkey: nullas absent, refusing neither and constructing a key from neither (conformance/codec/nodekey_null_leniency.json,#lzkeynullstrict). Replayed on BOTH optional-key sites (NodeSnapshot, theNodeAdddelta op) in both codecs, and the fixture pins the RE-ENCODED field set as well: reading null as absent and writing it back out is a correct decode with a non-conforming encoder. Before the audit lazily-py and lazily-zig refused the null form, and lazily-kt decoded it into a real key namednull— all three had the same field right onCrdtOp, in the same file. ↩ -
Capability negotiation (
SessionHandshake) ↩ -
Cross-process zero-copy transport (
BlobBackend/ shm / arrow) ↩ -
C-ABI FFI boundary ↩
-
Message-passing + RPC command plane (
command-plane-v1) ↩ -
Reliable sync — resync coordinator + at-least-once durable outbox + OR-set/LWW liveness (
#lzsync) ↩ -
Storage-independent durable outbox (
OutboxStore+ shared outbox protocol; SQLite/Room/IndexedDB/file adapters) ↩ -
Reliable-sync transport seam + full-duplex
SyncDriverloop (IpcSink/IpcSource,#sync-driver) ↩ -
Distributed CRDT plane (
CrdtPlaneRuntime/ anti-entropy) ↩ -
Distributed plane — WebRTC transport + signaling ↩
-
State projection / mirror ↩
-
Causal receipts (
CausalReceiptsoutcome projection) ↩ -
Permission boundary (
PeerPermissions/RemoteOp) ↩ -
Membership + failure detection —
MembershipCell(SWIM + Phi-accrual) /PeerSet/PeerChangeEvent(#lzmemb) ↩ -
Distributed coordination —
LeaseCell/LeaderCell/LockCell/SemaphoreCell/BarrierCell+QuorumCell(#lzcoord) ↩ -
Presence + ephemeral plane —
PresenceCell/AwarenessCell/EphemeralCell+Ephemeral/Durablemarkers (#lzpresence) ↩ -
Temporal sources —
TimerCell/IntervalCell/CronCell/DeadlineCellover a logical clock (#lztime) ↩ -
Rate-shaping operators —
DebounceCell/ThrottleCell/SampleCell/ProbabilisticSampleCell(#lzrateshape) ↩ -
Stream windowing —
TumblingWindow/SlidingWindow/SessionWindowover the merge algebra (#lzwindow) ↩ -
Fault tolerance —
CircuitBreakerCell/RetryPolicyCell/BulkheadCell/TimeoutCell(#lzresilience) ↩ -
Portable stdlib
Timer(stdlib_timer_v1) — canonical fixture + mutation-gate verified ↩ -
Portable stdlib caller-driven
Timeout<T>(stdlib_timeout_v1) — distinct from reactiveTimeoutCell↩ -
Portable stdlib
RevisionBarrier(stdlib_revision_barrier_v1) — register/recheck lost-wakeup guard ↩ -
Embedded-service plane —
HealthCell/ReadinessCell/DiscoveryCell/ServiceRegistry(#lzservice) ↩ -
Instrumentation / benchmarks ↩
lazily Wire Protocol Specification
Normative source of truth for all lazily language bindings.
Shared Types
NodeId / PeerId
NodeId = u64
PeerId = u64
Wire-stable identifiers decoupled from internal SlotId. Serialized as bare JSON numbers.
Producer bound. A peer whose runtime represents integers as IEEE-754 doubles MUST keep emitted
NodeId/PeerId values at or below 2^53 − 1 (Number.MAX_SAFE_INTEGER). The constraint is the
runtime’s numeric representation, not the source language: it binds every JavaScript target —
which includes Dart, Kotlin, and TypeScript compiled to JavaScript — and does not bind a Dart
or Kotlin peer running on the VM/JVM. Earlier wording named “JavaScript/TypeScript peers”, which
read as a source-language rule and left compiled-to-JS peers apparently exempt.
Decoder obligation — refuse, never round (#lzspecdecoderbound). A decoder that cannot
represent a received NodeId/PeerId exactly MUST reject the frame. It MUST NOT round,
truncate, saturate, wrap, or otherwise substitute a nearby representable value. Rounding is the
worst available behaviour: the frame decodes cleanly, the identifier now addresses a different
node, and nothing downstream can detect the substitution — whereas a rejected frame is a visible
protocol error a peer recovers from by resync (§ Resync / gap handling). The producer bound keeps
a conforming producer from ever provoking this; the decoder clause is what makes a non-conforming
producer, or a corrupted frame, fail loudly instead of silently. The bound stated only the producer
half until #lzspecdecoderbound, and the receiving half is exactly where bindings diverged.
A binding’s exact range MAY be narrower than u64. The wire type stays u64; a binding whose
identity type is narrower is interoperable over the sub-range it represents and satisfies the
clause by failing the decode outside it — it is not required to widen. Audited ranges:
| Binding | Identity type | Exact range | Outside the range |
|---|---|---|---|
| lazily-rs | u64 | full u64 | serde decode error |
| lazily-zig | u64 | full u64 | error.Overflow — std.json yields .number_string, parseInt(u64) refuses |
| lazily-cs | ulong | full u64 | FormatException from JsonElement.GetUInt64 |
| lazily-py | int | unbounded | n/a — a Python int has nothing to round to |
| lazily-go | int64 | [0, 2^63) | json: cannot unmarshal number … of type int64 |
| lazily-kt | Long | [0, 2^63) | NumberFormatException from JsonPrimitive.long |
| lazily-cpp | int64_t | [0, 2^63) | std::runtime_error from the json parser / msgpack reader |
| lazily-js | number | [0, 2^53) | TypeError — a Number.isSafeInteger guard in both codecs |
| lazily-dart | int | [0, 2^63) on the VM, [0, 2^53) on web | UnsupportedError (#lzdartintwidth) |
Only lazily-js and lazily-dart carried the rule deliberately; the other seven inherited it from a
parser that happens to fail closed, which is a property no test held in place — and two of them did
not hold it. lazily-cpp’s msgpack reader cast a uint 64 straight to int64_t, so u64::MAX
decoded as -1: a well-formed identifier for a different node, with no error anywhere. Its json
parser refused the same value, but by letting std::stoll throw std::out_of_range, which is a
std::logic_error and therefore escaped the std::runtime_error every caller guards a decode
with. Both are fixed; the fixture is what found them.
conformance/codec/nodeid_exact_range.json
makes the clause executable in both json and msgpack form. Its wire frames are carried as
raw text (json) and hex bytes (msgpack), and the expected identifier as a decimal
string, because a runner that parsed the fixture as JSON on a double runtime would round the
expected value before it ever compared anything. The 2^53 − 1 scenarios are exact — every
binding MUST decode them to that value, which is what stops a runner that rejects everything from
passing vacuously; 2^53 + 1 and u64::MAX are exact_or_reject. A decoder that yields a
different number fails.
NodeKey
NodeKey = string // a "/"-joined path, e.g. "scores/alice", "outer/k1/inner/k2"
An optional, wire-stable keyed address for a collection entry (a SourceMap / ComputedMap entry). Unlike NodeId — the volatile internal handle a producer may re-mint after a resync or remove-then-readd — a NodeKey is producer-defined and stable across NodeId churn, so a peer can subscribe to “entry scores/alice” without an out-of-band key→NodeId map. A multi-segment path addresses nested collections (an entry of a SourceMap inside a SourceMap entry) with no extra machinery.
NodeKey is additive — it never changes NodeId semantics. It appears only as the optional key field on NodeSnapshot and the NodeAdd delta op.
Bounds (reject on construction and on the wire): path ≤ 1024 bytes; ≤ 32 /-separated segments; empty path and empty segments (leading/trailing/double /) are rejected.
Serialization is format-aware. Self-describing codecs (JSON, MessagePack) omit the key field when absent, so pre-key encoders/decoders and existing conformance fixtures round-trip unchanged; positional Postcard always carries the optional discriminant for binary schema stability. Cross-language implementations (lazily-py, lazily-zig, lazily-js, lazily-kt, lazily-go) add the optional nullable key field; they need not emit it when no key is set. Multi-producer key uniqueness (last-writer rule) is owned by the distributed CRDT plane, not this protocol.
Omit-when-absent binds the ENCODER; the decoder is lenient (#lzkeynullstrict). A conforming
encoder omits key when there is no key. A decoder MUST accept both the omitted field and an
explicit key: null, and read both as absent — it MUST NOT refuse the null form, and MUST NOT
construct a key from it. This was previously stated only as “a decoder that sees no key field
treats it as absent”, which settled the omitted form and left the null form undefined; three
bindings then diverged. The null form is not hypothetical: a serde-based peer that simply did not
apply skip_serializing_if emits it, rmp_serde’s own decoder reads it back as absent, and a
decoder that refuses is stricter than the reference implementation on a frame the reference
implementation produces.
Note the asymmetry that makes this easy to get wrong. CrdtOp.key is always written, null
when unset, because an anti-entropy op’s addressing is part of its merge identity (§ Anti-entropy
wire format) — so every decoder is already obliged to read key: null as absent one field over.
Every binding that got NodeSnapshot/NodeAdd wrong had CrdtOp right, in the same file.
Audited (#lzkeynullstrict) — the field appears on NodeSnapshot and on the NodeAdd delta op:
| Binding | Before the audit |
|---|---|
| lazily-rs, lazily-go, lazily-js, lazily-dart, lazily-cs, lazily-cpp | already lenient |
| lazily-py | refused — "key" in d was true for the null form, so None reached NodeKey.from_wire and raised |
| lazily-zig | refused — error.ExpectedString on the JSON null |
| lazily-kt | silently wrong — JsonNull is a JsonPrimitive and its content is the string "null", so the node decoded with a real wire-stable key literally named null, which then re-encoded as "key": "null" |
All three are fixed. lazily-kt’s is the reason the clause says MUST NOT construct a key from it rather than only MUST NOT refuse: a decoder that invents an entry address is worse than one that fails loudly, and a refusal-only rule would have called it conforming.
conformance/codec/nodekey_null_leniency.json
replays the rule in both codecs and on both fields. It pins the decoded key as absent and
pins the re-encoded frame: a decoder may read the null form, but an encoder must still emit only
the omitted one, so a binding cannot satisfy the clause by round-tripping null straight through.
IpcValue (payload)
A DeltaOp cell payload is carried as an externally-tagged IpcValue:
IpcValue = { "Inline": [u8] } // inline byte array (JSON array of 0..255)
| { "SharedBlob": ShmBlobRef } // descriptor into a shared-memory arena
NodeState
A NodeSnapshot / NodeAdd node body is carried as an externally-tagged NodeState:
NodeState = { "Payload": [u8] } // concrete serialized value bytes
| { "SharedBlob": ShmBlobRef } // concrete value in shared memory
| "Opaque" // visible node whose value cannot be serialized
Opaque serialized value bytes are owned by the producing language; type-aware decoding is fixed by the stable type_tag carried on the node. Over the JSON codec, bytes are transmitted as JSON arrays of integers in 0..255 (not base64). Under the negotiated json-base64 feature (#lzspecbase64, see § Capability Negotiation), a peer MAY instead encode Inline/Payload byte arrays as a base64 string, cutting ~4× wire bloat and ~3× parse cost; json-u8 stays the canonical fixture form.
type_tag
Each serializable node carries a type_tag: &'static str — a stable cross-process key that maps to a language-local deserialization constructor. The type-tag registry is per-implementation; tags must not collide across nodes.
Batch string-intern table (#lzspecintern)
A Snapshot, Delta, or CrdtSync batch that repeats the same type_tag string or the same NodeKey prefix across many nodes pays a per-node string cost that dominates wire size at scale. A batch MAY carry an optional sidecar intern table that assigns small integer ids to repeated strings; nodes then reference the id instead of inlining the string.
InternTable = { strings: [string] } # index → string; position is the id
- An empty or absent
internfield is the default and means “all strings are inlined” — existing decoders are unaffected (additive, backward-compatible). - A sender populates
intern.stringswith the deduplicatedtype_tagvalues (and, opt-in, repeatedNodeKeynamespace prefixes) appearing in the batch, then writes the integer id in each node’s tagged field. The reference is scoped to the batch and does not persist across frames. - A receiver resolves ids against
intern.strings; an id with no entry is a decode error (fail closed). - The intern table is a pure wire optimization: the decoded
IpcMessageis identical whether or not interning was used. Conformance fixtures exercise both the interned and inlined forms.
IPC: Snapshot + Incremental Update Protocol
lazily-IPC transmits a reactive graph’s state to a remote observer and keeps it in sync as the graph mutates.
Two message kinds
- Snapshot — full graph state. Sent on connect and on resync.
- Delta — incremental change set. Sent once per outermost batch-flush invalidation pass.
Epoch / versioning
A context-level monotonic ipc_epoch: u64 advances once per outermost batch flush, not per write.
Snapshotcarriesepoch.- Each
Deltacarries{ base_epoch, epoch }withepoch >= base_epoch + 1. The common single-flush case isepoch == base_epoch + 1; a multi-epoch-span delta (epoch > base_epoch + 1) coalesces several accepted-event epochs into one op batch (see § Multi-epoch-span delta). The spanepoch - base_epochis the count of accepted (deduped) events this batch advances past — a re-emit that dedups to a no-op adds no span.epoch < base_epoch + 1(empty or backward) is never valid. - Deltas are contiguous by base, not strictly unit-stepped. A receiver detects gaps, reorders,
or sender restarts by checking
base_epoch == last_epoch: a delta whosebase_epoch != last_epochis a gap and triggers resync (§ Reliable Sync) regardless of its span.
Snapshot
{
"Snapshot": {
"epoch": 1,
"nodes": [
{ "node": 1, "type_tag": "i32", "state": { "Payload": [1, 2, 3, 4] } }
],
"edges": [
{ "dependent": 2, "dependency": 1 }
],
"roots": [1]
}
}
| Field | Type | Description |
|---|---|---|
epoch | u64 | Current IPC epoch |
nodes | NodeSnapshot[] | All serialized nodes |
edges | EdgeSnapshot[] | Dependency edges (dependent → dependency) |
roots | NodeId[] | Cell and source slot ids |
NodeSnapshot
{ "node": 1, "type_tag": "i32", "state": { "Payload": [1, 2, 3, 4] } }
| Field | Type | Description |
|---|---|---|
node | NodeId (u64) | Wire-stable node identifier |
type_tag | string | Stable cross-process type key for decoding state |
state | NodeState | {"Payload":[u8]} | {"SharedBlob":ShmBlobRef} | "Opaque" |
key | NodeKey? | Optional wire-stable keyed address; omitted in JSON/MessagePack when absent |
EdgeSnapshot
{ "dependent": 1, "dependency": 0 }
Delta
{
"Delta": {
"base_epoch": 40,
"epoch": 41,
"ops": [
{ "CellSet": { "node": 1, "payload": { "Inline": [10] } } },
{ "SlotValue": { "node": 2, "payload": { "Inline": [20] } } },
{ "Invalidate": { "node": 3 } },
{ "NodeAdd": { "node": 4, "type_tag": "u64", "state": { "Payload": [64] } } },
{ "NodeRemove": { "node": 5 } },
{ "EdgeAdd": { "dependent": 2, "dependency": 1 } },
{ "EdgeRemove": { "dependent": 3, "dependency": 1 } }
]
}
}
| Field | Type | Description |
|---|---|---|
base_epoch | u64 | Epoch this delta applies to (must equal the receiver’s last_epoch) |
epoch | u64 | New epoch, >= base_epoch + 1; epoch - base_epoch is the accepted-event span (usually 1, > 1 for a multi-epoch-span delta) |
ops | DeltaOp[] | Ordered operations; applied as an ordered fold, atomically advancing last_epoch from base_epoch to epoch |
DeltaOp variants
All DeltaOp, IpcValue, NodeState, and IpcMessage variants are externally tagged: a single-key JSON object whose key is the PascalCase variant name and whose value is the body (or a bare "Opaque" / unit string).
| Op | Body fields | Description |
|---|---|---|
CellSet | node, payload: IpcValue | Changed-value cell write (PartialEq-guarded) |
SlotValue | node, payload: IpcValue | A recompute published a new value |
Invalidate | node | Dirtied, not yet recomputed (lazy) |
NodeAdd | node, type_tag, state: NodeState, key: NodeKey? | New node (optional wire-stable key, omitted in JSON/MessagePack when absent) |
NodeRemove | node | Removed node (free-list reuse: Remove then Add) |
EdgeAdd | dependent, dependency | New dependency edge |
EdgeRemove | dependent, dependency | Removed dependency edge |
QueuePush | node, payload: IpcValue | Op-log: append to a QueueCell tail |
QueuePop | node | Op-log: remove a QueueCell head (no value; determined by replay) |
QueueClose | node | Op-log: close a QueueCell (idempotent, terminal) |
QueueCell op-log delta form (#queue-oplog)
A QueueCell reconciles by two complementary wire forms, chosen by plane:
- Snapshot plane → storage-snapshot form. The full queue state is the storage backend’s
snapshot (the reference
VecDequeStorageserializes as a FIFO-ordered JSON array — cell-model.md § Wire and snapshot shape). This is the state form. - Delta plane → op-log form. Incremental change is the ordered op-log of
QueuePush/QueuePop/QueueCloseshell ops above, carried in aDeltaexactly like any otherDeltaOp(samebase_epoch/epochspan, gap rule, andbatch = foldsemantics). This is the op-log form.
Why a queue needs the op-log form and cannot use state-supersede coalescing: a queue’s order and
multiplicity are its value, and the consumer’s pop position is receiver-side state a sender
snapshot cannot see — so collapsing the unacked suffix to a state snapshot would break exactly-once
FIFO consumption. Instead the op-log fuses a maximal run of same-direction QueuePush ops into
one multi-epoch batch Delta (order-preserving, lossless; a QueuePop/QueueClose is a fusion
boundary). Fusion bounds frame count, not memory — see § Backpressure & outbox coalescing.
SPSC crossing rule. In the default producer→consumer sync the producer’s op-log carries
QueuePush/QueueClose; the remote consumer owns consumption, so its QueuePop is local and does
not cross. Only a mirror sync (an authoritatively-replicated consumer) also carries QueuePop so
mirrors converge on the consumption position. A re-delivered queue op (base_epoch < last_epoch) is
Ignored by the ResyncCoordinator, so at-least-once replay neither drops nor double-applies a push
or pop.
Consistency invariants
- PartialEq cell guard: An equal
setemits noCellSetand no downstream ops. - Memo equality suppression: A dirty
memo()that recomputes to an equal value emits noSlotValueand no downstreamInvalidate. - Coalesced frontier: A dependent reached through many changed cells in one batch appears at most once per delta.
- Eager Signal values are concrete: A changed eager Signal emits a concrete
SlotValuefor its backing slot, not a bareInvalidate.
The companion Lean model in formal/lean encodes these IPC transition rules and
checks them with lake build.
Eager Signal nodes
A Signal is the eager derived value in the Slot -> Cell -> Signal family. It
is not a separate wire type. A Signal is represented by the ordinary backing
slot node that stores its materialized value:
- Snapshot: the backing slot appears as a
NodeSnapshotwith a concretepayload/shared-blob payload like any other readable slot. - Delta: a value change appears as
SlotValuefor the backing slot’sNodeId. Because the value is recomputed during the invalidation flush, eager Signals do not emit bareInvalidateops for their own changed value. - Memo guard: an eager recompute that yields an equal value suppresses
SlotValueand downstream invalidation exactly like a lazy memoized slot. - Local puller: the producer-side effect that keeps the Signal eager is local execution state and is not serialized as a graph node.
Consumers therefore need no protocol extension to read eager Signals from a
producer. They observe the same permission-filtered Snapshot/Delta state
plane and see Signals as slots whose changed values are reliably materialized.
Lazy reconciliation
- Value-mirror (default): At flush, the sender resolves each invalidated allowlisted slot so the delta carries concrete
SlotValues. The receiver holds no compute closures. - Mirror-lazy: The sender emits bare
Invalidate; the receiver keeps a stale marker. Requires compute closure replication. Deferred tolazily-distributed.
Wire shape. The value-mirror default means an allowlisted dirty slot appears in a flush
Deltaas a concreteSlotValue, never a bareInvalidate(the latter is the mirror-lazy form). This invariant — and the eager-Signal rule that a changed Signal publishes aSlotValuefor its backing slot, not anInvalidate— is pinned by the IPC fixturesdelta_sequential.jsonanddelta_shared_blob.json, both of which carrySlotValueops for resolved slots.
Resync / gap handling
On a Delta whose base_epoch != last_epoch:
- Receiver discards the delta.
- Receiver requests a fresh
Snapshot. - Sender replies with
Snapshot { epoch }. - Deltas resume from the new epoch.
This narrative is the informal shape; the normative decision function (inbound frame →
Apply / RequestSnapshot / Ignore) is the ResyncCoordinator state machine specified in
§ Reliable Sync, which also fixes the durable-outbox replay and
sync-driver loop that make gap recovery, reconnect backfill, and at-least-once delivery a
protocol rather than an each-consumer hand-roll.
Messages are length-prefixed and tagged Snapshot / Delta. The protocol is transport-agnostic (unix socket, pipe, WebSocket, shared memory).
Multi-epoch-span delta
A Delta MAY advance more than one epoch in a single frame: epoch > base_epoch + 1. This models
a producer whose epoch is a cumulative count of accepted (deduped) events and who coalesces
several such events into one flush (agent-doc’s WireDelta is exactly this: a delta “may span
multiple epochs”, epoch = per-document accepted-event count). The op list is still the ordered
change set; epoch - base_epoch records how many accepted events the batch folds.
Normative apply semantics:
- Batch = fold. Applying one
Delta { base_epoch, epoch, ops }MUST equal applying the sameopsin order as a run of unit deltas that advanceslast_epochfrombase_epochtoepoch. The receiver observes only the endpoints (base_epoch,epoch); intermediate epochs are not separately materialized. Proven equivalent inlazily-formal(ReliableSync.multi_epoch_apply_eq_fold). - Atomic advance. The receiver advances
last_epochtoepochonly after the whole op list applies; a partial application never leaveslast_epochat an intermediate value. - Gap rule unchanged. Acceptance still requires
base_epoch == last_epoch; the span does not relax gap detection. A delta withbase_epoch != last_epochis a gap at any span. - Idempotent re-emit adds no span. A re-emitted delta that dedups to no accepted change carries
epoch == base_epochworth of new effect and is either omitted or applied as a no-op; it never advanceslast_epoch.
Wire-compat: the unit-step case (epoch == base_epoch + 1) is the span-1 special case, so every
existing Delta fixture remains valid. Conformance:
conformance/reliable-sync/multi_epoch_delta.json.
Shared-memory IPC
ShmBlobArena provides the shared-memory payload path (a required layer wherever the
platform supports it — see § Shared-memory payload path is required):
- Arena writes a fixed header before each payload:
{ generation, epoch, length, checksum }. - Readers validate the header before accepting a descriptor.
IpcMessagecontrol frames carryShmBlobRefdescriptors instead of embedding large bytes inline.
Shared memory carries large blob payloads; ordinary control transport carries framed IpcMessages. Each process keeps its own local Context / ThreadSafeContext and reconciles via snapshots and deltas. A binding whose platform cannot host a shared-memory arena carries every large payload Inline over the control transport (IPC / WebSocket / WebRTC) — the I/O channel accesses the memory directly, so the peer still receives the bytes without a shared-memory descriptor.
Capability Negotiation
Each non-local session starts with a compatibility handshake:
{
"protocol_id": "lazily-ipc",
"protocol_major_version": 1,
"codec": "json",
"max_frame_size": 1048576,
"fragmentation_supported": false,
"ordered_reliable": true,
"peer_id": 1,
"session_id": "abc-123",
"features": ["shared-blob", "signaling-relay"]
}
| Field | Description |
|---|---|
protocol_id | Must be "lazily-ipc" |
protocol_major_version | Breaking change indicator |
codec | "json", "msgpack" (cross-language binary default), or "postcard" (Rust/same-schema fast path) — see § Frame codecs |
max_frame_size | Maximum unfragmented frame this endpoint can receive, in bytes |
fragmentation_supported | Whether this endpoint can send and reassemble fragmented frames |
ordered_reliable | Delivery guarantee requirement |
peer_id | PeerId for this session |
session_id | Shared non-empty session/graph identifier |
features | Supported feature flags |
max_frame_size is a positive receive ceiling, not an equality constraint. Peers with
different ceilings remain compatible and negotiate
min(local.max_frame_size, remote.max_frame_size). Both directions use that common
ceiling; a binding MUST retain the negotiated value rather than continuing with its local
advertisement. A zero ceiling is invalid and fails closed.
Fragmentation is available to the session only when both peers advertise
fragmentation_supported = true. A true/false pair remains compatible but negotiates
false; treating one peer’s support as sufficient would ask the other peer to reassemble a
frame it said it cannot accept.
session_id names the shared graph/session, while peer_id names one endpoint. The two
handshakes MUST carry the same non-empty session_id; their peer_id values may and normally
do differ.
If peers disagree on protocol_major_version, codec, ordered_reliable, session_id, or
required features, or if either max_frame_size is zero, they fail closed before applying any
Snapshot or Delta.
Feature flags. The features array advertises optional capabilities both peers must offer to use. Defined flags:
| Flag | Effect when both peers advertise it |
|---|---|
shared-blob | Large payloads travel as SharedBlob descriptors into a shared-memory arena (§ zero-copy transport). |
signaling-relay | A signaling relay may mediate peer discovery (§ signaling). |
command-plane-v1 | The command/RPC message plane is active (§ message-passing). |
json-base64 (#lzspecbase64) | Over the json codec, Inline/Payload byte arrays MAY be encoded as a base64 string instead of a JSON array of 0..255 integers (~4× smaller, ~3× faster to parse). The array form (json-u8) remains the canonical form for conformance fixtures; a decoder that advertises json-base64 MUST accept both forms. |
protobuf-graph-boundary-v1 | Peers may exchange the generated, versioned graph-boundary algebra described in Protobuf graph-boundary interoperability. |
protobuf-graph-boundary-v1 is an additive typed boundary family, not another
encoding of IpcMessage and not a second graph runtime. Its
ProtocolEnvelope carries stable identity, causality, bounded mutations,
projections, effect intents, and receipts. Canonical JSON remains the fixture
and diagnostic authority, and logical identity hashes canonical values rather
than Protobuf bytes.
Frame codecs
Every serialized IpcMessage frame (Snapshot, Delta, CrdtSync) carries the same
logical wire schema; the codec handshake field selects only how those fields are bytes-encoded.
Three codecs are defined; a binding advertises the set it can encode/decode and the peers
negotiate one.
Two senses of “canonical” — read this first. This spec keeps two terms distinct:
- Reference codec = a role: the required, dependency-free, human-inspectable encoding that every binding MUST speak, that the FFI baseline re-encodes to, and that conformance fixtures / logs / receipts are written in.
jsonis the reference codec. It is chosen for this role because it is universal (a parser in every stdlib), inspectable, and deterministic — not because it is efficient (it is the least efficient codec).- Canonical bytes = a property of one codec+message: the single deterministic byte string a given codec produces for a given
IpcMessage.jsonand positionalpostcardare byte-canonical (one byte form per message per codec).msgpacknamed-field maps are not byte-canonical across encoders (map key order is encoder-defined), so msgpack is the reference-compatible efficient transport, never the reference codec.These are orthogonal: “reference codec” answers which encoding is the required interop floor; “canonical bytes” answers does this codec produce one deterministic byte form.
| Codec token | Self-describing | Role | Required of a binding |
|---|---|---|---|
json | yes | The reference codec: the required, dependency-free, human-inspectable interop floor — what the FFI baseline re-encodes to (§ FFI Boundary) and what fixtures/logs/receipts are written in. Byte-canonical. Least efficient (blob bytes travel as arrays of integers 0..255). | MUST |
msgpack | yes | The negotiated cross-language binary default on any binary boundary (IPC / WebSocket / WebRTC) between differing languages. Compact binary, self-describing, evolution-safe — the portable efficient transport every binding can host. Reference-JSON-compatible field names, but not byte-canonical across encoders. | MUST |
postcard | no | A compact, positional Rust/same-schema fast path for two peers that share the exact Rust struct layout. Smallest and fastest on the wire, byte-canonical, but not cross-language. | MAY |
Default selection. A local, same-process, same-language pairing MAY default to postcard.
A boundary that crosses languages (the plugin⇄controller boundary, browser peers) negotiates
msgpack; json is the interoperable fallback and the required reference codec (also the FFI
baseline form). Peers never apply a frame under a codec they did not negotiate.
MessagePack encoding is named-field. msgpack frames encode each struct as a MessagePack
map keyed by the JSON field name (Rust: rmp_serde::to_vec_named), not as a positional array.
This keeps the field names identical to the json schema, so the omit-when-absent rule for
optional fields (the nullable key on NodeSnapshot/NodeAdd/CrdtOp, § NodeKey) holds
uniformly across json and msgpack, and a decoder that predates a later optional field ignores
it. Positional postcard instead always carries the optional discriminant for binary schema
stability.
Conformance is semantic round-trip, not byte-identical, for the map codecs. Because a
MessagePack map’s key order is encoder-defined, two conforming bindings MAY emit byte-different
msgpack frames for the same IpcMessage; conformance requires only that
decode(encode(m)) == m and that decoding a peer’s frame yields the equal IpcMessage. Only
postcard (positional) is byte-canonical across encoders. Cross-language msgpack fixtures
therefore pin the decoded value, never a golden byte string.
The codec requirement is executable, not prose (#lzmsgpackparity). Stating “every frame MUST
round-trip through both json and msgpack” in this document proved nothing: the conformance
ladder verifies fixture content replay (was the file opened, were its keys consumed, were they
asserted, were all its scenarios replayed), and content replay never exercises a codec. A binding
could therefore carve out a MUST-level codec and stay green everywhere. Two canonical fixtures now
carry the obligation:
| Fixture | Codec | Obligation |
|---|---|---|
conformance/codec/frame_roundtrip_json.json | json | MUST — the reference codec; every binding replays it |
conformance/codec/frame_roundtrip_msgpack.json | msgpack | MUST — a binding that has not implemented it declares the gap in its conformance-coverage ledger |
Both carry the same three wire values (one per IpcMessage variant) so one runner shape serves
both codecs. A runner MUST decode wire, re-encode the decoded message, decode it again, and
evaluate every expect key against that second decode — asserting against the fixture literal
proves nothing, because the literal never passed through the codec. The msgpack fixture
additionally pins encoded_envelope_key and the sorted encoded_body_field_names, which is
the executable form of the named-field rule above: a positional encoder round-trips a value
correctly and fails those keys. postcard carries no cross-language fixture — it is MAY, and by
construction only two peers sharing a struct layout can speak it.
A binding that does not implement msgpack is not silently exempt. Its
scripts/check-conformance-coverage.sh sees the canonical fixture and fails until the binding
either replays it or names it in KNOWN_UNCOVERED with a reason — the same ledger every other
declared gap lives in — and coverage.json carries one row per codec flavor so
the matrix shows the carve-out beside every other parity fact.
Shipping a MessagePack codec is not implementing msgpack. The codec token names one
wire, not a serialization technology: the externally-tagged frame ({"Snapshot": {…}}) over
named-field maps whose keys are the json field names, with the same omit-when-absent rule for
optional fields. A codec that packs the same data as an internally-tagged envelope
({"type": 0, "value": …}), gives NodeState/IpcValue integer kind discriminators instead
of the Payload/Inline external tags, or uses positional arrays, is a private codec that
happens to use MessagePack framing — a peer that negotiated msgpack with it would not decode
its frames. That distinction is invisible to a file-presence audit and is exactly what the
round-trip fixture makes checkable, which is why the matrix distinguishes ✅ from ~ on this
row.
Causal Receipts
Some integrations send a command or publish an effectful request and need a durable, queryable outcome for that causation id. lazily supplies a generic causal receipt primitive for that use case; it is not a transport ACK and does not make delivery success authoritative.
{
"CausalReceipts": {
"receipts": [
{
"receipt_id": "receipt-1",
"causation_id": "patch-123",
"observer": "editor",
"generation": 7,
"outcome": "applied",
"reason": null,
"payload_hash": "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
}
]
}
}
| Field | Description |
|---|---|
receipt_id | Idempotency key for this receipt event. Duplicate receipt_ids are no-ops. |
causation_id | Stable id of the command, event, or effect request the receipt observes. |
observer | Peer, process, or subsystem that produced the receipt. |
generation | Monotonic producer/editor generation. Consumers discard receipts whose generation does not match the current authority generation for the causation id. |
outcome | "observed", "accepted", "applied", or "rejected". |
reason | Optional human/debug rejection reason; null when absent. |
payload_hash | Optional hash of the state/payload the receipt observed; null when absent. |
Receipt projection rules:
observedandacceptedare non-terminal. They may model ACK-like transport or queue admission observations, but a domain MUST NOT treat them as proof that an effect happened.appliedandrejectedare terminal. They are the generic outcome vocabulary that domain-specific facts refine (for example an editor may publishEditorPatchApplied/EditorPatchRejectedfacts keyed by the samecausation_id).- A stale-generation receipt is ignored by the current projection and may be retained only as audit/debug data.
- A second terminal receipt for the same
causation_idand generation with a different terminal outcome is a terminal conflict; consumers fail closed instead of selecting a winner. - The primitive is state/projection data. Transports may still provide their own delivery acknowledgements internally, but lazily does not expose delivery ACKs as authority.
Command / RPC Message Plane
Editor and runtime integrations issue commands — Run Agent Doc, sync,
focus, save, session operations — and need one reusable admission, dedupe,
cancellation, generation-guard, progress, and reconnect story instead of a
per-caller ad hoc request/response contract. lazily supplies an evented command
message plane for that use case. It is an additive sibling family to
Snapshot / Delta / CrdtSync; it does not add new state-plane variants.
The plane is feature-gated. Peers advertise command-plane-v1 in the
Capability Negotiation features array. A peer that
lacks command-plane-v1 fails closed before accepting command traffic; a
command that requires the plane is not silently downgraded.
Four externally-tagged frames make up the family:
CommandSubmit
{
"CommandSubmit": {
"command_id": "cmd-run-1",
"causation_id": "cmd-run-1",
"source": "vscode-plugin",
"target": "project-controller",
"namespace": "agent-doc",
"name": "editor_route",
"authority_generation": 42,
"idempotency_key": "project-root:plan.md:run",
"deadline_ms": 120000,
"policy": { "dedupe": "same_idempotency_key", "supersede": false, "cancel_on_preempt": true },
"payload_type": "agent-doc.editor_route.v1",
"payload_hash": "sha256:…",
"payload": { "Inline": [123, 34, 102, 34, 58, 34, 46, 34, 125] },
"required_features": ["causal-receipts", "command-events"]
}
}
lazily owns the envelope (command_id, correlation, idempotency, generation,
policy, payload framing). The namespace owns the payload: lazily never
interprets the payload body, which is a normal IpcValue
(inline bytes or a shared-memory blob). payload_type and payload_hash
identify and pin the domain body.
CommandCancel
CommandCancel preempts a still-non-terminal command by command_id at a given
authority_generation, with an optional reason. A stale-generation cancel is
ignored. A cancel after a terminal outcome never rewrites it.
CommandEvents
CommandEvents batches progress/detail events keyed by command_id. Event
kinds are observed, accepted, started, progress, cancelled,
superseded, timed_out. Events are UX and diagnostics only — queue
position, retry advice, copied CLI output. They are never terminal proof. Even
cancelled / superseded / timed_out events are surfaced for UX; their
terminal authority is a matching rejected causal receipt.
CommandProjection
CommandProjection is the folded, queryable image of known command state:
per-command status, an explicit terminal flag, generation, terminal
reason, and the terminal_receipt_id / last_event_id that produced it. It is
also the reconnect resync frame: after a controller handoff/recycle a plugin
folds a fresh CommandProjection and recovers in-flight and terminal state
without replaying the underlying events.
Projection rules
- Terminal authority is the causal receipt, not the event or the transport.
A command becomes terminal (
applied/rejected/cancelled/superseded/timed_out) only when a terminalCausalReceiptfor itscommand_idfolds in.observed/accepted/started/ queued admission are non-terminal progress. A network ACK is never terminal. - Generation guards. Events and receipts whose
generationdoes not match the command’s current authority generation are ignored by the projection and retained only as audit data. - Idempotency. A replayed
CommandSubmit, event, or receipt (samecommand_id/event_id/receipt_id) is a no-op; the projection is unchanged. - Cancel before terminal only. A cancel terminally rejects a non-terminal
command; a cancel after
appliedis ignored. - Terminal conflict fails closed. Two terminal receipts at the same generation with different outcomes is a conflict; consumers fail closed rather than pick a winner (the same rule as Causal Receipts).
- Reconnect equivalence. Folding a
CommandProjectionimage is equivalent to folding the events and receipts it summarizes.
RPC facade
Bindings expose an RPC-style API (call / submit / cancel / observe /
projection) implemented entirely over these frames:
callbuilds and sends aCommandSubmit, observes events and receipts, and resolves only when the command projection reaches a terminal causal receipt. A transport ACK, controller admission, oraccepted/ queued event never resolves a unarycall.submitreturns thecommand_idimmediately for callers that manage events and projection themselves.cancelsendsCommandCanceland returns the resulting projection.observe/streamexposesCommandEventsand projection updates for UI progress.- Reconnect uses
CommandProjection; callers replay acallonly when the idempotency policy says replay is safe. - The terminal-result error shape exposes the terminal command projection
(
status,reason), not a collapsed boolean.
Unary RPC completion means command effect completion, proven by a terminal
causal receipt — not network delivery. The schema is
schemas/message-passing.json; fixtures live
in conformance/message-passing/.
Cross-language Channel Compatibility
All channels carry the same IpcMessage state plane:
| Channel | Strategy |
|---|---|
| FFI | C ABI: opaque context/session handles + owned byte buffers |
| IPC | Unix sockets, pipes, local TCP: length-prefixed serialized IpcMessage |
| WebSocket | One WebSocket frame = one serialized IpcMessage |
| WebRTC data | Reliable ordered data channels carry serialized IpcMessage |
Cross-language rules
- Compute closures are language-local. Cross-language sync shares the cell state plane.
- Permission filtering happens before serialization on every channel.
- Back-pressure: if frames gap, reorder, or truncate, the receiver requests a fresh
Snapshot.
Binding Conformance Matrix
A lazily binding is a language port that intends to interoperate with the wider lazily
ecosystem (lazily-rs, lazily-py, lazily-zig, lazily-js, lazily-kt, lazily-dart, lazily-go, …). The
layers below are required of every binding; none is an optional lazily-rs extension.
A binding that omits a MUST row is non-conforming and MUST advertise its missing surface
via Capability Negotiation rather than failing silently.
| Layer | Required | Spec | Conformance |
|---|---|---|---|
| Reactive core (Cell / Slot / Effect / Signal) | MUST | Reactive Graph, Cell Model | — |
Keyed cell collections (SourceMap, SourceTree, keyed reconciliation) | MUST | Cell Model § Keyed cell collections | conformance/collections/ |
| Flat state machine | MUST | State Machine | — |
| Harel state charts | MUST | State Charts | conformance/statechart/ |
| Thread-safe reactive context | MUST² | Reactive Graph § Context layers | — |
| Async reactive context | MUST² | Async Reactive Context | — |
IPC (Snapshot + Delta) | MUST | § IPC | conformance/ IPC fixtures |
Frame codecs (json reference + msgpack cross-language binary; postcard optional) | MUST | § Frame codecs | conformance/codec/frame_roundtrip_json.json and conformance/codec/frame_roundtrip_msgpack.json — each carries one scenario per IpcMessage variant (Snapshot/Delta/CrdtSync), replayed through the codec rather than read as data |
Shared-memory payload path (ShmBlobArena / ShmBlobRef) | MUST³ | § Shared-memory IPC | conformance/ shared-blob fixtures (snapshot_shared_blob, delta_shared_blob) |
C-ABI FFI boundary (LazilyFfiBytes, LazilyFfiStatus, LazilyFfiMessageKind) | MUST¹ | § FFI Boundary, ffi.json | every binding decodes the FFI frame to IpcMessage and re-encodes canonical JSON bytes |
Distributed CRDT plane (CrdtSync / WireStamp) | MUST | § Distributed: CRDT Cell Plane, distributed.json | conformance/ CrdtSync round-trip |
Causal receipts (CausalReceipt, terminal outcome projection) | MUST | § Causal Receipts, receipts.json | conformance/receipts/causal_receipts.json |
Permission boundary (RemoteOp / PeerPermissions) | MUST | § Permission Boundary | — |
| Capability negotiation | MUST | § Capability Negotiation | — |
| Signaling (WebSocket) | MAY | § Signaling | only for bindings that bridge browser/runtime peers |
| WebRTC data transport | MAY | § Cross-language channels | only for bindings that peer over WebRTC |
¹ C-ABI FFI has a platform carve-out — see § C-ABI FFI is required. ² The thread-safe and async context layers have a platform carve-out — see § Concurrency layers are required. ³ The shared-memory payload path has a platform carve-out (I/O-channel fallback) — see § Shared-memory payload path is required. CRDT and the keyed cell collections have no carve-out: they are wire/logic properties implementable on any runtime that speaks the wire.
C-ABI FFI is required
Every binding whose platform can host a native in-process boundary MUST expose and consume
the C-ABI FFI boundary: the LazilyFfiBytes / LazilyFfiStatus /
LazilyFfiMessageKind contract with explicit allocation ownership, panics caught before
crossing the C ABI, and a channel that decodes each accepted frame as IpcMessage and
re-encodes canonical JSON bytes. The FFI message kind discriminant MUST include
CrdtSync = 3 and the reliable-sync control frames ResyncRequest = 4 / OutboxAck = 5
(§ Reliable Sync). This is the lingua franca that lets any binding embed
any other without a language-specific bridge.
Platform carve-out. A binding whose runtime structurally cannot host a native C ABI
declares the ffi capability as none; otherwise it declares host. none is reserved
for platforms with no shared in-process address space (for example browser/Worker JS, or a
fully-sandboxed runtime) — a binding MAY NOT declare none merely because FFI is
inconvenient or unimplemented. This is a binding-level conformance declaration (advertised
in the binding’s conformance statement and discoverable at build/link time), not a
per-session wire flag, since in-process embedding is not a runtime session.
A ffi = none binding:
- conforms to the interop contract but not the in-process embedding contract, and MUST NOT advertise itself as embeddable;
- MUST still expose the full state plane — including
CrdtSync— over IPC/WebSocket/WebRTC, so it interoperate without in-process embedding; and - MUST be treated by peers/host tooling as unable to be loaded in-process (fail closed on any attempt to do so, rather than silently degrading).
This reuses the existing fail-closed principle: the limitation is explicit and advertised, never silent. There is no equivalent carve-out for CRDT or the cell/collections model — those are implementable on any Turing-complete runtime that speaks the wire.
CRDT is required
Every binding MUST implement the CrdtSync plane — the
merge: crdt mechanism, the WireStamp version-vector frontier, and the
causal-stability watermark / GC contract — and MUST round-trip a CrdtSync IpcMessage
byte-identically. Multi-write convergence is a property of the cell model, not an
optional distributed extra; a binding that ships only the single-producer
Snapshot/Delta mirror conforms only to the single-writer subset and MUST downgrade its
advertised capability accordingly.
Concurrency layers are required
The reactive core ships as three context layers — single-threaded (base), thread-safe (lock-backed), and async (future-returning) — defined in Reactive Graph § Context layers and Async Reactive Context. The single-threaded base context is unconditionally required of every binding (it is the reactive-core row above). The thread-safe and async layers are required conditionally: a binding whose platform structurally supports a layer MUST implement that layer.
Thread-safe context. A binding whose platform exposes preemptive multi-threading or
shared-memory concurrency (native threads, OS goroutines, JVM threads, Kotlin
coroutines over a shared heap, etc.) MUST ship the lock-backed context whose handles are
clonable and whose transition function and state are Send + Sync, so observers fire
synchronously within the invalidating send/batch preserving glitch-free pull-based
ordering. “Synchronously within” mandates glitch-free ordering, not literal in-lock
dispatch — a threaded binding MAY defer observer dispatch out of the graph lock provided the
ordering invariant holds (#lzspecobserverclarify). A platform with no shared-memory threading model — a strictly single-threaded
runtime, or a process/actor-isolation model (e.g. a Dart isolate, a browser Worker) where
peers do not share an address space — declares the thread_safe capability as none.
Async context. A binding whose platform exposes an async/future runtime (async/await,
promises, coroutines, an executor that suspends and resumes across .await points) MUST
ship the async context with its full slot state machine (Empty /
Computing / Resolved / Error), revision tracking, five-point cancellation contract,
and get_async re-resolve loop. A platform with no notion of suspendable async
computation declares the async capability as none.
A none declaration, for either layer:
- is a binding-level conformance declaration (advertised in the binding’s conformance statement and discoverable at build time), not a per-session wire flag, because in-process concurrency structure is not a runtime session;
- MUST be reserved for platforms that structurally lack the primitive — a binding MAY
NOT declare
nonemerely because the layer is inconvenient or unimplemented; and - MUST be advertised rather than fail silently, reusing the existing fail-closed principle.
There is no carve-out for the keyed cell collections, the state machine, the state charts, the reactive core, or CRDT — those are implementable on any Turing-complete runtime that speaks the wire, single-threaded or not.
Shared-memory payload path is required
The shared-memory payload path (ShmBlobArena + ShmBlobRef
descriptors) is the zero-copy large-payload transport for IpcValue and NodeState. A
binding whose platform exposes a shared-memory primitive (POSIX shm / memory-mapped
files / OS shared memory / a peer-reachable arena) MUST implement it: payloads above the
inline threshold are written into the arena, the control frame carries the ShmBlobRef
descriptor, and readers validate the { generation, epoch, length, checksum } header
before accepting it.
Platform carve-out — I/O-channel fallback. A binding whose runtime structurally cannot
host a shared-memory arena (browser/Worker JS, a sandboxed runtime, WASM without shared
memory) declares the shared_memory capability as none. Such a binding MUST fall back
to I/O channels accessing the memory: every payload that would have been a
SharedBlob descriptor is instead carried Inline over the ordinary IPC / WebSocket /
WebRTC transport, so the same IpcMessage state plane reaches the peer without a
shared-memory descriptor. The wire format already supports both paths (IpcValue and
NodeState are externally-tagged Inline | SharedBlob); the fallback simply never emits
the SharedBlob variant on that binding and treats an absent descriptor as Inline on
read.
Shared memory is negotiated per session via the shared-blob entry in the
Capability Negotiation features array: two peers that both
advertise shared-blob MAY exchange ShmBlobRef descriptors; if either peer omits it,
both sides carry payloads Inline over the control transport for that session. (The
thread_safe, async, and ffi capabilities, by contrast, are binding-level
declarations — in-process properties — and are not per-session wire flags.)
This reuses the existing fail-closed principle: the limitation is explicit and
advertised, never silent. The shared-blob conformance fixtures
(snapshot_shared_blob.json / delta_shared_blob.json)
fix the descriptor shape for bindings that ship the arena; an Inline-only binding
round-trips the same fixtures with the blob bytes inlined.
The backend discriminator: absence is lenient, an unknown value is not (#lzblobbackendstrict)
ShmBlobRef.backend selects which pluggable backend resolves a descriptor
(shm | arrow | in_process, optional, default shm — see
docs/zero-copy-transport.md). The two ways a decoder
can fail to recognise it are not the same fact and MUST get opposite answers:
- An OMITTED
backendMUST decode asshm. This is the forward-compatibility channel, and the only one. Every descriptor minted before the field existed has this shape, which is exactly why the field is optional. A conforming encoder MUST also omit it when the backend isshm, so a pre-field descriptor round-trips byte-identically — the same encoder/decoder split § NodeKey makes forkey. - A PRESENT
backendoutside the enum MUST be rejected, with the offending token named in the error. A decoder MUST NOT normalize it toshm, to any other backend, or to a sentinel. - An explicit
backend: nullis the ABSENT form, not a present-unknown one, and MUST decode asshm. This follows § NodeKey rather than the bullet above, and for the same reason: a serde-style peer that did not applyskip_serializing_ifto an optional field emitsnullwhere a conforming encoder omits, so refusing it is stricter than the reference implementation on a frame the reference implementation produces. Four bindings raised this edge independently while implementing the clause, and they had already split three ways on it — accept-as-shm,error.ExpectedStringnaming nothing, and a refusal naming the token''. shmis the permanent default. A future revision MUST NOT redefine which backend an omittedbackenddenotes. The omitted form is the only shape a pre-field descriptor can have, so changing its meaning would silently reinterpret every descriptor ever written rather than break them visibly — the one genuinely undecidable case on this wire, and cheaper to pin as a sentence than to discover as an incident.
Two obligations on the refusal itself.
It must be catchable through the codec’s documented decode-error type. A refusal
raised as a type outside the family every caller already guards a decode with is
invisible: the frame still fails, but it fails past the handler. In C++ that means
std::runtime_error and specifically not std::invalid_argument, which derives
from std::logic_error — the identical hierarchy trap that let a std::out_of_range
from std::stoll escape every decode guard until the NodeId bound was audited. The
general rule: a decoder’s refusals belong in one catchable family, and this clause
adds no exception to it.
Naming the token is bounded. A binding MAY truncate the offending token in its error (64 bytes is sufficient), and a truncated name satisfies the obligation. The refusal path is the path a hostile or corrupt producer controls, so an unbounded “echo the token back” requirement is an allocation and log-flood vector; a truncated name still identifies the producer, where an absent one does not.
The cost this accepts, stated plainly. A refusal fails the whole frame, not
just the descriptor. So a fourth backend added later is a hard break for every older
peer rather than a degraded read, and at the decoder a corrupt frame and a
future-version frame are indistinguishable — nothing on the frame carries a protocol
version, only the session-level shared-blob capability flag. That is the deliberate
trade: a visible break a peer recovers from by resync, in exchange for never
resolving a descriptor against a backend its producer did not name. The alternative
buys graceful degradation with a guarantee that holds only until a checksum collides.
The asymmetry is not stylistic. A new backend enters the protocol by adding an enum value — a spec change with a fixture, not a wire event (§ Pluggable backends in the zero-copy transport doc) — so an unknown token is not a newer peer talking to an older one. It is a corrupt frame or a non-conforming producer, and both are conditions a peer recovers from by resync (§ Resync / gap handling).
Normalizing it is worse than it looks, and the reason is a theorem the model already
proves. resolve_wrong_backend states that a descriptor of one kind never resolves
against a different backend’s table — receivers route by kind. Reading an unknown
kind as shm is routing a non-shm descriptor into the shm table; the
{generation, epoch, len, checksum} verification then usually rejects it, so the
observable outcome is right most of the time. That is the trap: the guarantee is
supposed to be structural, discharged by routing, and normalization silently
downgrades it to a probabilistic one discharged by a 64-bit checksum, against a
backend this build genuinely resolves. It is the same substitution the decoder bound
already outlaws for NodeId in § NodeId / PeerId — the frame decodes cleanly,
addresses something other than what the producer named, and nothing downstream can
tell.
Audited (#lzblobbackendstrict) across the library decode surface of all nine
bindings, which had split 5–2 — and every one of the five documented its choice
as deliberate wire forward-compat:
| Binding | Before the audit |
|---|---|
| lazily-rs, lazily-js | rejected — the conforming behaviour, reached independently |
| lazily-go, lazily-py, lazily-kt, lazily-zig, lazily-cpp | normalized to shm, each with a written forward-compat rationale |
| lazily-cs, lazily-dart | no dispatch site — the descriptor decodes through a field-typed path |
The lazily-cs row is wrong and is left standing as a record of the error. Replaying
fixture v2 found a real dispatch site there — IpcWire.ReadBlob on decode,
IpcWire.WriteBlob on encode, and BlobTransport.EffectiveBackend at every routing
site. “No dispatch site” was a review conclusion, and it is the kind of conclusion a
replay overturns: an audit that reads for a switch finds nothing when the
discriminator is read through a field-typed path that dispatches anyway.
The five agreed with each other on the defence, too: the checksum catches it, so the
descriptor resolves to nothing rather than to another backend’s bytes. That argument
is what the theorem exists to make unnecessary. An undocumented default and a
deliberate one are indistinguishable from the outside — and, it turns out, so are two
deliberate ones pointing in opposite directions.
conformance/codec/blob_backend_discriminator.json
replays both halves in both codecs. It carries an arrow scenario so the leniency
cannot be implemented by ignoring the discriminator outright, and pins the re-encoded
frame so a binding cannot satisfy the clause by round-tripping whatever it received.
The encoder half is a separate obligation and is not implied by the decoder half:
lazily-rs has carried skip_serializing_if since the commit that introduced the
field, while a binding can implement strict rejection and still emit
backend: "shm" — which is why the re-encode assertions exist rather than being
inferred.
Fixture v2 (#lzblobbackendstrict) closes four holes v1 left. Every one of them
was found by a binding replaying v1, not by reviewing it — which is the argument
for replaying a fixture in nine places before trusting it. The fixture is now 14
scenarios, seven wire shapes in both codecs:
in_processis carried, not merely declared. v1 listed three backends inassertions.backendsand shipped scenarios for two, so a binding that knew only{shm, arrow}rejectedin_process— naming the token, conformingly, by the letter of this clause — and passed all eight scenarios while implementing a smaller enum than the clause declares. Found independently by three bindings. Thearrowcontrol proves the discriminator is read;in_processproves the vocabulary is complete, and no scenario count substitutes for it: the missing fact was a set difference, so the corpus-side guard is now a set difference (test_backend_fixture_carries_every_backend_it_declares).- The
nullbullet above is executed, not just written. Four bindings raised it independently while implementing v1 and had already split three ways. It is anacceptscenario that is deliberately schema-invalid: the string-typed enum binds the ENCODER, and the decoder’s leniency is the separate fact under test. The re-encode assertion still applies, so the null does not survive a round trip. - A non-string
backendis a scenario. The clause is written entirely about tokens, so a runtime whose reader coerces rather than throws on a number in a string position normalizes silently there — the same failure this clause names, arriving through a door it does not describe. One binding’s only real defect under v1 was exactly this. Both reject forms now assertexpect.rejection_is_decode_error, which is the It must be catchable obligation above made executable;expect.rejection_kinddistinguishes them, and onlyunknown_tokencarrieserror_names_token, because a type error has no token to name and requiring the field name would pin a message format no codec’s native type error produces. expect.epochis gone, replaced byframe_epoch(9) andblob_epoch(5). v1 carried9in both theDeltaframe and theShmBlobRefdescriptor, so a runner reading the frame’s epoch and one reading the descriptor’s both passed a single assertion. Found by two bindings. The key was removed rather than redefined so a runner still reading it fails loudly instead of silently reading the other one.
Two codecs are still not two implementations. Several bindings bridge MessagePack
into the same DOM the JSON decoder produces, or share one serde impl, so the
msgpack half of a scenario pair can yield one discriminator verdict rather than an
independent second one. It still covers the bridge and the encoder; a fully green run
must not be read as two implementations agreeing, and a binding whose two codecs
share a decode path should record that in its own ledger rather than infer
independence from the scenario count. This one is not fixable in the corpus — it is a
property of the bindings — so it stays stated rather than pinned.
What v2 found when the nine bindings replayed it. The four holes above were predictions about what an unexecuted clause hides; this is what was actually behind them:
-
Four of nine bindings REFUSED the explicit
null— lazily-rs (#[serde(default)]supplies the default when the key is missing, so a presentnullstill reachedBlobBackendKind::Deserializeand failed as a type error in both codecs), lazily-cs (aValueKinderror), lazily-zig (error.ExpectedString), lazily-kt (JsonNullis aJsonPrimitivewhoseisStringis false, so it fell into the non-string arm). Each returned a decode error for a frame the reference implementation emits. The bullet stating the rule had been in this document since the clause landed; it was prose, and four bindings read it and still got it wrong — including lazily-rs, which the row above records as reaching the conforming answer on the unknown token independently. Being right about the hard half of a clause is not evidence of being right about the easy half. That ratio is the argument for making an adjudication executable rather than writing it down.The lazily-rs fix carries a detail the other bindings do not face:
serde_json’sdeserialize_strconstructs theinvalid_typeerror itself on a null and never calls the visitor, so reading the null requiresdeserialize_option— and that forced a branch onis_human_readable(), since postcard writes no option tag for this field and asking it for one would misalign the frame. The strictdeserialize_strpath stays for non-self-describing codecs. -
Two bindings had no decode-error family at all. lazily-zig’s decode errors were an inferred error set nobody had written down, and lazily-kt raised
IllegalStateExceptionfrom one refusal andIllegalArgumentExceptionfrom the other — neither a subtype of the other, so a caller catching the documented one had the other fail past the handler. Both now name the family (ipc.BlobDescriptorDecodeError;sealed class IpcDecodeException). The It must be catchable obligation above was unassertable untilrejection_is_decode_errorexisted, which is why a refusal being wrong in this specific way survived a nine-binding audit. -
rejectedandrejection_is_decode_errorare genuinely two facts. lazily-cpp demonstrated it on demand: throwingstd::invalid_argumentfor the non-string turnsrejection_is_decode_errorred whilerejectedstays green. A bare is-error assertion passes the hierarchy trap. -
A fifth false-green shape, distinct from the four already catalogued.
rejection_kindmust be asserted against what the LIBRARY raised — a distinct error type, or the message the refusal actually produced. A runner that derives it from the scenario’s ownbackend_formcompares the fixture to itself and passes while proving nothing. Raised by lazily-kt, which made its two refusals distinct types precisely so the assertion has something of the library’s to compare against. The four already on record are: a stale build artifact, a filter matching nothing, a probe aimed at a value the corpus never carries, and a filtered build run reporting N/N passed while selecting the target test not at all.
Two bindings’ green is worth less than it looks, and both say so in their own
runners rather than letting the scenario count imply otherwise: lazily-cpp bridges
MessagePack into the same JsonValue DOM decode_json produces, and lazily-py
bridges into the same DOM as well, so each scenario pair there yields one
discriminator verdict. The msgpack half still covers the bridge and the msgpack
encoder, which are distinct failures.
FFI Boundary
Types
typedef struct {
uint8_t* ptr;
size_t len;
} LazilyFfiBytes;
typedef enum {
LazilyFfiStatus_Ok = 0,
LazilyFfiStatus_Empty = 1,
LazilyFfiStatus_NullPointer = 2,
LazilyFfiStatus_InvalidMessage = 3,
LazilyFfiStatus_EncodeFailed = 4,
LazilyFfiStatus_Panic = 5,
} LazilyFfiStatus;
typedef enum {
LazilyFfiMessageKind_Unknown = 0,
LazilyFfiMessageKind_Snapshot = 1,
LazilyFfiMessageKind_Delta = 2,
LazilyFfiMessageKind_CrdtSync = 3,
LazilyFfiMessageKind_ResyncRequest = 4, /* #lzsync control frame */
LazilyFfiMessageKind_OutboxAck = 5, /* #lzsync control frame */
} LazilyFfiMessageKind;
Contract
- All allocation ownership is explicit: caller owns input bytes; Rust owns output buffers until the paired free function is called.
- Errors return
LazilyFfiStatus; panics are caught before crossing the C ABI. - The channel decodes each accepted frame as
IpcMessage, then re-encodes canonical JSON bytes.
Signaling Protocol (WebSocket)
Client → Server
| Type | Fields | Description |
|---|---|---|
join | peer, capabilities? | Register with session |
offer | to, sdp | WebRTC SDP offer |
answer | to, sdp | WebRTC SDP answer |
ice | to, candidate | ICE candidate |
relay | to, payload | Relay opaque payload |
leave | — | Disconnect |
Server → Client
| Type | Fields | Description |
|---|---|---|
welcome | peer, peers | Roster on join |
peer-joined | peer | New peer in session |
peer-left | peer | Peer disconnected |
offer | from, sdp | Forwarded offer |
answer | from, sdp | Forwarded answer |
ice | from, candidate | Forwarded ICE |
relay | from, payload | Forwarded payload |
error | code, message | Error response |
Anti-spoofing
The from field on every forwarded frame is the sender connection’s registered peer id, never client-supplied.
Example frames
{ "type": "join", "peer": 1 }
{ "type": "welcome", "peer": 1, "peers": [] }
{ "type": "offer", "to": 2, "sdp": "v=0\r\n..." }
{ "type": "answer", "from": 2, "sdp": "v=0\r\n..." }
{ "type": "ice", "from": 2, "candidate": "candidate:..." }
{ "type": "relay", "to": 2, "payload": { "any": "json" } }
{ "type": "peer-joined", "peer": 2 }
{ "type": "leave" }
Permission modes
| Mode | Description |
|---|---|
open | Any peer may join and signal any other joined peer |
allowlist | Default-deny: peers require explicit grants; directed frames only to allowed targets |
Distributed: CRDT Cell Plane
This plane specifies merge: crdt — the first multi-write merge mechanism of the
Cell Model. CRDT is one mechanism among several the
cell model reserves (lww, ot, lease, custom); it is the first defined because it
converges without coordination. Everything here applies to a multi-write cell that
declares merge: crdt; the cell-kind classification, ingress-on-roots-only boundary,
and cell-as-merge-unit granularity are defined once, mechanism-independently, in the
Cell Model.
Cell register types
These are the CRDT-mechanism register types (the value shapes available within
merge: crdt); they are distinct from the cell-model’s MergeMechanism axis.
| Type | Merge | Description |
|---|---|---|
| LWW-register | Last-write-wins (HLC timestamp) | Default; “current value” semantics |
| MV-register | Multi-value | Surfaces concurrent writes as a set |
| PN-counter | Additive | Positive-negative counter |
CRDT properties
- Each replicated cell is keyed by a hybrid logical clock (HLC): wall-clock for human-meaningful ordering, logical counter for causal tiebreak.
- Local PartialEq invalidation guard applies after merge: equal values invalidate nothing.
- Memo equality suppression holds post-merge.
lazily-ipc’sDeltageneralizes to per-peer causal stamps: each peer keeps its own sequence; cross-peer order comes from HLC/dot metadata.
Anti-entropy wire format (CrdtSync)
The plane rides the same lazily-ipc transport as Snapshot/Delta. Alongside the
single-producer mirror, a third IpcMessage variant carries multi-writer plane traffic:
# Snapshot/Delta/CrdtSync are the forward (state) plane; ResyncRequest/OutboxAck
# are the reverse-channel reliable-sync control frames (§ Reliable Sync, #lzsync).
IpcMessage = Snapshot(Snapshot) | Delta(Delta) | CrdtSync(CrdtSync)
| ResyncRequest(ResyncRequest) | OutboxAck(OutboxAck)
WireStamp = { wall_time: u64, logical: u64, peer: u64 } # total order (wall, logical, peer)
CrdtOp = {
node: NodeId, # volatile target id
key: NodeKey?, # optional wire-stable address (survives NodeId churn, #lzwirekey)
stamp: WireStamp, # the HLC stamp that produced this state
state: IpcValue, # the converged CRDT state to merge (state-based / CvRDT)
}
CrdtSync = {
frontier: [(peer: u64, WireStamp)]?, # the sender's per-peer stamp frontier
ops: [CrdtOp], # the op batch this frame ships
}
WireStamp is the wire mirror of the runtime HLC stamp (all plain integers), so the wire
format is codec-stable whether or not a peer compiles the CRDT runtime in. It round-trips
across all three codecs (JSON, MessagePack, postcard) and is classified by the FFI message
kind (CrdtSync = 3).
Frontier suppression (#lzspecfrontiersuppress). frontier is optional. A frame
that omits frontier (JSON/msgpack) or carries the sentinel frontier: [] means “the
sender’s stamp frontier is unchanged since the last frame the receiver accepted” — the
receiver reuses its last-merged frontier to compute the watermark. This is a pure wire
savings on chatty peers (a frame shipping only ops need not repeat an unchanged frontier).
The self-describing codecs (JSON, msgpack) omit the field when unchanged; positional
postcard carries a 1-bit discriminant. A sender MUST include frontier whenever its
advertised stamp has advanced; a receiver that receives a frontier-less frame with no prior
frontier (cold start) MUST request a full CrdtSync (fail closed). Conformance risk is
medium: paired fixtures exercise both the suppressed and the full forms.
State-based, idempotent. Each CrdtOp ships the converged register/sequence/text
state for a node. The receiver merges state into its local replica; because every cell
CRDT merge is commutative, associative, and idempotent (proven in
formal/lean/LazilyFormal/CRDT.lean, stampJoin_{comm,assoc,idem}), out-of-order,
duplicated, or batched delivery all converge. Re-sending a frame the receiver already has
is a no-op.
Stamp-frontier exchange. CrdtSync.frontier advertises the highest WireStamp the
sender has observed from each peer. The receiver merges it into its own frontier (per-peer
max); the causal-stability watermark is the min over membership of that frontier —
the causal point every replica has provably passed.
Watermark / GC contract. A tombstone whose delete stamp is ≤ the stability watermark
is collectable on every replica, so dropping it cannot lose an edit. This safety property
is formally proven (LazilyFormal.CRDT.collectable_implies_observed_everywhere: a
collectable stamp is ≤ every member’s observation) and drives the runtime
SeqCrdt::gc / TextCrdt::gc_with. A single replica’s local clock is explicitly not a
sound watermark; only the version-vector minimum is.
GC scheduling (#lzspecgcdefer). The contract above fixes when a tombstone is
collectable; it does not mandate when collection runs. A binding MAY defer GC to an
idle window, batch it under memory pressure, or run it incrementally (sweep a bounded number
of tombstones per anti-entropy round), provided (a) no tombstone below the watermark is
re-examined after reclaim and (b) unbounded accrual is reported via the instrumentation
surface. Deferring GC changes only memory footprint, never observable values.
Permission filtering. CrdtSync.filter_readable(peer) drops ops for non-readable nodes
entirely (omission, not redaction — like Delta). The frontier advertisement is retained
in full: it names peers and stamps, not node content, and the receiver needs the whole
frontier to compute a sound watermark.
Status. The wire format, codec round-trips, permission filtering, and point-to-point
IpcSink/IpcSourcedelivery are implemented (#lzcrdtplane5a). Wiring the plane to livemerge: crdtroot cells (local edits →CrdtOps; remoteCrdtOps →ReplicatedCellingress merge) andBridgeHubfan-out ofCrdtSyncis the runtime-integration slice (#lzcrdtplane5b).
Delta-CRDT sync (#lzspecdeltacrdt). Today each CrdtOp ships the full converged
register state per anti-entropy round (state-based / CvRDT). For LWW/MV registers and
PN-counters whose state is small this is acceptable, but for OR-set membership and large
register values the full-state ship is wasteful when little changed. The watermark + join
algebra already exist (formal/lean/LazilyFormal/CRDT.lean, stampJoin_{comm,assoc,idem});
the #lztextsync pattern (cell-model.md § Delta sync) already proves the
version_vector / delta_since(their_vv) / apply_delta triad for text.
This section lifts the same delta pattern to the cell-register plane as an additive, optional control frame:
DeltaSinceRequest = { node: NodeId, their_vv: [(peer, counter)] }
- A receiver of a
DeltaSinceRequestresponds with aCrdtSyncwhoseopscarry only the states whose stamp is pasttheir_vv(the delta), instead of the full converged state. An empty delta (nothing pasttheir_vv) is a valid response. - The join is the same semilattice:
apply_delta≡merge— commutative, associative, idempotent — so a delta is safe to resend and applies in any order. - A binding that does not implement delta-CRDT sync continues to ship full state; the
DeltaSinceRequestframe is opt-in and the receiver falls back to full state when it does not recognize the request. Conformance fixtures exercise both the full-state and the delta-sync paths and assert they converge to the same state (#lzspecdeltacrdt).
Reactive keyed-map sync (#lzfamilysync)
A keyed reactive map (ReactiveMap — its SourceMap / ComputedMap specializations;
cell-model.md § “Keyed cell collections”) is a local keyed reactive
collection. This section fixes its distributed contract: what a peer does with a keyed
CrdtOp (NodeKey = namespace/suffix) for a map entry it has not registered locally.
The base plane, given such an op, resolves the node by NodeKey; if no cell is registered
under that key the op is dropped. For a keyed map that is wrong: a key added on one replica
would never appear on another, and any derived aggregate over the map (a count of entries)
would diverge.
Map-granularity sync closes the gap with materialize-on-ingest: a replica registers a
keyed map under a namespace; an inbound keyed op whose first NodeKey segment matches a
registered map materializes a fresh entry (a new local NodeId, indexed by the wire
NodeKey) seeded from the op’s converged register, then merges. Because the materialized
entry is seeded from the op state, materialize-on-ingest is exactly the pointwise CRDT
merge, so it inherits the full semilattice convergence.
Contract (proven in lazily-formal FamilySync.lean):
- Materialize, never drop. A keyed op for an absent family entry makes that key present
and adopts the op’s value (
applyOp_present,applyOp_absent_adopts). - Membership propagation. After sync a key is present iff it was present on either
replica — the union (
present_merge). The present set only grows (deferral-not-dealloc); a removed entry is a value-level tombstone, not a dropped key. - Convergence + idempotence. Materialize-on-ingest equals merging the op’s single-entry
state (
applyOp_eq_merge), so op delivery is order-independent (applyOp_comm) and re-delivery is a no-op (applyOp_idem). - Derived-aggregate transparency. Once two replicas converge, any derived count over the
keyed map agrees regardless of sync direction/batching (
aggregate_converges,aggregate_batch_invariant) — e.g. a live-editor / open-document count converges across editors. ANodeId-churn-stable membership signal drives the recompute so a remote-materialized key is picked up by the derived aggregate.
Conformance: conformance/familysync/materialize_on_ingest.json.
Single-writer effect authority
CRDT convergence covers state. For irreversible external actions (send email, charge card, fire webhook), gate the effect behind a single-writer authority — a designated peer (or small Raft group) decides when the effect fires, at-most-once.
Durable effect sinks (#lzdurablesink)
Durable storage is an effect sink, not a transition authority. While a Lazily runtime is live, transitions are decided from Lazily state; durable storage receives a projection or an ordered fact as an effect, and a sink MUST NOT reload storage to arbitrate the transition it is currently persisting. A projection (latest recoverable state) is an idempotent upsert of the settled epoch from an Effect / AsyncEffect; lossless history uses the existing TopicCell / DurableOutbox drain (append / replay / ack with a stable cursor). Success advances a monotone durable_through(epoch); failure stays represented in live state as pending / retrying / backpressured. Cold loading and migration belong to a separate startup hydrator, never the decision seam. Ephemeral-plane values MUST NOT enter a durable sink — reuse the existing Durable marker. Full authority rule, the projection-vs-history shape table, and the two reference examples (coalesced projection; lossless ordered fact sink) live in docs/durable-sinks.md; the formal backstop is lazily-formal/LazilyFormal/DurableSink.lean.
Reliable Sync (#lzsync)
The Snapshot/Delta and CrdtSync planes above define what is on the wire and how state
converges once delivered. They do not define delivery reliability: what a receiver does with a
gap, what a sender does with a send that failed, or how a reconnected peer catches up. Today each
consumer hand-rolls that loop (and Delta::apply_status → ResyncRequired is a signal with no
production handler). This section fixes the reliable-sync protocol so gap recovery, reconnect
backfill, and at-least-once delivery are specified and cross-language-conformant, not
re-invented per integration.
Layering — mechanism here, policy injected. The three components below are pure protocol:
identical logic in every binding, no I/O, no clock, no storage engine. The environment-specific
choices — which byte transport, which persistence backend, retry cadence, threading — are supplied
by the host application behind the named seams (SnapshotProvider, DurableOutbox store,
Clock/scheduler, the IpcSink/IpcSource transport). A binding ships the protocol and a default
in-memory backend; the host plugs durable storage and a real transport.
Control frames are IpcMessage variants, not a side channel. ResyncRequest and OutboxAck
are two new externally-tagged IpcMessage variants (FFI message kinds 4 / 5), riding the
same framed, codec-negotiated, bidirectional message plane as Snapshot/Delta/CrdtSync —
the reverse (receiver → sender) direction of that plane. This is deliberate over a separate control
type: they share one encode/decode path, one demux point, one FFI kind discriminant, and — because
they interleave in the same ordered stream as the deltas — a well-defined in-band position (an
OutboxAck { through_epoch: N } is meaningful relative to the deltas already sent on that channel).
The liveness ops are CrdtOps on the existing CrdtSync variant. A binding MUST add both variants
to its IpcMessage enum and its FFI message-kind mapping.
Codec. Every reliable-sync frame (the two control variants and the liveness CrdtOps) is an
ordinary framed message on the negotiated codec. Per § Frame codecs the
cross-language boundary negotiates msgpack (self-describing, evolution-safe, named-field),
with json as the required reference codec; postcard stays the Rust-only fast path. Conformance
requires every new frame to round-trip through both json and msgpack (semantic round-trip,
not byte-identical, for the map codecs), the same discipline the three prior IpcMessage variants
already hold.
ResyncCoordinator (#resync-coord)
A pure receiver-side decision function over the inbound frame stream. It holds one piece of state
per source — last_epoch (the highest epoch this receiver has fully applied) — and classifies each
inbound Snapshot/Delta:
enum ResyncAction { Apply, RequestSnapshot { from: u64 }, Ignore }
// pure; no I/O. `ingest` inspects a frame and returns the action; the caller performs it.
fn ingest(&mut self, msg: &IpcMessage) -> ResyncAction
Decision table (given receiver last_epoch = L):
| Inbound | Condition | Action | Effect on L |
|---|---|---|---|
Snapshot { epoch: e } | always | Apply | L := e (adopt snapshot state) |
Delta { base_epoch: b, epoch: e } | b == L and e >= b + 1 | Apply | L := e after fold |
Delta { base_epoch: b } | b < L | Ignore (already applied / re-delivery) | unchanged |
Delta { base_epoch: b } | b > L | RequestSnapshot { from: L } | unchanged until snapshot |
Delta { epoch: e, base_epoch: b } | e < b + 1 | Ignore (malformed/empty) | unchanged |
RequestSnapshot { from }is emitted at most once per detected gap; a coordinator that has already requested and not yet applied a coveringSnapshotsuppresses duplicate requests for the same gap (it stays in aresyncingsub-state,Ignore-ing further deltas until the snapshot lands). This bounds request storms under a burst of ahead-of-cursor deltas.- Convergence guarantee. A receiver that drops an arbitrary suffix of deltas, then applies the
resync
Snapshot, reaches the same graph state as one that saw every delta — gap recovery is state-equivalent, not lossy (provenReliableSync.resync_convergence). This holds because aSnapshotis a full-state frame, not an incremental one. - Idempotent re-delivery. A
Deltawithbase_epoch < L(a frame the receiver already folded, re-sent by the outbox) isIgnored, so at-least-once delivery yields exactly-once effect.
The application supplies snapshots for the sender side of a resync via:
trait SnapshotProvider { fn snapshot(&self, from_epoch: u64) -> IpcMessage; } // returns Snapshot { epoch >= from_epoch }
DurableOutbox (#durable-outbox)
The sender-side contract that makes delivery at-least-once across a crash/reconnect. Today a failed send leaves a permanent gap (the out-epoch is bumped before the send; a reconnect is a fresh peer at epoch 0 with no backfill). The outbox closes that: every frame is durably recorded before it is sent, retained until the peer proves receipt, and replayed from the peer’s cursor on reconnect.
trait DurableOutbox {
fn append(&mut self, epoch: u64, msg: &IpcMessage); // MUST persist before the send is attempted
fn ack_through(&mut self, epoch: u64); // peer proved receipt through `epoch`; retained frames <= epoch may be pruned
fn replay_from(&self, cursor: u64) -> impl Iterator<Item = (u64, IpcMessage)>; // frames with epoch > cursor, in epoch order
fn retained_depth(&self) -> usize; // unacked queue depth — the backpressure fill level (Progress.retained)
fn coalesce_to_snapshot(&mut self, epoch: u64, snapshot: &IpcMessage) -> bool { false } // state cells: replace the unacked suffix with one snapshot; op-log frames decline (false)
}
Normative contract:
- Append-before-send. A frame MUST be durably appended before it is handed to the transport.
If the process dies between append and a confirmed send, the frame is still in the outbox and is
replayed on reconnect. (The pre-send epoch bump that caused the permanent-gap bug is replaced by:
append at
epoch, send, and onlyack_throughretires it.) - Replay-from-cursor. On (re)connect the peer advertises the highest epoch it has applied
(its
ResyncCoordinator.last_epoch, carried in the reconnect handshake or anOutboxAck); the senderreplay_from(cursor)re-sends every retained frame withepoch > cursorin order. A frame the peer already applied (base_epoch < last_epoch) isIgnored by the coordinator — replay is safe. - At-least-once ⇒ exactly-once effect. Replay delivers every op at least once; idempotent apply
(the coordinator’s re-delivery
Ignore+ the CRDT/PartialEq guards) makes the net effect exactly-once — no lost op, no doubled op (provenReliableSync.outbox_at_least_once_exactly_once_effect). - Ack semantics.
ack_through(e)is a retention signal (safe to prune<= e), never a delivery-success authority for a domain effect (that is a causal receipt). An outbox MAY prune lazily; correctness does not depend on prompt pruning, only on not pruning un-acked frames. OutboxAckframe. The receiver periodically (or on request) sendsOutboxAck { through_epoch: u64 }— a new framedIpcMessage— so the sender can advance retention and so a reconnect handshake can carry the resume cursor.- Coalescing (optional, element-algebra-gated). An outbox MAY bound its retained depth without
dropping an op, per § Backpressure & outbox coalescing.
A state outbox (LWW/OR-set/counter/graph frames) collapses the unacked suffix to one
Snapshotviacoalesce_to_snapshot(memory-bounded — the coalesce is the cell’s own join, e.g. LWW → last value, provenReliableSync.coalesce_by_join_sound/coalesce_to_snapshot_state_equiv). An op-log outbox (QueueCell frames) declines snapshot-coalesce (returnsfalse) and instead fuses a run of same-direction ops into onebatchframe (framing-bounded, order-preserving,ReliableSync.batch_fusion_state); it relies on source-sideQueueCell.is_fullfor memory bounding. Coalescing is a retention optimization only — it never changes which effects the receiver observes.
Bindings ship an InMemoryOutbox (default; correct within a process lifetime) and a reference
file-backed impl for tests/conformance (proves the crash-replay path deterministically). The host
plugs its own durable store (agent-doc: SQLite) behind the same trait; the cursor math is protocol,
the storage is not.
SyncDriver (#sync-driver)
The loop shape that wires an outbound producer to a transport through the outbox, and drives
resync on reconnect. It owns no clock and no runtime: the host calls tick() from its own
scheduler, so the driver stays a pure state machine with injected seams.
struct SyncDriver<S: IpcSink, R: IpcSource, O: DurableOutbox, C: Clock> { /* transport, outbox, coordinator, clock */ }
impl SyncDriver {
// One scheduler-driven step. Returns Progress (sent N, applied M, resynced) or a DriverError.
fn tick(&mut self) -> Result<Progress, DriverError>;
}
tick() performs, in order:
- Drain inbound. Pull available frames from
IpcSource; feed each toResyncCoordinator.ingestand perform the returned action (Applyinto the local graph, emit aResyncRequest/OutboxAck, or drop). Advancelast_epochon applied frames. - Send outbound. For each new local flush,
outbox.append(epoch, frame)thenIpcSink.send(frame). A send error does not unwind the driver and does not lose the frame: the frame stays in the outbox, the driver records the transport as degraded, and the send is retried from the outbox on a latertick(cadence/backoff is the injectedClock’s policy). - Resync on reconnect. When the transport reports a fresh/reopened peer, exchange cursors
(peer’s
OutboxAck.through_epoch) andreplay_from(cursor); if the local receiver is behind, emit itsResyncRequest. - Retention. On an inbound
OutboxAck,outbox.ack_through(through_epoch).
Contract:
- No frame lost on send failure. The append-before-send + retain-on-error rule means a frame is
delivered on a subsequent tick once the transport recovers (the exact bug the current
?-propagatingpollhas, where a failed send bumps the epoch and unwinds). - Bounded work per tick.
tick()does a bounded amount of drain/send work and returns; the host controls cadence. No internal blocking, no async runtime baked in. - Injected clock/transport. Retry cadence, backoff, and “is the peer fresh” come from the
injected
Clockand transport signals — policy stays in the host, mechanism in the driver. - Backpressure is host policy, not driver mechanism. The driver does not bound its outbound
staging or block the producer:
enqueueis unbounded and theDurableOutboxretains every unacked frame until the peer’sOutboxAck(this is the at-least-once durability guarantee — a frame is not dropped to relieve pressure). Instead the driver exposes the signals a host uses to apply backpressure itself: the stall state (is_stalled/stalled_for) and the retained-outbox depth (Progress.retained). A host bounds memory by (a) rate-limiting or coalescing enqueues off those signals — a re-emit at an already-accepted epoch is an idempotent no-op delta, so coalescing is the natural limiter — and (b) choosing aDurableOutboxthat spills to durable storage rather than RAM (e.g. a SQLite-backed outbox), so a long-stalled peer grows disk, not heap. A bounded outbox with anis_full/ high-watermark cap, and the per-item coalesce modes that keep it bounded without shedding an op, are specified in § Backpressure & outbox coalescing.
IpcSink/IpcSource are the existing abstract transport traits (feature ipc); the byte carrier
is any DataChannel. Un-gating: the driver and the BridgeHub fan-out it can wrap depend only
on these abstract ipc traits, so they MUST be usable without the webrtc feature (a caller
supplying a Unix-domain-socket DataChannel gets reliable sync with no WebRTC dependency). The
per-document channel decision (one driver/transport per document vs one hub with
document_hash+NodeKey namespacing) is a host concern; the pull-only consumer path may not need
BridgeHub fan-out at all.
Backpressure & outbox coalescing (#lzsync-backpressure)
The unacked outbox is a bounded cursor-queue, and that framing makes backpressure a queue
property rather than a bolt-on: append is a push to the tail, OutboxAck.through_epoch is the
consumer cursor, replay_from is a FIFO peek of the unacked suffix, and the retained depth
(Progress.retained) is the fill level. A peer that stops acking is a consumer that stops advancing
its cursor, so the queue fills — and a bounded outbox surfaces that as an is_full /
high-watermark signal a host uses to throttle the producer. Recovery is the reactive dual: an
OutboxAck advances the cursor, the queue drains below the watermark, and the fill signal clears so
the producer resumes — the same reactive backpressure the QueueCell model pins
(queuecell_bounded_backpressure),
applied to the outbox. Non-ack is backpressure; the liveness cells
disambiguate a slow consumer (queue fills — wait) from a dead one (queue drains — reconnect +
replay). Formalized as ReliableSync.outbox_is_bounded_queue (enqueue grows depth by one, ack
dequeues the acked front FIFO, a coalesced suffix is one frame).
A bounded queue must not grow unbounded under a persistently-slow peer, and the outbox stays bounded by coalescing its unacked suffix — never by dropping an op. How it coalesces is dispatched by the element’s merge algebra, not by a single global switch. Every cell type supplies its own coalesce, and each falls into one of two families by whether the type has an idempotent join:
| Element | Coalesce | Bounds | Basis |
|---|---|---|---|
| LWW register | keep the last (max-stamp) value | memory | WireLwwRegister::join folded over the suffix |
| OR-set | union of adds/removes | memory | OrSet::join folded |
| Counter / PN-counter | sum the deltas | memory | additive monoid |
| Sequence CRDT / graph projection | merged Snapshot | memory | SnapshotProvider |
| QueueCell (op-log) | batch-fuse a run of same-direction ops | frames only | op concatenation |
The unifying law: coalesce = the element’s semilattice join folded over the unacked suffix.
- Join / snapshot coalesce (idempotent cells: LWW, OR-set, counter, graph). The suffix collapses
to one merged value — order-, regroup-, and retry-independent (
ReliableSync.coalesce_by_join_sound; for the whole-graph frame, adopting the snapshot subsumes the dropped deltas —ReliableSync.coalesce_to_snapshot_state_equiv). This bounds memory: a peer arbitrarily far behind costs one frame, not the whole backlog. It is the CRDT answer to “don’t block the producer” — the producer keeps appending; the queue keeps collapsing. LWW-to-last-value is exactly the join being max-stamp. - Batch-fusion coalesce (op-log cells: QueueCell). A queue has no idempotent join — order and
multiplicity are the value — so it cannot collapse; it only fuses a run of same-direction ops
(
QueuePush(a); QueuePush(b); QueuePush(c)→ one atomic batchDelta, the op-log delta form of § QueueCell op-log delta form). Lossless and order-preserving (ReliableSync.batch_fusion_state; a fused multi-epoch delta equals its expanded unit run,multi_epoch_apply_eq_fold). This bounds frame count and per-frame overhead (one header, one codec envelope, one reactive invalidation on apply), not memory — a queue cannot drop elements and stay a queue. An op-log sync therefore still needs source-sidetry_push → Full(QueueCell.is_full) to bound memory; batch-fusion complements it, it does not replace it. Fuse only runs of same-direction ops; do not fuse across apopin a bidirectional log.
Rule of thumb: a state sync (LWW/OR-set/counter/graph) absorbs pressure by collapsing
(memory-bounded, never blocks the producer); an op-log sync (QueueCell) absorbs it by rejecting at
the source (is_full, blocks the producer) and may fuse frames to cut overhead. Both keep every
accepted op’s effect; neither sheds load silently.
Bounding producer memory (op-log). is_full / try_push → Full bounds the queue, not the
producer: if the producer answers Full by stashing the rejected item in an unbounded side-buffer,
the queue is bounded but memory is not — the backlog just moved up one hop. A QueueCell sync bounds
producer memory only when the Full signal terminates at a stoppable or sheddable source, and no
hop in between holds an unbounded buffer:
- Suspend (the lazily-native path). The producer is a reactive effect that reads
is_full; while full it does not run, and the pop that clears full (true→false) invalidates the reader and re-fires it. Producer memory is bounded because a not-running effect holds no backlog — it generates on demand, gated by the queue. This is thequeuecell_bounded_backpressureloop used as a producer gate rather than a mere status read. - Propagate upstream. An imperative producer fed by its own bounded input returns
Full/pending to its source, recursively, until the chain reaches something that can actually stop (a pull-based reader; a socket whose reads pause, so TCP advertises a zero window and the remote sender stops). Every hop is bounded-and-backpressuring; the stop propagates to the origin. - Shed. When the source cannot stop (a real-time feed, an external peer that will not slow), the
only way to bound memory is an overflow drop policy —
drop-oldest/drop-newest, the non-rejectQueueStoragebackends the QueueCell model names — or a lossy summary. Dropping is a deliberate, observable choice, not silent truncation.
The invariant: backpressure bounds memory only if it ends at a stoppable or sheddable source; an unbounded buffer at any hop defeats it. The queue supplies the signal and the bound at the queue; the producer must convert that signal into actually-not-generating (suspend/propagate) or actually-discarding (shed) — never into unbounded local absorption. (A state cell escapes this because its coalesce is an unbounded-absorption-that-stays-bounded: churn collapses into the join. A queue has no such join, which is exactly why its producer must pause or drop.)
Two seam refinements this makes explicit, both host-invisible in the current traits:
- Ack-as-credit. For non-ack to mean consumer backpressure (not merely non-receipt), a receiver
SHOULD gate its
OutboxAck.through_epochon its consumer’s drain cursor, not on mere fold-into-projection — otherwise a backed-up consumer keeps acking and the credit signal never forms. The unacked depth then behaves as a sliding credit window; withholding the ack pushes back end-to-end onto the sender’s outbox. - Sink would-block.
IpcSink::sendreturning onlyOk/Errcannot express a transport that is up but momentarily full (kernel socket buffer,DataChannel.bufferedAmount); that fullness is a distinct layer from consumer-slowness and is invisible to the ack. A binding MAY widen the sink outcome to distinguishFull(re-stage the frame + back off — not a reconnect) from a hard failure. This is the one backpressure axis the ack cursor structurally cannot cover.
Transport seam (IpcSink / IpcSource, #lzsync-transport-seam)
The SyncDriver is generic over exactly two host-supplied seams. Every binding that ports the
driver MUST provide the same two-method contract so the loop above is identical across
languages; the seams are the injected boundary the design assigns to the host (“which socket” is a
deployment choice), so they carry no wire form of their own — what crosses the wire is the
codec-encoded IpcMessage frame (msgpack is the cross-language default; see § Frame codecs). A
binding names them idiomatically (Rust traits, Kotlin/TS interfaces); the semantics are normative:
// Outbound: deliver exactly one already-encoded protocol frame.
trait IpcSink { fn send(&mut self, msg: &IpcMessage) -> Result<(), SinkError>; }
// Inbound: poll for the next frame without blocking.
trait IpcSource { fn recv(&mut self) -> Result<Option<IpcMessage>, SourceError>; }
sendMAY fail and MAY be lossy. Asenderror means the frame was not durably handed to the peer. At-least-once is a driver property, not a sink property: on a send error the driver keeps the frame in theDurableOutboxand replays it after the next reconnect. A sink is therefore free to be a plain best-effort write (one connect-send-receipt on a Unix socket, oneDataChannelframe, …) — it never has to buffer or retry, because the outbox already does.recvis poll, not block.Ok(None)means the source is currently exhausted or closed; the driver treats it as “no inbound progress this tick” and returns — it never parks a thread on the source (cadence is the host’sClock/scheduler policy).Ok(Some(frame))yields one frame.- A
recvErris the reconnect signal. A source read failure surfaces fromtick()asDriverError::Source; the host re-establishes the byte carrier and callson_reconnect(), after which the nexttick()replays the unacked outbox suffix from the peer ack cursor and re-advertises the receiver cursor. (A sink failure, by contrast, is retain-and-stall, not aDriverError— it is reported throughProgress/stall signals so the host can back off.)
Because the seam has no wire representation, it adds no conformance fixture: the reliable-sync
fixtures already pin the driver’s observable behavior (gap→resync convergence, outbox replay,
idempotent redelivery) at the message-sequence level, which is the correct abstraction — the seam
sits deliberately below it. This is also why the seam is not formalized in ReliableSync.lean:
“send/receive a frame” has no algebraic content to prove; the invariants that matter
(resync_convergence, outbox_at_least_once_exactly_once_effect, the liveness lattice joins) are
proven over frame sequences, above the transport.
Liveness cells: OR-set and LWW (#lzsync-liveness)
Cross-process liveness — “editor pid X has doc Y open”, “pid X holds the owner lease” — is carried as CRDT cells on the CrdtSync plane, not as a bespoke frame, so it inherits that plane’s idempotent, frontier-resumable, re-delivery-safe convergence. Two register shapes cover the liveness needs:
- OR-set membership — the open-set of
(doc, pid)presence. An observed-remove set: a(doc, pid)is present iff some add-tag for it is not shadowed by a remove-tag that observed that add. This gives the exact “add wins over a concurrent stale remove” bias liveness needs (a re-open concurrent with a lagging close keeps the doc open), and re-delivery of an add/remove is a lattice join → idempotent. Whole-editor death removes every(doc, pid)for that pid. - LWW liveness flag — a per-pid
alive: booland the owner lease as HLC-stamped last-writer-wins registers (the CRDT plane’s default register, § Cell register types). The OS process-exit event writesalive[pid] = false; the highest-stamp write wins, and a stale re-assert is dominated.
Normative semantics:
- Frontier-resumable. A liveness op that fails to send while the peer is down is re-sent on
reconnect from the
CrdtSync.frontier(the plane is already “safe to resend”); no liveness state is lost across a disconnect. This is what lets the open-set/lease survive a controller recycle without the editor re-announcing. - Derived authority is reactive. “Is doc Y live” = any present
(doc, pid)in the OR-set whosealive[pid]is true — a derived aggregate over the liveness keyed map (the#lzfamilysyncmaterialize-on-ingest + derived-count contract). Onealive[pid] = falsewrite fans out to every doc that pid held (whole-editor death cascade), reactively. - Idempotent + convergent. OR-set join and LWW join are semilattice joins, so out-of-order,
duplicated, or batched delivery all converge and re-delivery is a no-op (proven
ReliableSync.crdt_liveness_convergence_under_retry, building on the existing CRDT lattice proofs). - Per-doc isolation. Liveness keys are namespaced
(document_hash, …)like every other keyed op, so a stale overlay for doc B cannot flip doc A’s authority.
Partition & eviction (#lzsync-partition-eviction)
A slow peer, a partitioned peer, and a dead peer are three different states, and dropping a peer is the last rung of an escalation ladder — never the first response to a missed ack. This section fixes when a peer is dropped, how the network is isolated from one bad peer, and how a dropped peer rejoins.
Per-peer outbox isolation (normative). In a multi-peer fan-out each peer MUST have its own
DurableOutbox and ack cursor. A single shared outbox would make the
slowest peer the network’s pace-setter (head-of-line blocking) and let one partition stall every
peer. With a per-peer outbox, a slow or partitioned peer fills only its own cursor-queue; fast
peers are untouched, and the drop decision is per-peer, not network-wide. This is what lets
backpressure (§ Backpressure & outbox coalescing)
be contained rather than contagious.
Escalation ladder. A degraded peer is handled in this order; each rung is exhausted before the next:
| Rung | Trigger | Action | Drops data? |
|---|---|---|---|
| 1. Backpressure | consumer behind, outbox filling | is_full throttles the producer on that peer’s channel | no |
| 2. Coalesce / suspend / shed | outbox at watermark | state cell → collapse suffix to one frame; op-log queue → suspend the producer, or shed (drop-oldest/drop-newest) if the source can’t stop | state: no; queue: only on explicit shed |
| 3. Retain + replay | send/recv error (partition) | keep the unacked suffix; replay from the peer cursor on on_reconnect | no |
| 4. Evict | liveness lease expiry, or bounded outbox exceeded by an un-coalescible op-log with an un-stoppable source | remove the peer’s OR-set presence, reclaim its outbox | reclaims retained frames; peer full-resyncs on return |
Eviction is gated on the liveness lease, not on missed acks. A peer is dropped only when its
OR-set / LWW liveness lease expires (the
partition/death signal) — or, for a persistently-slow op-log peer, when its own bounded outbox is
exceeded and neither coalescing (a queue cannot state-supersede coalesce) nor producer-suspension
(the source cannot stop) can bound it. Eviction MUST be observable (it removes the peer’s (doc, pid)
presence, a visible OR-set remove), never silent. A state peer effectively never reaches rung 4 on
slowness alone: coalescing collapses its backlog to one frame, so a slow state peer costs one snapshot,
not an eviction.
A returning peer rejoins as fresh. An evicted peer that reconnects is a fresh receiver (state
empty, last_epoch = 0): it MUST full-resync by adopting a Snapshot, not replay from a stale cursor
(its retained frames were reclaimed on eviction). Convergence is lossless — the fresh receiver reaches
the sender’s full state (ReliableSync.evicted_peer_resyncs_fresh, a corollary of
resync_convergence). The OR-set add-wins-over-stale-remove bias
(ReliableSync.orset_add_wins_over_stale_remove) protects the race where a peer returns just as its
lease expires: the rejoin add is not shadowed by the lagging eviction remove, so a peer that is coming
back is not wrongly dropped.
Partition rule differs for queues (CAP). Because a queue’s total order is authoritative (a single
sequencer in SPSC) and has no idempotent join, divergent queue order across a partition cannot
losslessly converge — a queue cannot be AP-and-lossless. So a distributed (consensus-backed) queue
favors CP under partition: the minority side (no quorum) blocks writes rather than dropping
the peer — it refuses writes it could not later reconcile, and rejoins the majority order on heal. You
do not “drop the partition”; you decline minority writes and evict individual peers only on lease
expiry. State cells (LWW/OR-set) are the opposite: their join converges, so they MAY stay AP —
accept writes on both sides of the partition and merge on heal. This is why distribution of a
QueueCell is a consensus storage-backend property (the
distributed-queue PRD / RaftQueueStorage), while liveness/register cells
distribute on the plain CrdtSync plane. See § QueueCell op-log delta form.
Conformance
New fixtures under conformance/reliable-sync/ pin the protocol
cross-language (rs/js/kt):
| Fixture | Pins |
|---|---|
resync_gap_converge.json | drop a delta suffix → RequestSnapshot → apply Snapshot → same graph as the no-drop receiver (ResyncCoordinator decision table + convergence) |
outbox_replay_after_crash.json | append-before-send, replay-from-cursor after a simulated crash, ack_through retention, exactly-once effect under replay |
idempotent_redelivery.json | a re-delivered (base_epoch < last_epoch) delta is Ignored; net state unchanged |
multi_epoch_delta.json | a Delta with epoch > base_epoch + 1 applies equal to the unit-delta fold; atomic last_epoch advance |
liveness_orset_lww.json | OR-set open-set membership + LWW alive/lease; whole-editor-death cascade; derived live-doc aggregate converges under retry/re-delivery |
coalesce_bounds_outbox.json | state outbox collapses an unacked delta suffix to one Snapshot (retained depth → 1) with the receiver reaching the same graph as the full-run receiver; op-log outbox declines snapshot-coalesce and fuses a run into one batch (order-preserving); non-ack fill / ack drain of the cursor-queue (#lzsync-backpressure) |
liveness_lease_eviction.json | escalation ladder (backpressure → coalesce/suspend/shed → retain+replay → evict); per-peer outbox isolation; eviction gated on liveness-lease expiry not missed acks; evicted peer rejoins fresh + full-resync (evicted_peer_resyncs_fresh, add-wins race); distributed-queue CP minority-blocks-writes vs state-cell AP (#lzsync-partition-eviction) |
Every fixture’s frames MUST round-trip through both json and msgpack. The ResyncCoordinator,
DurableOutbox, and liveness models are the shared cross-language pins; lazily-formal
ReliableSync.lean is the correctness backstop the implementations must match.
Permission Boundary (RemoteOp)
Only nodes on the per-peer allowlist are serialized into a snapshot or delta. Non-allowlisted nodes are omitted entirely (not even as Opaque).
RemoteOp
RemoteOp = { kind: OpKind, node: NodeId }
OpKind = "read" | "write" | "trigger_effect"
- Three kinds are gated independently: a read grant never implies write or effect-trigger.
PeerPermissionsis default-deny per-peer allowlist.filter_readable(peer, nodes)drops non-readable nodes from results before serialization.
Protobuf graph-boundary interoperability
Protobuf is an optional, capability-negotiated encoding of Lazily’s graph boundary algebra. It does not own a second graph runtime. The canonical semantics remain in this specification and its executable conformance traces; canonical JSON remains the diagnostic and fixture representation, and msgpack remains part of the existing interop matrix.
The canonical schema is
proto/lazily/graph_boundary/v1/graph_boundary.proto.
Its reviewed semantic classification is
proto/field-ledger.json.
Ordinary text mutation uses GraphInput.cell_text_splice, bounded to one stable
cell with a local UTF-8 offset and expected revision. bootstrap_snapshot is a
different oneof variant and is legal only for bootstrap, explicit recovery, or
checkpoint compaction. A cache or native-library reload is therefore never
promoted into operator mutation authority.
Peers advertise the protobuf codec, the protobuf-graph-boundary-v1 feature,
and a compatible protocol range before exchanging these envelopes. The feature
is disabled unless both sides advertise it. Unsupported versions, unknown
semantic enum values, stale generations or epochs, sequence gaps, and invalid
snapshot purposes fail closed at admission. Duplicate sequences are idempotent.
Logical hashes are computed from the existing canonical logical representation, never raw Protobuf bytes. Protobuf map ordering, unknown-field preservation, and implementation-specific serialization make byte equality unsuitable as logical identity.
The six canonical traces in
conformance/protobuf/graph_boundary_traces.json
pin partial typing, cross-cell bounds, cache fencing, native reload behavior,
duplicate/reordered delivery, and the snapshot/mutation variant boundary.
The binding parity ledger in proto/field-ledger.json records Rust, Kotlin, and
TypeScript as the reproducible-generation pilot. Python, Go, C++, Dart, Zig, and
C# remain capability-gated known-uncovered findings until their native
generators and reducers replay the same logical traces.
Assertion-block schemas
Canonical conformance claims use three historical object names:
assertions, expect, and expected. They are executable contract data, not
free-form fixture metadata.
schemas/assertion-blocks.json gives every such object a fail-closed Draft
2020-12 schema. Each route is identified by its fixture-relative path and a JSON
pointer whose array indexes are normalized to *. Route schemas:
- reject unknown keys with
additionalProperties: false; - require every key present in all examples at that route;
- validate scalar, array, object, and nested value shapes;
- cover every fixture area, including families without a whole-fixture schema;
- are checked in both directions, so an unvalidated block and a stale unused schema route both fail.
The file is generated from reviewed corpus state, but it is intentionally checked in. A fixture edit therefore has to carry a visible schema diff:
python3 scripts/gen_assertion_block_schema.py
make check runs the generator in --check mode and mutation-checks unknown
keys, missing required keys, wrong value types, and unknown fixture routes. This
guard complements binding-side read/assert ledgers: JSON Schema constrains what
a fixture may say, while the runtime ledgers prove a binding actually consumed
and asserted what the fixture said.
Causal Receipts
CausalReceipt is lazily’s generic outcome projection for commands and effect
requests keyed by a stable causation_id.
It is intentionally not a transport ACK. observed and accepted are
non-terminal receipt outcomes; they can record that a peer saw or queued work.
applied and rejected are terminal outcomes. Domain-specific facts may refine
those terminal outcomes, but they should not invent a delivery-ACK authority.
{
"description": "Causal receipts fold by causation id and generation. observed/accepted are non-terminal; applied/rejected are terminal; stale generations are ignored by the current projection.",
"protocol_version": 1,
"kind": "Receipt",
"model": "CausalReceipt",
"assertions": {
"receipt_count": 4,
"current_generation": 7,
"causation_id": "patch-123",
"terminal_outcome": "applied",
"stale_receipt_ids": ["receipt-stale"],
"nonterminal_outcomes": ["observed", "accepted"]
},
"wire": {
"CausalReceipts": {
"receipts": [
{
"receipt_id": "receipt-observed",
"causation_id": "patch-123",
"observer": "editor",
"generation": 7,
"outcome": "observed",
"reason": null,
"payload_hash": null
},
{
"receipt_id": "receipt-accepted",
"causation_id": "patch-123",
"observer": "editor",
"generation": 7,
"outcome": "accepted",
"reason": null,
"payload_hash": null
},
{
"receipt_id": "receipt-applied",
"causation_id": "patch-123",
"observer": "editor",
"generation": 7,
"outcome": "applied",
"reason": null,
"payload_hash": "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
},
{
"receipt_id": "receipt-stale",
"causation_id": "patch-123",
"observer": "editor",
"generation": 6,
"outcome": "rejected",
"reason": "stale generation",
"payload_hash": null
}
]
}
}
}
The normative field list and projection rules live in
protocol.md § Causal Receipts. The schema is
schemas/receipts.json.
Command / RPC Message Plane
Editor and runtime integrations need to issue commands — Run Agent Doc,
sync, focus, save, session operations — with a single reusable admission,
dedupe, cancellation, generation-guard, progress, and reconnect story. Without
one, every caller reinvents in-flight/dedupe/supersede/retry/timeout logic, and
accepted / queued feedback gets mistaken for terminal success.
lazily’s command plane (command-plane-v1) is that shared substrate. It is an
additive sibling to Snapshot / Delta / CrdtSync, not a replacement:
command frames can ride the same transports and reflect into the normal state
graph, but they carry command traffic, not cell state.
The four frames
| Frame | Role |
|---|---|
CommandSubmit | Submit a command: envelope + domain payload (an IpcValue) |
CommandCancel | Preempt a still-non-terminal command by command_id |
CommandEvents | Progress/detail events (UX + diagnostics only, never proof) |
CommandProjection | Folded, queryable command state; also the reconnect resync image |
lazily owns the envelope; the namespace owns the payload. lazily never
decodes payload. Agent-doc publishes its own payload schemas
(agent-doc.editor_route.v1, agent-doc.sync_tmux_layout.v1, …) and only
references lazily’s envelope.
Progress is not proof
The single hard rule: terminal authority is the causal receipt. A command is
terminal only when a terminal CausalReceipt for its
command_id folds in (applied, or rejected — including the cancelled /
superseded / timed_out reasons). observed / accepted / started /
queued-admission events are non-terminal progress. A transport ACK is never
terminal.
This keeps command events from becoming a second proof system. Events may carry queue position, retry advice, or copied CLI output; the effect still folds through receipts and domain facts.
RPC is a facade
call / submit / cancel / observe / projection are implemented entirely
over the four frames:
await client.call("agent-doc.editor_route", payload, {
commandId,
idempotencyKey: "project-root:plan.md:run",
authorityGeneration: 42,
deadlineMs: 120000,
policy: { dedupe: "same_idempotency_key", supersede: false, cancelOnPreempt: true }
});
call resolves only on a terminal causal-receipt projection. A network ACK,
controller admission, or accepted / queued event never resolves a unary
call. submit returns the command_id for callers that manage events and
projection themselves. Reconnect uses CommandProjection; a call replays only
when the idempotency policy says replay is safe.
Rules
- Generation guards — events/receipts outside the command’s current authority generation are ignored (kept only as audit data).
- Idempotency — replaying a submit/event/receipt with a known id is a no-op.
- Cancel before terminal only — a cancel after
appliedis ignored. - Terminal conflict fails closed —
appliedvsrejectedat the same generation is not resolved by winner selection. - Reconnect equivalence — folding a
CommandProjectionequals folding the events and receipts it summarizes.
The normative field list and rules live in
protocol.md § Command / RPC Message Plane.
The schema is schemas/message-passing.json.
Conformance fixtures live in conformance/message-passing/; each binding
replays them through its CommandProjection reducer and RPC facade.
{
"protocol_version": 1,
"kind": "Command",
"model": "CommandProjection",
"description": "observed/accepted events are progress only; the command becomes terminal ONLY when the applied CausalReceipt folds in.",
"frames": [
{
"schema": "message-passing",
"wire": {
"CommandSubmit": {
"command_id": "cmd-run-1",
"causation_id": "cmd-run-1",
"source": "vscode-plugin",
"target": "project-controller",
"namespace": "agent-doc",
"name": "editor_route",
"authority_generation": 42,
"idempotency_key": "project-root:plan.md:run",
"deadline_ms": 120000,
"policy": {
"dedupe": "same_idempotency_key",
"supersede": false,
"cancel_on_preempt": true
},
"payload_type": "agent-doc.editor_route.v1",
"payload_hash": "sha256:f753816ec570dcaa98e80dafcf975b020a5ccc9db551c69e121567a48ee16346",
"payload": {
"Inline": [
123,
34,
102,
105,
108,
101,
34,
58,
34,
47,
104,
111,
109,
101,
47,
117,
115,
101,
114,
47,
112,
114,
111,
106,
101,
99,
116,
47,
112,
108,
97,
110,
46,
109,
100,
34,
44,
34,
114,
101,
108,
97,
116,
105,
118,
101,
95,
112,
97,
116,
104,
34,
58,
34,
112,
108,
97,
110,
46,
109,
100,
34,
44,
34,
100,
105,
115,
112,
97,
116,
99,
104,
95,
111,
110,
108,
121,
34,
58,
102,
97,
108,
115,
101,
44,
34,
112,
108,
97,
105,
110,
95,
116,
114,
105,
103,
103,
101,
114,
34,
58,
102,
97,
108,
115,
101,
44,
34,
119,
97,
105,
116,
95,
98,
117,
100,
103,
101,
116,
95,
109,
115,
34,
58,
49,
50,
48,
48,
48,
48,
44,
34,
108,
97,
121,
111,
117,
116,
95,
97,
114,
103,
115,
34,
58,
123,
34,
99,
111,
108,
117,
109,
110,
115,
34,
58,
50,
44,
34,
102,
111,
99,
117,
115,
34,
58,
116,
114,
117,
101,
125,
44,
34,
114,
111,
117,
116,
101,
95,
107,
101,
121,
34,
58,
34,
112,
114,
111,
106,
101,
99,
116,
45,
114,
111,
111,
116,
58,
112,
108,
97,
110,
46,
109,
100,
58,
114,
117,
110,
34,
44,
34,
101,
100,
105,
116,
111,
114,
95,
97,
116,
116,
101,
109,
112,
116,
95,
105,
100,
34,
58,
34,
97,
116,
116,
101,
109,
112,
116,
45,
55,
34,
125
]
},
"required_features": [
"causal-receipts",
"command-events"
]
}
}
},
{
"schema": "message-passing",
"wire": {
"CommandEvents": {
"events": [
{
"event_id": "ev-1",
"command_id": "cmd-run-1",
"kind": "observed",
"generation": 42,
"detail": null
},
{
"event_id": "ev-2",
"command_id": "cmd-run-1",
"kind": "accepted",
"generation": 42,
"detail": "queued at position 1"
},
{
"event_id": "ev-3",
"command_id": "cmd-run-1",
"kind": "started",
"generation": 42,
"detail": null
}
]
}
}
},
{
"schema": "receipts",
"wire": {
"CausalReceipts": {
"receipts": [
{
"receipt_id": "rcpt-1",
"causation_id": "cmd-run-1",
"observer": "project-controller",
"generation": 42,
"outcome": "applied",
"reason": null,
"payload_hash": "sha256:f753816ec570dcaa98e80dafcf975b020a5ccc9db551c69e121567a48ee16346"
}
]
}
}
}
],
"expect": {
"projection": {
"generation": 42,
"commands": [
{
"command_id": "cmd-run-1",
"status": "applied",
"terminal": true,
"generation": 42,
"reason": null,
"terminal_receipt_id": "rcpt-1",
"last_event_id": "ev-3"
}
]
},
"terminal_after_frame_index": 2
}
}
Cross-Process Zero-Copy Transport
#lzzcpy— large payloads cross the IPC plane as descriptors, not copies.
A Snapshot / Delta / CrdtSync message may carry large cell/slot payloads
(an Arrow record-batch, an image, a serialized sub-document). Copying those bytes
through the wire codec on every hop is the dominant cost of a distributed lazily
deployment. The zero-copy transport instead spills a large payload to a blob
backend and ships a small Descriptor; the receiver resolves the descriptor
against the same backend and reads the bytes in place — no copy, no checksum
recompute.
This chapter defines the transport model, the pluggable backend (adapter)
contract, and the wire Descriptor. The invariants are proven in
lazily-formal/LazilyFormal/ZeroCopyTransport.lean
(spill-then-resolve identity, backend isolation, ABA/generation safety, checksum
integrity) and pinned by
conformance/delta_zero_copy_arrow.json.
Model
producer receiver
──────── ────────
bytes ──spill──▶ backend.write ──▶ Descriptor Descriptor ──resolve──▶ backend.read_view ──▶ bytes (view)
(mint id+gen+csum) │ (kind routes to the right backend)
└──────── wire (msgpack) ──────▶
- Spill (producer). For an
IpcValue/NodeStatepayload above a session-defined threshold, the producer callsbackend.write(bytes)and gets aDescriptor({kind, offset, len, generation, epoch, checksum}). It puts the descriptor in the message asSharedBlobinstead ofInlinebytes. - Wire. Only the descriptor crosses the codec — the message stays small.
- Resolve (receiver). The receiver reads
descriptor.kind, routes to the matching backend, and callsbackend.read_view(descriptor)→ aconstview of the backend’s own bytes. No copy, no checksum recompute (the checksum was computed once at write and is validated against the cached value).
Threshold policy: payloads below the threshold stay Inline (copied through the
codec — cheaper than a backend round-trip for tiny values). The threshold is a
session/deployment knob, not a protocol constant.
The wire Descriptor
The descriptor is ShmBlobRef (schemas/defs.json) extended
with an optional backend discriminator:
| field | type | meaning |
|---|---|---|
offset, len | u64 | byte range within the backend’s resolved buffer |
generation | u64 | ABA guard — a slot reused at a later generation is not misread |
epoch | u64 | validity epoch (advanced on backend compaction/restart) |
checksum | u64 | FNV-1a-64 over the bytes (computed once at write, validated at read) |
backend | enum shm | arrow | in_process (optional, default shm) | which pluggable backend resolves this descriptor |
backend is optional and defaults to shm, so every legacy descriptor
validates unchanged — the transport is a strict superset of the pre-existing
shared-memory blob path. A receiver routes resolution by kind: a shm
descriptor never resolves in an Arrow table and vice versa (the
resolve_wrong_backend theorem).
Pluggable backends (adapters)
A backend is anything that satisfies the blob-backend contract:
| operation | contract |
|---|---|
write(bytes) → Descriptor | mint a fresh id at the current generation/epoch, store the bytes immutably, return a descriptor whose checksum is the bytes’ FNV-1a-64 |
read_view(Descriptor) → const bytes* | return the stored bytes iff kind + id + generation + epoch + checksum all match; nullptr otherwise. No copy, no recompute |
| lifecycle | advance epoch on compaction/restart; never mutate a stored buffer in place (entries are immutable + stable-addressed for the lifetime a descriptor may reference them) |
Because the contract is stated only over a backend’s issued-blob table + the
read_view lookup, the transport theorems hold uniformly for every backend
that maintains the contract — this is the universal guarantee no single-adapter
fixture can establish. Three backends ship / are anticipated:
backend | what holds the bytes | cross-process? | typical use |
|---|---|---|---|
shm | POSIX shared-memory region (shm_open + mmap) | yes (same host) | the default — host ↔ binding / peer ↔ peer on one host |
arrow | Apache Arrow IPC stream / Flight-resolved buffer | yes (Arrow’s zero-copy columnar IPC) | analytics / columnar payloads — the bytes are an Arrow IPC stream the receiver imports as an Array/RecordBatch with no copy |
in_process | an in-process arena (single address space) | no | the FFI host ↔ a binding loaded in the same process (editor plugin) |
An Apache Arrow adapter implements the contract by holding spilled payloads
as Arrow buffers and resolving a descriptor to the buffer’s raw bytes (or, for
columnar consumers, directly to the Arrow Array — the descriptor’s bytes are
an Arrow IPC stream). Because Arrow’s IPC format is itself zero-copy across a
shared buffer, shm and arrow compose: an Arrow batch can live in a shm
region and be resolved by either backend. New backends (e.g. a RDMA/verbs
adapter, a Cuda IPC adapter) plug in by implementing the contract and adding a
backend enum value — no transport or codec change.
Backend-agnostic invariants (proven)
The formal model parameterises over an abstract backend and proves, for any
backend satisfying the contract (ZeroCopyTransport.lean):
resolve_write/transport_roundtrip— resolving the descriptor a backend minted viawritereturns exactly the bytes written. The consumer reads the backend’s own bytes, not a copy: the end-to-end zero-copy guarantee.resolve_wrong_backend— a descriptor of onekindnever resolves against a different backend’s table → receivers route bykind.resolve_stale_generation— a slot reused at a later generation (or a stale ref to a freed slot) does not resolve against the new occupant (ABA safety viageneration).resolve_corrupt_checksum— a descriptor corrupted in transit is rejected rather than resolving to the wrong bytes.
Conformance
conformance/delta_zero_copy_arrow.json— aDeltawhoseSlotValuepayload is aSharedBlobwithbackend: "arrow", validating the optional discriminator againstschemas/delta.json.conformance/delta_shared_blob.json— the legacybackend-absent (=shm) form, unchanged → backward compatibility.conformance/codec/blob_backend_discriminator.json— the decoder half of the discriminator (#lzblobbackendstrict, protocol.md § Shared-memory payload path). The two fixtures above are both conforming frames, so between them they establish that a known backend round-trips and an absent one defaults — neither says what happens to a token this build does not know. That gap is where five of nine bindings independently decided to normalize an unknownbackendtoshm, which routes a non-shmdescriptor into theshmtable and leavesresolve_wrong_backendto be discharged by the checksum instead of by routing. The fixture makes the refusal executable and pinsarrowalongside it, so a binding cannot pass by ignoring the discriminator.
Relationship to the wire codec
The transport is codec-agnostic but pairs with the msgpack codec: spill replaces
Inline bytes (which the codec would copy) with a small SharedBlob descriptor
(which the codec serializes as a handful of integers). The codec already
distinguishes Inline vs SharedBlob; the transport adds the policy (when
to spill) and the backend contract (how SharedBlob is resolved without a
copy). See Wire Protocol.
Conformance Fixtures
The conformance/ directory contains canonical test fixtures that all IPC-capable bindings must
validate against. Each binding’s CI should deserialize the wire field, run the assertions, and
re-serialize to confirm round-trip fidelity.
Fixture schema
{
"description": "Human-readable summary",
"protocol_version": 1,
"kind": "Snapshot" | "Delta" | "Receipt",
"assertions": {
"prose": ["…keys below that are PARAGRAPHS, not comparable values…"],
"…language-agnostic field checks…": "…"
},
"wire": { "…canonical protocol JSON…" }
}
assertions.prose declares which sibling keys state an obligation in English rather than
carrying a comparable value. A runner discharges those by naming the executable keys that
prove them — see Prose assertion keys.
Fixture discriminability (#lzfixturediscrim)
Opening a fixture, consuming every key, and comparing every executable value still does not
prove that the fixture data can distinguish the behavior it names. For example, every ordering
predicate returns true for an empty or singleton roster. A runner can faithfully assert
roster_sorted_ascending: true while a library that reverses its roster order remains green.
scripts/check-fixture-discriminability.mjs audits this fourth rung:
- Non-boolean routes are exact-value oracles. A changed observation differs from the canonical value.
- Boolean routes carrying both
trueandfalsesomewhere at the same normalized fixture pointer have an in-corpus control that rejects either constant mutant. - Every remaining single-valued boolean route must appear in
audits/fixture-discriminability.json. A claim is eithermutation-killed, with the binding, library-source mutation, command, observed failure, and a fixture witness, oruntestedwith a reason. Missing and stale ledger entries fail CI.
untested is deliberately not a weaker spelling of pass. It records that the corpus claim has
no registered library mutation proof, so coverage reports and follow-up audits cannot silently
treat an asserted predicate as behavior-discriminating.
The signaling transcript is the first recorded proof. Three peers join; the third welcome
must carry the two-element roster [1, 2]. Reversing SignalingRoom.roster in lazily-js
now fails the canonical replay with actual [2, 1], while removing self-filtering and stamping from with
the target rather than the registered sender fail the other two signaling assertions.
To initialize ledger entries after intentionally adding claims:
node scripts/check-fixture-discriminability.mjs --write-initial-ledger
The initializer preserves existing evidence and creates new entries as explicit untested
claims. Replace that status only after running the cited library-source mutant and confirming
that a canonical fixture—not merely an independent unit test—reddens.
Custom assertion callback consumption (#lzassertwithseam)
Every binding exposes a custom assertion helper for relations that plain equality cannot express: tolerances, containment, decoding, and derived projections. The helper passes the fixture value to a callback. Calling the callback is not itself proof that the callback used that value: an ignored parameter previously marked the key asserted while comparing nothing.
A binding MUST therefore:
- require the callback to expose and syntactically read its fixture-value parameter;
- record the key as asserted only after the callback completes successfully; and
- keep extraction explicit by using a projection helper (for example Python’s
assert_key_into) when the caller performs the comparison after the helper returns.
scripts/check-assert-with-consumption.py enforces rule 1 for every maintained binding.
scripts/check-assertion-ordering.py runs it as part of the existing binding conformance
gate, so comments and string literals do not count as reads and unsupported callback forms
fail closed. Its self-test includes an accepted and an ignored callback for every supported
language. The runtime helper ordering enforces rule 2: a callback that throws, rejects, or
returns a failing verdict cannot leave the assertion ledger green.
Current fixtures
| Fixture | Kind | Description |
|---|---|---|
snapshot_minimal.json | Snapshot | One payload node, no edges |
snapshot_multi_node.json | Snapshot | Multiple nodes and edges |
snapshot_shared_blob.json | Snapshot | SharedBlob node state |
delta_sequential.json | Delta | All 7 DeltaOp variants, sequential |
delta_non_sequential.json | Delta | Non-sequential delta with gap |
delta_shared_blob.json | Delta | CellSet/SlotValue with SharedBlob |
receipts/causal_receipts.json | Receipt | Causal receipt projection with non-terminal and terminal outcomes |
Publishing a corpus change (#lzspecpushbeforebindings)
Push the corpus change to lazily-spec FIRST, then verify and push the bindings.
Never the other way round, however green the bindings look locally.
The asymmetry is the whole reason. Local verification resolves the corpus through the
sibling working tree (../lazily-spec/conformance/...), so it sees your change the
moment you save it. Every binding’s CI instead clones published main — for example
lazily-cs/.github/workflows/ci.yml runs
git clone --depth 1 https://github.com/lazily-hub/lazily-spec.git ../lazily-spec.
So a corpus change that is green in nine local checkouts is invisible to all nine CI runs
until it is pushed, and any binding that pins a scenario or fixture count fails on the
mismatch.
That is not hypothetical. Landing the SeqCrdt fork-clock scenarios
(#lzspecforkclockfixture) in the wrong order turned lazily-cs red with
Expected: 8 / Actual: 6 — its runner census correctly counted the eight scenarios in the
working tree while CI replayed the published six — and forced lazily-cpp to pin
MIN_SCENARIOS to the published 144 rather than the local 146, then raise it again in
a second commit once the push landed.
The order that works:
- Fix the bindings’ behaviour first if the new fixture would otherwise redden them, but do not push a binding whose census (scenario counts, coverage floors) encodes the new corpus.
- Commit and push the
lazily-specchange, including any vendored mirrors re-synced byscripts/sync-conformance-fixtures.mjs --sync. - Only then push binding changes that assume the new corpus, and re-run any CI that failed against the old one.
A floor or count in a binding must always describe what CI’s clone guarantees, never
what your working tree happens to hold. make check here warns when the local corpus is
ahead of origin/main, which is the moment this rule applies — see
scripts/check-corpus-published.mjs.
Adding a new binding
Copy the fixture-loading pattern from lazily-rs/tests/conformance.rs. Each test should:
- Load the fixture.
- Parse the
wirefield into the binding’s nativeIpcMessagetype. - Assert the
assertionsfields. - Re-serialize and compare for byte-for-byte round-trip fidelity, subject to the equivalence exemptions below.
Round-trip equivalence exemptions
Byte-for-byte comparison is the default, but it is not the contract for a field whose schema declares two encodings equivalent. Where the schema says a field may be omitted or sent in some canonical empty form, a binding MUST NOT be required to reproduce the sender’s choice: both encodings decode to the same value, so a binding is free to emit either one.
For such fields the round-trip comparison is semantic: normalize the fixture’s wire and
the binding’s re-serialized output to the same canonical form (fill in the declared default for
an absent field) before comparing. All other fields remain byte-for-byte.
Exempt fields:
| Field | Equivalent encodings | Declared by |
|---|---|---|
CrdtSync.frontier | omitted ≡ [] — “unchanged since the last accepted frame” (#lzspecfrontiersuppress) | schemas/distributed.json (required is ["ops"] only) |
A binding MUST accept an omitted frontier on decode and treat it as empty. Rejecting the
absent form is a conformance failure; re-emitting it as [] is not.
Prose assertion keys (#lzprosekeyconvention)
An assertions block mixes two kinds of key. Most carry a value a runner can compare
against observed behaviour — a list, a count, a vocabulary. A few carry an English
paragraph that states an obligation and nothing comparable: clause, anti_vacuity,
null_form, theorem, note. This section says what a runner MUST do with the second
kind, because nothing did, and the nine bindings each decided.
The failure this closes
Replaying blob_backend_discriminator.json v2 — which added four new paragraphs
(backend_form_vocabulary, null_form, non_string_form, epoch_disambiguation) —
produced four different treatments of the same four keys:
| Binding | Treatment |
|---|---|
| lazily-js | excused all four, its own assert-key.js warning against comparing an English paragraph to a literal |
| lazily-py, lazily-dart, lazily-go, lazily-kt, lazily-cs, lazily-zig | excused them with individually-worded reasons naming the assertion that discharges each — falsifiable in principle, checked by nothing |
| lazily-rs | marked them Expect::prose, a third tracker state exempt from every check, which requires a reason and then discards it |
| lazily-cpp | asserted all four against tallies computed from the run |
Every one is defensible alone. That is the point: this is the same shape as the 5-2 split the blob-backend clause itself came from — an undocumented default and a deliberate choice are indistinguishable from the outside, and so are four deliberate ones.
Definition
A prose key is a key of a fixture’s top-level assertions block whose value is a
natural-language paragraph: it states an obligation and carries no value a runner can
compare against observed behaviour.
The corpus declares which keys those are, in assertions.prose — an array of sibling
key names. A binding MUST NOT decide for itself. The declaration is itself a key of the
block, so the existing consumption guards see it: a runner that ignores it fails with an
unconsumed key, which is what makes the rollout self-enforcing.
Prose nested inside a data key is not a prose key. assertions.outcomes maps a
vocabulary to English glosses; the assertion is the key set, and the parent key’s own
assertion discharges it.
The rule
A prose key is discharged, never asserted and never excused. To discharge it a runner names the executable assertion keys that carry its obligation, and its assertion-key tracker verifies the naming. A binding’s tracker MUST fail the run when:
-
a key listed in
assertions.proseis asserted — comparing a paragraph, or a tally derived from one, to an English string pins wording, not behaviour. A copy-edit reddens the run and a library regression does not.reject_obligationsays exactly this about error message formats; it applies no less to the paragraph stating it; -
a key listed in
assertions.proseis excused with free text — an unfalsifiable reason (“prose”, “explains why the wire is text/hex”) is indistinguishable from the undocumented default this clause exists to remove; -
a key not listed in
assertions.proseis discharged; -
the set of discharged keys differs from
assertions.prose— this is the comparison that consumesproseitself, and it is what makes a forgotten key fail rather than vanish; -
a discharge names no keys;
-
a discharge names a key that the same fixture’s run did not assert;
-
a discharge names a key that is itself prose, or names
prose. The second half is not redundant:prosenever lists itself, so without it rule 7 missesdischargedBy: ["prose"]— and rule 4’s own comparison marksproseasserted, so rule 6 would wave it through. A paragraph discharged by the declaration that it is a paragraph proves nothing. Seed the prose-name set withproseitself. -
an opened fixture whose block declares
prosenever reaches verification. Rules 1-7 are all satisfied over an empty population, so a fixture that is opened and then never replayed passes every one of them while proving nothing — the vacuity the corpus’s ownanti_vacuitykeys exist to name, reappearing in the guard meant to enforce them. Derive the required verifications from the corpus, never from a hand-kept count.
Evaluate rule 7 before rule 6. A paragraph can never be asserted — rule 1 forbids it — so a discharge naming another paragraph always also violates rule 6. Check 6 first and rule 7 becomes dead code that never reports, and the run fails with “names a key this run never asserted” when the real defect is “names a paragraph”. The numbering is not the evaluation order; this one pair is.
Rule 6 means ASSERTED, not merely satisfied: an excused key does not discharge anything,
because an excuse is precisely the absence of a comparison. A discharge naming a key the
fixture does not carry at all is a distinct failure — the discharge has rotted, exactly as a
stale excuse has. One executable key may discharge several paragraphs; decoded_backend
carries five in blob_backend_discriminator.json, and that is expected rather than a
collision.
Rule 6 is the whole convention: the excuse becomes falsifiable, because the tracker can
check it. “epoch_disambiguation is discharged by frame_epoch and blob_epoch” is a
claim about the run; “epoch_disambiguation is prose” is not.
A discharge may name a key that carries the obligation only by PROXY. theorem names a
Lean theorem in another repository; the run can only prove its consequence. Naming a proxy
is conforming. Naming a key that has nothing to do with the obligation is not, and no
tracker can tell the two apart — that judgement stays with review, which is why the
discharge is written at the call site where review sees it. wire_encoding is no longer a
proxy: every declaring scenario carries expect.wire_input_fnv1a64, and the runner hashes
the exact UTF-8 JSON bytes or decoded MessagePack bytes it passes to the library decoder.
Scope of “the same fixture’s run”
The discharge ledger is fixture-scoped, not block-scoped. An obligation stated in
assertions is routinely carried by a per-scenario expect key: epoch_disambiguation is
discharged by expect.frame_epoch and expect.blob_epoch, which are asserted long after
the assertions block itself is finished. A named key is therefore matched by key name,
in any block of that fixture, and verification happens when the fixture’s replay is
finished. A runner that never verifies MUST fail — an unverified discharge claim is
reported by the ledger’s own teardown, exactly as an unconsumed key is.
A block declaring prose MUST also carry at least one non-prose key. A block that is
entirely prose has nothing that could discharge it.
A block is identified by its path in the fixture, not by a human-written label. Rules 3
and 4 are block-local, so two blocks whose tracker labels happen to collide would merge
silently and each would satisfy the other’s declaration. Key them on assertions,
scenarios[3].expect, steps[7].expect — something the fixture determines.
Which rules are block-local and which are fixture-wide. The declaration is block-local:
each block owns its own prose array, and rules 3 and 4 compare a block’s discharged set
against that block’s array. Only the NAME MATCHING of rules 6 and 7 is fixture-wide, because
that is the half that has to reach a per-scenario expect key.
A “run” is one test, not one process. Where a fixture is replayed by several tests, the ledger is scoped to each and cleared at its verification. Unioning asserted keys across tests would let a discharge in one test be satisfied by an assertion in another, which is the same accident-of-collocation the fixture-scoped ledger exists to bound.
Arming the verification net is a LIFO hazard. Every cleanup mechanism the nine use —
Drop, t.Cleanup, addTearDown, IDisposable, a destructor — runs in reverse
registration order. A net armed by the first prose_key call therefore fires BEFORE a
verification the runner registered earlier in its body, and reports a false failure. Arm the
net from the same seam that already owns the block’s consumption check, which is
structurally guaranteed to run last.
Reserved annotation names
note, description and reason inside a per-step or per-scenario block are
annotations, exempt by name in every binding.
A corpus declaration overrides the by-name exemption, and a tracker MUST apply it first.
Both frame_roundtrip_json.json and frame_roundtrip_msgpack.json declare note prose. A
tracker that subtracts its reserved-name set before consulting assertions.prose makes that
declaration invisible to its consumption guards: the key is exempt from the unread guard and
exempt from the unasserted guard. Three of the nine hit this independently while
implementing the clause. Evaluate prose on the raw block, before any name-based exemption;
the exemption applies only to keys the block did not declare.
How bad the inversion is depends on where rule 4 reads from, and this is worth stating
precisely because one binding measured it rather than assuming. Where rule 4’s comparison
reads the RAW block — the declared set kept separately, never subtracted from — the
inversion degrades to a worse error message: the forgotten note is still caught, just by
the set comparison rather than by the guard that would have named it. Where rule 4 reads
through the exempted view, nothing is left: both fixtures skip the convention entirely and
the binding still reports conforming. Keep the declared set independent of the exemption and
the inversion cannot be fatal — but evaluate in the stated order anyway, because the guard
that names the key is the one a reader acts on.
Inside a block that declares prose, the name exemption is off entirely: the corpus wins,
so a note sitting in a declaring block but absent from its array needs an assertion or an
excuse like any other key. Everywhere else the exemption stands as-is.
Naming a discharge that discharges nothing
The tracker checks that a named key was asserted. It cannot check that the assertion proves the paragraph, and three bindings found the gap the same way: a key that compares the fixture to itself is asserted, satisfies rule 6, and discharges nothing. Two shapes recur.
scenario_count asserted against len(fixture["scenarios"]), and codecs / key_forms
compared to hand-written literals, are green over a runner that decodes nothing — which is
the exact vacuity anti_vacuity exists to name. Compare them against what the run really
replayed before naming them.
Worse, nodekey_null_leniency.json’s wire_encoding obligation — that the ABSENT and
explicit-null wire forms stay distinguishable into the runner — was dischargeable by
nothing at all in at least one binding: key ?? null collapses the two the instant the
value is decoded, and every key in that fixture’s expect blocks is identical for the
omitted and null families. The four null scenarios were the four omitted ones
wearing a different id. The fix is a control that reads the raw wire slot BEFORE the decoder
runs, which is what the sibling blob-backend runner already does for backend. Adding the
missing control is conforming; naming a key that merely happens to be asserted is not. That exemption is only safe while they
annotate; an annotation MUST NOT state an obligation, because a reserved name is a place no
runner can be made to discharge anything. scripts/check-prose-keys.mjs enforces this and
carries a both-directions allowlist of the instances that already do — five reactive-graph
step notes, each a real normative rule (teardown is idempotent, a stale cell handle whose id has been recycled MUST be a no-op) that no binding checks today. New instances redden.
Tracker API
The nine trackers differ in mechanism — a Drop guard, a global recorder plus a manifest
script, a t.Cleanup, an IDisposable — but the SPELLING is fixed here, because a
convention whose name drifts per binding is how four treatments of one rule went unnoticed
in the first place.
| Binding | Discharge | Fixture-end verification |
|---|---|---|
| lazily-rs | exp.prose_key("clause", &["backends", "scenario_count"]) | expect::verify_prose(fixture), armed by a ProseLedger guard |
| lazily-py | prose_key(block, "clause", discharged_by=["backends"]) | verify_prose(fixture) |
| lazily-js | proseKey(block, "clause", ["backends"]) | verifyProse(fixture) |
| lazily-go | proseKey(t, block, "clause", "backends") | verifyProse(t, fixture), registered with t.Cleanup |
| lazily-dart | proseKey(block, 'clause', dischargedBy: ['backends']) | verifyProse(fixture), registered with addTearDown |
| lazily-kt | proseKey("clause", listOf("backends")) | verifyProse(fixture) |
| lazily-cs | ProseKey("clause", "backends") | VerifyProse(fixture) |
| lazily-cpp | block.prose_key("clause", {"backends"}) | verify_prose(fixture) |
| lazily-zig | proseKey("clause", &.{"backends"}) | verifyProse(fixture) |
prose_key replaces whatever the binding did before — lazily-rs’s third prose() state
goes away rather than gaining a sibling, and the free-text excuse_key reasons written for
these keys are deleted, not kept alongside. Two paths to satisfy one key is the ambiguity
this clause removes.
What this does not check, and what is still open
A discharge is checked for truth, not for sufficiency. Rule 6 proves the named key was
asserted. Nothing proves it is relevant: discharging all nine of
blob_backend_discriminator.json’s paragraphs with scenario_count alone would satisfy
every rule. The excuse is now falsifiable, which it was not before, but it is not yet
load-bearing — that judgement stays with review, which is why the discharge is written at
the call site rather than in a table. Do not read a green run as “every paragraph is
proven”; read it as “no paragraph is discharged by a claim the run contradicts”.
generator is provenance metadata, not an assertion. A generated fixture that records
its source script places the path in the top-level generator field. It MUST NOT place that
field under assertions: a replay cannot observe which script emitted its input, so an
assertion tracker would have nothing executable to compare. Binding runners do not consume
top-level provenance metadata.
The former wire_encoding gap is executable (#lzwireencodingrunner).
check-prose-keys.mjs requires every wire_json to be raw parseable text, every
wire_msgpack_hex to be even-length lowercase hex, both codecs to be represented, and
every scenario’s expect.wire_input_fnv1a64 to match those exact bytes. Each binding then
computes the same digest over the buffer it passes to its library decoder and asserts that
key before decoding. A re-serialized value therefore changes the decoder-input witness
instead of silently satisfying a proxy discharge.
What is checked where
| Half | Where | What |
|---|---|---|
| Corpus | scripts/check-prose-keys.mjs (make prose-keys-check) | every paragraph declared, no stale or comparable entries, prose not self-listing, a declaring block carrying at least one non-prose key, no obligation hiding in a reserved annotation name |
| Binding | the binding’s own assertion-key tracker, at runtime | rules 1-7 above |
| Binding | the binding’s coverage / ledger guard, beside the fixture-open and scenario-replay rungs | rule 8 |
Rule 8’s row is separate on purpose. It cannot live in the test host: a test that never runs reports nothing, so the very run rule 8 exists to catch is the run that would have to report itself. It belongs where the other “did the suite actually do this?” rungs already live — the guard that reads the runtime manifest after the suite finishes.
The split is not arbitrary. Only the run knows which keys it asserted, so rule 6 cannot be checked from this repo; and only the corpus can settle which keys are prose, so leaving that to nine trackers is what produced four answers.
Object-valued assertion keys (#lzsubblockkeyset)
An assertion key whose value is a JSON object carries two obligations, not one: the value of each sub-field, and the sub-field key set. Bindings were discharging the first and not the second, which makes the object the null form one level down — a field added to it upstream is compared by nothing, and the fixture reports clean over the very change it exists to catch.
The failure this closes
arena_blob.json’s assertions.descriptor carries five sub-fields. Every binding that
replayed it compared those five by name and stopped. Planting a sixth key inside the object
left lazily-zig’s suite green while every scalar sibling in the same block reddened —
found only by the corpus perturbation pass, because no rung above it can see inside a key
it considers consumed. The consumption ledger saw descriptor read and asserted; the
read-but-not-asserted rung saw an assertion; the bind ledger saw a bound block. All three
were satisfied by a check that could not fail.
The rule
A binding’s assertion-key tracker MUST fail the run when an assertion key whose value is an object is consumed without its key set being checked. The tracker owns this, not the call site. Two ways to discharge it, and the tracker MUST recognise both:
- Descend. The tracker hands the caller a CHILD tracker bound to the object. The child owns the same unconsumed-key teardown the parent has, so a sub-field nothing reads fails exactly as an unconsumed top-level key does. This is the form to prefer: the obligation moves down rather than being restated.
- Compare the key set. The tracker compares the object’s key set against the set the
run actually produced, in both directions — a fixture token nothing replayed and a
replayed token the fixture omits are each failures. This is the form for a key whose
sub-fields are a vocabulary rather than data:
nodeid_exact_range.json’sassertions.outcomesmaps outcome tokens to English glosses, and the assertion is which tokens exist, not what the glosses say.
Anything else — a plain value comparison, a hand-written field-by-field check, a count of sub-fields — leaves the run reporting nothing about a field the corpus grows later. A per-call-site field count is not conformance. It relies on the next author remembering, which is the property this rung exists to remove; it is at best a stopgap and MUST be replaced by 1 or 2.
A key the corpus declares in assertions.prose is out of scope here: a paragraph is a
string, and prose nested inside a data key is not a prose key (see § Prose assertion keys).
The rule is not scoped to top-level assertions
It applies to every block a runner binds to its assertion-key tracker — expect,
expected, per-step, per-scenario, per-frame — because the defect is identical wherever
an object value is compared field by field. This matters because it is the one thing about
this class that is consistently underestimated: a scan of top-level assertions blocks
finds exactly two object-valued keys in the whole corpus, and the guard, once landed,
found between 14 and 26 distinct key shapes per binding. Do not scope the audit from a
corpus scan; land the guard and let it name the sites.
The recurring shapes, for orientation rather than as a checklist: invalidates, scopes,
receipts, reads, subscriptions, values, handle_stable, observe, states,
present, discovery, projection, frame, state_after, converged_nodes, text_on,
version_vector_on, order_on, get / get_on, final_state, after_publish,
authority, retry, dependents_of, readable, read. The ingress fixtures nest four
levels deep (invalidates.scopes.<key>.<reader kind>), and every level is a key set.
What is checked where
| Half | Where | What |
|---|---|---|
| Corpus | the fixture itself | which assertion keys have object values — the corpus decides, a binding never assumes |
| Binding | the binding’s assertion-key tracker, at runtime | an object-valued key consumed by neither 1 nor 2 fails the run |
Neither half is provable from the other. The corpus cannot know whether a runner descended, and a runner cannot be trusted to notice that a value it compared happened to be an object.
Validating a tracker that claims to enforce this
The guard is only as good as its own falsifiability, and this whole rung exists because a
check that cannot fail reads identically to one that passes. Plant an extra key inside each
object-valued assertion value in a scratch copy of the corpus — never lazily-spec in
place, where a probe reddens every other binding concurrently — and confirm the suite goes
RED. A tracker that reports clean over a planted sub-field has not implemented this section,
whatever its code says.
Assertion observation ordering (#lzassertordering)
An executable assertion has to remain reachable when the behavior it names is wrong. Two ordering rules follow:
- Fully evaluate and finish a fixture-owned assertion block, including its prose obligations, before applying runner-side coverage floors. A redundant floor may still guard the runner, but it must not preempt the fixture assertion that owns the falsifying corpus mutation.
- Evaluate an assertion about a run only after performing that run, and compare the declared rule with an observed runtime outcome. Checking a label or literal before ingest, decode, replay, or dispatch is not an assertion of the behavior the label names.
The semantic priority is therefore fixture assertion first, runner floor second.
This matters for diagnostics as well as coverage: a mutated scenario_count
must fail as scenario_count, not disappear behind an earlier hard-coded count.
scripts/check-assertion-ordering.py is the cross-binding static guard. Every
binding invokes its own configured pass from make check; missing, duplicated,
or renamed anchors fail closed. The guard checks order and attachment, not the
sufficiency of the runtime comparison. Mutation probes remain required for the
runtime assertion itself.
For distributed/anti_entropy_converge.json, resolution: max_stamp is tied to
the runtime state selected after ingest. Its conflict witness delivers the
greatest-stamp operation before a lower-stamp tail, so arrival-order resolution
and max-stamp resolution produce different observations.
Keyed cell collections conformance
The conformance/collections/ directory contains canonical fixtures for the
keyed cell collections layer, which is required
of every binding (see the Binding Conformance Matrix).
These are compute fixtures, not wire fixtures: a binding loads the initial state,
replays each step’s op, and asserts the expected effects (resulting order,
values, membership, and which reader classes — value / membership / order —
invalidate). The reconciliation fixture is declarative: diff prior → target and
assert the emitted minimal op set.
| Fixture | Covers |
|---|---|
collections/cellmap_independence.json | value / set-membership / order reactivity independence |
collections/cellmap_atomic_move.json | atomic ordered move keeps handle, bumps order once |
collections/dependency_reactive_availability.json | exact-key observation before publication, unrelated-key isolation, stable unavailable/available identity |
collections/keyed_reconciliation_lis.json | LIS move-minimized reconciliation; stable entries not invalidated |
collections/semtree_incremental.json | memoized semantic tree: ancestor-chain-only recompute, sibling isolation, memo guard |
collections/seqcrdt_convergence.json | move-aware sequence CRDT: single-LWW move, concurrent-move/value-edit independence, tombstone convergence, commutativity |
collections/mergecell_algebra.json | Source<T, M> merge algebra (#relaycell): KeepLatest/Sum/Max policies; per-op converged value + whether ⊕(old,op)==old suppresses the cascade (idempotent/identity no-op = free dedup); Cell ≡ Source<KeepLatest> |
collections/textcrdt_convergence.json | Fugue/RGA character CRDT: concurrent same-point inserts, sticky tombstone, commutative/idempotent merge, GC |
collections/textcrdt_delta_sync.json | TextCrdt delta sync (#lztextsync): version_vector (insert + tombstone ids), delta_since / apply_delta; bidirectional exchange convergence, whole-snapshot fork identity preservation, idempotent apply |
collections/stableid_alignment.json | manufactured text identity: anchors / content hashes / word-LCS similarity alignment |
collections/workqueue_competing_delivery.json | competing-consumer exclusive FIFO claims, delivery ownership, ack/nack, identity-preserving redelivery |
collections/workqueue_lease_deadletter.json | strict visibility-timeout boundary, at-least-once requeue, max-delivery poison routing to DLQ |
ComputedMap materialization conformance
The conformance/materialization/ directory pins the eager-vs-lazy
materialization behavior (#lzmatmode) of a
ComputedMap — eager is a pre-mint loop over the
keyset; lazy is get_or_insert_with mint-on-access — proved in
lazily-formal’s Materialization module. These are compute
fixtures: a binding reads the spec (each key’s canonical value, and — for the
mixed fixture — its cell/slot entry kind), builds the keyed map under both
strategies, replays the reads sequence against the lazy build, and asserts
observational transparency plus the memory / entry-kind laws. Because
materialization is not observable on the value axis, there is no wire schema —
only the compute effects below.
| Fixture | Covers |
|---|---|
materialization/observational_transparency.json | identical observe values under eager vs. lazy; eager materializes all keys; lazy materializes only read keys; default mode eager (observe_canonical, eager_materializes_all, lazy_defers_slots, default_mode_eager) |
materialization/deferral_not_deallocation.json | lazy present set grows monotonically and is unchanged by re-reads; final lazy set is a subset of the eager set; no churn from allocation (materialize_present_monotone, lazy_present_subset_eager, materialize_preserves_observe) |
materialization/entry_kind_orthogonal_to_mode.json | entry kind ⟂ mode: cell (input) entries are present under either mode; slot (derived) entries are deferred under lazy until read (cell_entries_materialized_in_every_mode, slot_entries_deferred_under_lazy) |
Reactive graph disposal conformance
The conformance/reactive-graph/ directory pins
disposal and teardown scopes (#lzspecedgeindex) — the
explicit-lifetime half of the graph contract, whose scope law is proved as
disposeScope_eq_disposeAll in lazily-formal’s Reactive module.
These are compute fixtures: a binding builds the graph by replaying each step’s
op against a fresh Context and asserts that step’s expect. Disposal is not
observable on the wire — it changes what a context holds and what a publish reaches,
never a serialized frame — so there is no wire schema, only the compute effects below.
Every assertion is on observable state: dependent/dependency-set sizes, whether a node is readable, a read’s value or error, and which effects ran on a publish. Nothing here fixes a promotion threshold, a hash strategy, or an index layout — those are explicitly implementation-free per the implementation note, and a binding that dedups edges by linear scan at every degree conforms exactly as well as one that promotes to a hash index.
Op vocabulary (all ids are fixture-local labels, never a binding’s internal id):
| Op | Meaning |
|---|---|
cell {id, value} | Create a source cell |
computed {id, reads[], offset, scope?} | Create a derived, guarded Computed whose value is sum(reads) + offset; owned by scope when named |
effect {id, reads[], scope?} | Register an effect over reads; runs on creation and on each flush after a tracked invalidation |
read {id} | Read a node — expect.value, or expect.error for a disposed one |
set {id, value} | Publish; expect.observed_by names the effects that ran, expect.observed_count their number |
dispose {id} | Dispose one node, dispatching on its own kind |
fanout {id_prefix, reads[], count, read_each} / dispose_fanout {id_prefix, count} | Create / dispose count sibling readers, for widths a literal step list would bloat |
churn {source, id_prefix, live_width, cycles, mode, read_each} | Run cycles subscribe/unsubscribe cycles holding live_width subscribers live (mode: dispose_then_create or scope_per_cycle) |
begin_scope {scope} / end_scope {scope} | Open / end a teardown scope |
disarm {scope} | Cancel the scope’s teardown — ending it then disposes nothing |
dispose_stale_handle {handle_of, handle_kind} | Dispose through a handle whose id may have been recycled; a no-op unless the id still names a node of handle_kind |
subscribe {id, cell, callback?, on_notify?, on_notify_once?} | Register a Cell observer labelled id on cell. callback names a shared callable, so two registrations naming the same callback subscribe the same function (default: the callback is unique to id). on_notify is a list of subscribe/unsubscribe ops the callback performs reentrantly when it runs; on_notify_once restricts them to the first invocation. A reentrant subscribe may use id_prefix instead of id, minting <prefix>_0, <prefix>_1, … per invocation |
unsubscribe {id, times?} | Invoke the disposer returned by subscribe {id}, times times (default 1) — repeat calls exercise idempotency |
A fixture uses top-level steps, or scenarios plus expected when the claim is that
two differently-built runs agree (expected.observationally_equal).
| Fixture | Covers |
|---|---|
reactive-graph/dispose_detaches_edges_both_directions.json | disposal detaches upstream and downstream edges; a publish to a former source does not reach the disposed node; the surviving source is unaffected |
reactive-graph/read_after_dispose_is_an_error.json | reading a disposed Computed, a disposed source cell, or through a live reader that names one is an error — never a stale or default value; double-dispose is an idempotent no-op |
reactive-graph/recycled_id_inherits_nothing.json | a node minted on a recycled id starts with an empty edge set in both directions — the owner-keyed-side-table aliasing hazard; a stale cross-kind handle disposes nothing |
reactive-graph/scope_teardown_equals_fold_of_disposals.json | ending a scope is observationally equal to disposing each member individually (disposeScope_eq_disposeAll), including reverse-creation-order cleanup |
reactive-graph/scoping_bounds_teardown_not_visibility.json | a scope’s nodes read parent- and sibling-owned nodes freely in every direction; propagation crosses scope boundaries unchanged |
reactive-graph/disarm_disposes_nothing.json | disarm() leaves node state untouched — the nodes stay readable, keep propagating, and stay individually disposable; ending the scope disposes nothing |
reactive-graph/cross_scope_teardown_hazard.json | ending a scope tears down its nodes even when a node outside still reads them (required failure — a binding that keeps them alive is non-conforming); the mirror case is symmetric |
reactive-graph/churn_returns_to_baseline.json | a subscribe/unsubscribe cycle that disposes what it creates leaves the source’s dependent set at its starting size, under both individual disposal and one scope per cycle |
Cell observer conformance (#lzdartobservercow)
The same directory pins
observer semantics —
the hand-registered Cell.subscribe callbacks, which are a separate mechanism from the
tracked dependency edges above. Same compute-fixture shape, same op replay, two further
assertion keys:
| Assertion | Meaning |
|---|---|
observed_order | The exact sequence of observer labels invoked by this step |
observed_counts | Per-observer invocation counts for this step, where a shared callback runs more than once |
observed_order is deliberately a sequence and not a set. An unordered observer
collection satisfies a set-valued observed_by while firing in a fresh order on every
notification, which is precisely how the family’s divergence went unnoticed — so an
observer fixture that asserts only observed_by is rejected by the structural guard.
These fixtures fail against some bindings today, by design. The observer contract was unwritten until now and four bindings answered it differently; the fixtures encode the family position, and the gaps are listed as known divergences with migrations. A red result here is a binding bug, not a fixture bug.
| Fixture | Covers |
|---|---|
reactive-graph/observer_order_is_registration_order.json | observers fire in registration order, stably across notifications; a removal closes the gap without reordering survivors; a re-registered callback appends rather than resuming its old position; the == guard still suppresses the notification entirely (fails: py, zig) |
reactive-graph/observer_duplicate_registrations_are_independent.json | subscribing one callback twice yields two registrations that both fire and dispose independently — no dedup by identity or by equality (fails: py, zig) |
reactive-graph/observer_subscribe_during_notify_is_deferred.json | an observer registered mid-notification first runs on the next one, including when observers registered earlier are still unvisited; a self-feeding subscriber terminates because the pass is bounded by the pre-callback count |
reactive-graph/observer_unsubscribe_during_notify_takes_effect_immediately.json | an observer disposed mid-notification does not run in that pass even when unvisited; an already-visited observer’s invocation stands; self-unsubscribe completes the call it is in; the tail observer still runs, catching a swap-remove under a live cursor (fails: dart, go) |
reactive-graph/observer_disposer_is_idempotent.json | a disposer latches — repeat calls are silent no-ops that remove nothing else, and a spent disposer never reaches a later registration of an equal callable; cell teardown drops observers without invoking them, and a disposer outliving its cell is a no-op |
Signaling conformance
The conformance/signaling/ directory pins the WebSocket signaling wire protocol
(see protocol.md § Signaling Protocol),
the cross-language contract shared by every distributed-plane binding and the
reference TypeScript signaling server.
signaling/frames.jsonis a wire fixture: each entry’swirefield is the canonical JSON for oneClientMessage/ServerMessagevariant and validates againstschemas/signaling.json. A binding encodes its typed message to the same JSON and decodes the JSON back. Tags are kebab-case (peer-joined,peer-left);peerids are bare numbers ≤ 2⁵³−1. Client-directed frames carryto; server-forwarded frames carry a server-stampedfrom.signaling/anti_spoof_session.jsonis a compute fixture: a binding that implements the server room replays eachinputand asserts the emittedexpectframes. It pins the load-bearing invariants — thewelcomeroster excludes the joining peer, a forwarded frame’sfromis the sender’s server-registered id (never client-supplied), and an unknown target yields anerrorframe. Its three-peer join makes roster sorting observable: the third welcome must be[1, 2], not reverse order[2, 1]. Peer-joined broadcast order remains outside this fixture’s claim.
Distributed (CRDT plane) conformance
The conformance/distributed/ directory pins the CRDT anti-entropy plane
(see protocol.md § Distributed).
distributed/crdt_sync_frames.jsonis a wire fixture: eachwireis a{"CrdtSync": {frontier, ops}}envelope validating againstschemas/distributed.json(empty, keyed+keyless ops, and a multi-peer frontier).distributed/anti_entropy_converge.jsonis a compute fixture: a binding replays each scenario’sopsthrough itsCrdtPlaneRuntimeand asserts convergence toexpect.convergedindependent of delivery order, plus state-based idempotence (re-ingesting a seen frame applies 0 new ops). It models LWW cells where the planeWireStampis the decisive stamp under lexicographic(wall_time, logical, peer)order.
Lossless tree conformance
The conformance/lossless-tree/ directory pins the lossless full-document tree
CRDT (see Lossless Tree CRDT, #lzlosstree). These are
compute fixtures with the same {scenarios: [{seed, steps, expect}]} shape as
the collections fixtures: a binding builds the seed.tree on replica a
(addressing nodes by stable string labels), replays each step (fork /
clone / sync / deliver an op subset / on a replica an op — create,
edit_leaf, split, merge_leaves, reorder, tombstone), and asserts the
expect fields: render / render_on (exact rendered text per replica),
live_nodes (live element+leaf count, excluding the root), and converged (a set
of replicas that must render identically). Byte offsets in ops (at_byte) are
UTF-8 and must land on a char boundary.
The step vocabulary itself is schema’d by schemas/lossless-tree-fixture.json
(compute, not wire), so a step form the corpus can express is a step every runner
has been told how to read. Two properties of the schedule are load-bearing and
easy to miss:
- A mutation step may follow a
sync/deliverinto the same replica. There is no fork → edit → sync phase structure;stepsare simply applied in order. A post-sync mutation is the only way to observe that ingest advanced the ingesting replica’s Lamport counter, because a replica that only mutates before its first ingest never has to mint an id that outranks an ingested stamp. delivercarries exactly one selector,onlyororder— they are mutually exclusive and are never composed. Both are 0-based indexes into the canonically ordered diff (diff(to.frontier())onfrom, sorted by dotted(counter, peer)id), must be distinct, and must be in range; a runner fails the fixture rather than clamping.onlydelivers that subset in canonical order however it is listed — a delivery hole, where the omitted dots stay missing and re-requestable.orderdelivers the listed entries in the listed sequence, as oneapply_updatebatch: a runner must not re-sort them and must not split them across calls, because the point is to hand a replica an op whose parent/target orprevhas not arrived yet and require it to buffer and retry rather than drop.orderneed not be a permutation of the whole diff. Composing the two is disallowed on purpose: nothing would say whetherorder’s indexes address the diff or theonlysubset, and ten runners would answer that ten ways.
| Fixture | Covers |
|---|---|
lossless-tree/exact_roundtrip.json | Token/Trivia/Raw/Error leaves incl. an invalid span + multi-byte text; render == source |
lossless-tree/one_leaf_edit_delta.json | one-leaf edit at a UTF-8 byte offset in multi-byte text, delivered by anti-entropy |
lossless-tree/split_merge.json | split a leaf then merge back; render preserved; live-node count grows then restores |
lossless-tree/concurrent_insert_same_parent.json | two replicas insert into the same gap; both survive, deterministic order, converge |
lossless-tree/concurrent_reorder_and_leaf_edit.json | concurrent reorder + text edit both apply (position/text are independent registers) |
lossless-tree/non_contiguous_anti_entropy.json | a delivery hole is representable in the dotted frontier, re-requested, and converges |
lossless-tree/token_trivia_preservation.json | a leaf edit leaves adjacent Token/Trivia leaves byte-for-byte unchanged |
lossless-tree/invalid_source_roundtrip.json | unclosed fence/comment kept as Error leaves round-trips exactly; editing an adjacent Raw leaf keeps the Error spans |
lossless-tree/concurrent_conflict_preserves_text.json | incompatible concurrent shapes both survive with no bytes dropped (text preservation wins over semantic shape) |
lossless-tree/apply_update_advances_counter.json | ingest advances the Lamport counter, so a write minted AFTER a sync outranks the stamps that sync delivered (post-sync mutation step) |
lossless-tree/out_of_order_delivery_buffers.json | a reversed delivery batch (deliver.order) drains through the dependency buffer; a dropped op is never repaired by a later sync |
The op-delta wire form additionally validates against schemas/lossless-tree.json
(vocabulary) + schemas/lossless-tree-delta.json (the TreeUpdate message); the
Rust reference and each port validate their serialized TreeUpdate against these.
Causal receipt conformance
The conformance/receipts/ directory pins lazily’s generic outcome vocabulary
for commands and effect requests (see
protocol.md § Causal Receipts).
receipts/causal_receipts.jsonis a wire + compute fixture: itswirefield validates againstschemas/receipts.json, and bindings replay the receipts into their projection. The stale-generation receipt is ignored by the current projection;observed/acceptedremain non-terminal;appliedis the terminal outcome.
Examples
snapshot_minimal.json
{
"description": "Minimal snapshot with one payload node and no edges",
"protocol_version": 1,
"kind": "Snapshot",
"assertions": {
"epoch": 1,
"node_count": 1,
"edge_count": 0,
"root_count": 1,
"first_node_type_tag": "i32"
},
"wire": {
"Snapshot": {
"epoch": 1,
"nodes": [
{
"node": 1,
"type_tag": "i32",
"state": {
"Payload": [1, 2, 3, 4]
}
}
],
"edges": [],
"roots": [1]
}
}
}
snapshot_multi_node.json
{
"description": "Snapshot with payload nodes, opaque node, edges, and roots",
"protocol_version": 1,
"kind": "Snapshot",
"assertions": {
"epoch": 7,
"node_count": 3,
"edge_count": 2,
"root_count": 2,
"has_opaque_node": true,
"opaque_node_id": 3
},
"wire": {
"Snapshot": {
"epoch": 7,
"nodes": [
{
"node": 1,
"type_tag": "i32",
"state": {
"Payload": [1, 2, 3]
}
},
{
"node": 2,
"type_tag": "f64",
"state": {
"Payload": [0, 0, 0, 0, 0, 0, 240, 63]
}
},
{
"node": 3,
"type_tag": "opaque-type",
"state": "Opaque"
}
],
"edges": [
{ "dependent": 2, "dependency": 1 },
{ "dependent": 3, "dependency": 1 }
],
"roots": [1, 2]
}
}
}
snapshot_shared_blob.json
{
"description": "Snapshot with a shared-blob node referencing shared memory",
"protocol_version": 1,
"kind": "Snapshot",
"assertions": {
"epoch": 9,
"node_count": 1,
"edge_count": 0,
"root_count": 1,
"first_node_state_kind": "SharedBlob",
"blob_offset": 0,
"blob_len": 16,
"blob_epoch": 9
},
"wire": {
"Snapshot": {
"epoch": 9,
"nodes": [
{
"node": 7,
"type_tag": "text/plain",
"state": {
"SharedBlob": {
"offset": 0,
"len": 16,
"generation": 1,
"epoch": 9,
"checksum": 123456789
}
}
}
],
"edges": [],
"roots": [7]
}
}
}
delta_sequential.json
{
"description": "Sequential delta covering all 7 DeltaOp variants",
"protocol_version": 1,
"kind": "Delta",
"assertions": {
"base_epoch": 40,
"epoch": 41,
"is_sequential": true,
"op_count": 7,
"has_all_op_variants": true
},
"wire": {
"Delta": {
"base_epoch": 40,
"epoch": 41,
"ops": [
{ "CellSet": { "node": 1, "payload": { "Inline": [10] } } },
{ "SlotValue": { "node": 2, "payload": { "Inline": [20] } } },
{ "Invalidate": { "node": 3 } },
{
"NodeAdd": {
"node": 4,
"type_tag": "u64",
"state": { "Payload": [64] }
}
},
{ "NodeRemove": { "node": 5 } },
{ "EdgeAdd": { "dependent": 2, "dependency": 1 } },
{ "EdgeRemove": { "dependent": 3, "dependency": 1 } }
]
}
}
}
delta_non_sequential.json
{
"description": "Non-sequential delta with gap (requires resync)",
"protocol_version": 1,
"kind": "Delta",
"assertions": {
"base_epoch": 12,
"epoch": 13,
"is_sequential": true,
"resync_after_epoch_10": true
},
"wire": {
"Delta": {
"base_epoch": 12,
"epoch": 13,
"ops": []
}
}
}
delta_shared_blob.json
{
"description": "Delta with shared-blob payload referencing shared memory",
"protocol_version": 1,
"kind": "Delta",
"assertions": {
"base_epoch": 8,
"epoch": 9,
"op_count": 1,
"first_op_kind": "SlotValue",
"first_op_payload_kind": "SharedBlob"
},
"wire": {
"Delta": {
"base_epoch": 8,
"epoch": 9,
"ops": [
{
"SlotValue": {
"node": 7,
"payload": {
"SharedBlob": {
"offset": 40,
"len": 17,
"generation": 2,
"epoch": 9,
"checksum": 987654321
}
}
}
}
]
}
}
}
Portable standard-library primitives
This document specifies the language-neutral Timer, Timeout<T>, and
RevisionBarrier convenience APIs. Bindings may expose idiomatic names and
types, but an implementation earns the corresponding coverage feature only by
replaying the canonical fixtures against its public production API.
The three versioned features are:
stdlib_timer_v1stdlib_timeout_v1stdlib_revision_barrier_v1
All time in this contract is monotone logical time. Production implementations normally read a platform monotone clock; conformance runners inject a clock and wait driver and must never sleep or depend on wall-clock scheduling.
Shared rules
- Logical instants and durations are unsigned 64-bit integers.
start + durationis checked. Overflow returns typedUnavailableand never wraps.- A clock observation lower than the most recently accepted observation returns
Unavailable(clock_regression)without changing the stored state. - A deadline is inclusive:
now >= deadlinereaches the deadline. - The first terminal result is latched. Later reads return the same result and invoke no caller-supplied adapters.
- The adapter owns no executor, async runtime, or thread. Waiting is driven by a caller-owned clock/wait seam.
Timer
Starting a timer at start for duration produces a pending timer whose
deadline is start + duration. A zero duration therefore fires on the first
observation at start.
observe(now) has two non-error observations:
Pending(deadline)whennow < deadline;Fired(fired_at)whennow >= deadline.
The first firing latches its actual observation instant as fired_at; later
observations preserve that value. A timer cannot return to pending.
Timeout<T>
Timeout<T> is a caller-driven adapter around an operation probe and a
cancellation probe. It is not a future executor and it is not the reactive
TimeoutCell.
TimeoutCell emits a reactive timeout edge; this adapter resolves a single
operation to one of:
Completed(T)TimedOutCancelledUnavailable(reason)
Each nonterminal poll(now, operation, cancellation) uses this precedence:
- Reject a regressed clock without polling either adapter.
- If
now >= deadline, returnTimedOutwithout polling either adapter. - Sample the operation and cancellation adapters once each.
- A completed operation wins, including when cancellation is observed in the same poll.
- An unavailable operation returns
Unavailable(operation_unavailable). - Cancellation returns
Cancelled; an unavailable cancellation adapter returnsUnavailable(cancellation_unavailable). - Otherwise remain pending.
This order makes exact-deadline behavior and simultaneous completion/cancellation deterministic in every binding.
RevisionBarrier
A revision barrier waits for both:
current_revision >= required_revision; and- a caller-supplied derived predicate to be true.
Revisions never decrease. Registration increments a wake generation, then the waiter re-reads the revision and predicate after registration. This register-then-recheck rule closes the check-to-sleep race: an advance that lands between the initial check and waiter registration is observed immediately.
A waiter may have an inclusive deadline and a cancellation adapter. Deadline
dominance is checked before cancellation. Its terminal outcomes are
Satisfied(revision), TimedOut, Cancelled, Disposed, and
Unavailable(reason). Disposal wakes all waiters and latches.
Effect receipts and transport acknowledgements are application-owned observations. Recording a receipt cannot advance a barrier revision, change its wake generation, or satisfy a waiter.
Canonical fixtures and mutation gates
The canonical corpus is under conformance/stdlib/ and validates against
schemas/stdlib-fixture.schema.json. Each family declares a scenario and
assertion floor. A runner that opens zero fixtures, executes fewer than the
declared floor, or merely echoes fixture expectations fails conformance.
The spec test runner also applies named mutations. At minimum it proves that the corpus detects:
- changing
>=to>at a deadline; - checking cancellation before pre-deadline completion;
- failing to latch a terminal result;
- omitting the barrier’s post-registration recheck;
- treating an effect receipt as barrier authority; and
- replacing production transitions with fixture bookkeeping.
Bindings may advertise a feature to the peer suite only after their public API passes the matching canonical family and its mutation checks. Unsupported bindings stay visible as staged and receive no pass credit.
JSON Schemas
Schemas are provided as JSON Schema (Draft 2020-12). Each implementation must validate against these schemas. The JSON representation of the wire format is normative; future binary codecs encode the same shapes.
| Schema | Layer |
|---|---|
defs.json | Shared wire primitives (NodeId, NodeKey, NodeState, IpcValue, ShmBlobRef, WireStamp) |
snapshot.json | IPC — Snapshot message (externally-tagged {"Snapshot": …} envelope) |
delta.json | IPC — Delta message (externally-tagged {"Delta": …}, all 7 DeltaOp variants) |
ffi.json | Cross-language FFI boundary |
signaling.json | Signaling (WebSocket) |
distributed.json | Distributed — CrdtSync message ({"CrdtSync": …}) + CRDT/cell-model types |
receipts.json | Causal receipts ({"CausalReceipts": …}) + terminal outcome projection |
message-passing.json | Command / RPC message plane (CommandSubmit / CommandCancel / CommandEvents / CommandProjection) |
statechart.json | Compute (Harel/SCXML chart form — not a wire message) |
stdlib-fixture.schema.json | Deterministic Timer, Timeout, and RevisionBarrier conformance scenarios |
assertion-blocks.json | Routed, fail-closed schemas for every canonical assertions / expect / expected object |
The IPC schemas describe the normative externally-tagged envelope that
every binding serializes (the single-key {"Snapshot": …} / {"Delta": …} /
{"CrdtSync": …} form), with node addressing and value bytes as JSON arrays
of u8 (not base64). Shared wire primitives live in defs.json and are
referenced via absolute $ref so the primitive definitions never copy-drift.
Every conformance fixture’s wire field validates against its schema — enforced
by make test-schemas (see tests/test_schema_conformance.py).
Assertion blocks are validated independently of their fixture’s wire or compute schema, so families without a whole-fixture schema still reject unknown claim keys and stale value shapes. See Assertion-Block Schemas.
defs.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/defs.json",
"title": "Shared wire primitive definitions",
"description": "Primitive types shared across the lazily IPC schemas (snapshot, delta, distributed). Referenced via absolute $ref so every wire schema stays codec-faithful to protocol.md without copy-drift.",
"$defs": {
"NodeId": {
"type": "integer",
"minimum": 0,
"maximum": 18446744073709551615,
"description": "Wire-stable node identifier (u64). The full u64 range is wire-valid; the 2^53-1 bound is a PRODUCER obligation on peers whose runtime represents integers as IEEE-754 doubles, and a decoder that cannot represent a received value exactly MUST reject the frame rather than round it (protocol.md § NodeId / PeerId, #lzspecdecoderbound). Schema validity is therefore u64-wide — a narrower binding refuses at decode, not here."
},
"PeerId": {
"type": "integer",
"minimum": 0,
"maximum": 18446744073709551615,
"description": "Wire-stable peer identifier (u64). Same producer bound and decoder obligation as NodeId (#lzspecdecoderbound). This carried maximum 9007199254740991 until that audit, which contradicted both the u64 wire type and protocol.md's producer-only wording."
},
"NodeKey": {
"type": "string",
"minLength": 1,
"maxLength": 1024,
"pattern": "^[^/]+(/[^/]+)*$",
"description": "Optional wire-stable keyed address: a '/'-joined path (e.g. 'scores/alice'). The pattern rejects empty/leading/trailing/double slashes; construction-time code also enforces the ≤1024 *byte* and ≤32 *segment* bounds (see protocol.md § NodeKey). Serialized as a bare string; the field is omitted in self-describing codecs when absent."
},
"ShmBlobRef": {
"type": "object",
"description": "Descriptor into a blob backend (zero-copy transport). The standard fields locate and integrity-check a byte range within the backend's resolved buffer; `backend` selects which pluggable backend resolves it ('shm' = POSIX shared memory, the default for backward compatibility; 'arrow' = Apache Arrow IPC/Flight; 'in_process' = an in-process arena). See protocol.md § Zero-copy transport. `backend` is optional and defaults to 'shm' so legacy descriptors validate unchanged.",
"required": ["offset", "len", "generation", "epoch", "checksum"],
"additionalProperties": false,
"properties": {
"offset": { "type": "integer", "minimum": 0 },
"len": { "type": "integer", "minimum": 0 },
"generation": { "type": "integer", "minimum": 0 },
"epoch": { "type": "integer", "minimum": 0 },
"checksum": { "type": "integer", "minimum": 0 },
"backend": { "type": "string", "enum": ["shm", "arrow", "in_process"], "default": "shm", "description": "Which pluggable backend holds the blob. Optional; defaults to 'shm' (POSIX shared memory) for backward compatibility. The receiver routes resolution by this discriminator." }
}
},
"NodeState": {
"description": "Externally-tagged node body: concrete bytes, a shared-memory value, or an opaque (visible but non-serializable) node.",
"oneOf": [
{
"type": "object",
"title": "Payload",
"required": ["Payload"],
"additionalProperties": false,
"properties": {
"Payload": {
"type": "array",
"items": { "type": "integer", "minimum": 0, "maximum": 255 },
"description": "Serialized value bytes as JSON array of u8 (NOT base64)."
}
}
},
{
"type": "object",
"title": "SharedBlob",
"required": ["SharedBlob"],
"additionalProperties": false,
"properties": { "SharedBlob": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/ShmBlobRef" } }
},
{ "type": "string", "title": "Opaque", "const": "Opaque" }
]
},
"IpcValue": {
"description": "Externally-tagged delta payload: inline bytes or a shared-memory blob reference.",
"oneOf": [
{
"type": "object",
"title": "Inline",
"required": ["Inline"],
"additionalProperties": false,
"properties": {
"Inline": {
"type": "array",
"items": { "type": "integer", "minimum": 0, "maximum": 255 },
"description": "Inline serialized bytes as JSON array of u8 (NOT base64)."
}
}
},
{
"type": "object",
"title": "SharedBlob",
"required": ["SharedBlob"],
"additionalProperties": false,
"properties": { "SharedBlob": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/ShmBlobRef" } }
}
]
},
"WireStamp": {
"type": "object",
"description": "Wire mirror of the runtime HLC stamp — a total order (wall_time, logical, peer).",
"required": ["wall_time", "logical", "peer"],
"additionalProperties": false,
"properties": {
"wall_time": { "type": "integer", "minimum": 0, "description": "Wall-clock microseconds since the Unix epoch." },
"logical": { "type": "integer", "minimum": 0, "description": "Logical counter advancing causality within equal wall_time." },
"peer": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/PeerId", "description": "Originating peer; final tiebreak so equal (wall, logical) is a total order." }
}
}
}
}
snapshot.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/snapshot.json",
"title": "Snapshot",
"description": "Full graph state IPC message. Normative form (protocol.md § IPC) is the EXTERNALLY-TAGGED envelope {\"Snapshot\": {...}} — not a \"type\" discriminant. Serialized value bytes are JSON arrays of u8, not base64. A NodeSnapshot's optional `key` (NodeKey) is omitted when absent.",
"type": "object",
"required": ["Snapshot"],
"additionalProperties": false,
"properties": {
"Snapshot": {
"type": "object",
"required": ["epoch", "nodes", "edges", "roots"],
"additionalProperties": false,
"properties": {
"epoch": { "type": "integer", "minimum": 0, "description": "Current IPC epoch." },
"nodes": { "type": "array", "items": { "$ref": "#/$defs/NodeSnapshot" }, "description": "All serialized nodes." },
"edges": { "type": "array", "items": { "$ref": "#/$defs/EdgeSnapshot" }, "description": "Dependency edges (dependent → dependency)." },
"roots": { "type": "array", "items": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" }, "description": "Cell and source slot ids." }
}
}
},
"$defs": {
"NodeSnapshot": {
"type": "object",
"description": "Full state for one node in a snapshot.",
"required": ["node", "type_tag", "state"],
"additionalProperties": false,
"properties": {
"node": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"type_tag": { "type": "string", "description": "Stable cross-process type key for decoding state." },
"state": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeState" },
"key": {
"description": "Optional wire-stable keyed address. A conforming ENCODER omits it when absent; an explicit null is also wire-valid and a decoder MUST read both forms as absent (protocol.md § NodeKey, #lzkeynullstrict). This carried a bare NodeKey $ref until that audit, which made a frame the spec now requires decoders to accept schema-invalid.",
"oneOf": [
{ "type": "null" },
{ "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeKey" }
]
}
}
},
"EdgeSnapshot": {
"type": "object",
"description": "Directed dependency edge (dependent → dependency).",
"required": ["dependent", "dependency"],
"additionalProperties": false,
"properties": {
"dependent": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"dependency": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" }
}
}
}
}
delta.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/delta.json",
"title": "Delta",
"description": "Incremental change-set IPC message. Normative form (protocol.md § IPC) is the EXTERNALLY-TAGGED envelope {\"Delta\": {...}} — not a \"type\" discriminant. Each DeltaOp variant is itself externally tagged by its PascalCase name (single-key object). Serialized value bytes are JSON arrays of u8, not base64. NodeAdd's optional `key` (NodeKey) is omitted in JSON when absent.",
"type": "object",
"required": ["Delta"],
"additionalProperties": false,
"properties": {
"Delta": {
"type": "object",
"required": ["base_epoch", "epoch", "ops"],
"additionalProperties": false,
"properties": {
"base_epoch": { "type": "integer", "minimum": 0, "description": "Epoch this delta applies to (must equal the receiver's last_epoch)." },
"epoch": { "type": "integer", "minimum": 0, "description": "New epoch, >= base_epoch + 1. Usually base_epoch + 1; a multi-epoch-span delta has epoch > base_epoch + 1 (epoch - base_epoch = accepted-event span). See protocol.md § Multi-epoch-span delta." },
"ops": { "type": "array", "items": { "$ref": "#/$defs/DeltaOp" }, "description": "Ordered operations." }
}
}
},
"$defs": {
"DeltaOp": {
"description": "One incremental graph mutation. Externally tagged: a single-key object whose key is the PascalCase variant name.",
"oneOf": [
{
"type": "object",
"title": "CellSet",
"required": ["CellSet"],
"additionalProperties": false,
"properties": { "CellSet": { "$ref": "#/$defs/CellSetBody" } }
},
{
"type": "object",
"title": "SlotValue",
"required": ["SlotValue"],
"additionalProperties": false,
"properties": { "SlotValue": { "$ref": "#/$defs/SlotValueBody" } }
},
{
"type": "object",
"title": "Invalidate",
"required": ["Invalidate"],
"additionalProperties": false,
"properties": { "Invalidate": { "$ref": "#/$defs/NodeBody" } }
},
{
"type": "object",
"title": "NodeAdd",
"required": ["NodeAdd"],
"additionalProperties": false,
"properties": { "NodeAdd": { "$ref": "#/$defs/NodeAddBody" } }
},
{
"type": "object",
"title": "NodeRemove",
"required": ["NodeRemove"],
"additionalProperties": false,
"properties": { "NodeRemove": { "$ref": "#/$defs/NodeBody" } }
},
{
"type": "object",
"title": "EdgeAdd",
"required": ["EdgeAdd"],
"additionalProperties": false,
"properties": { "EdgeAdd": { "$ref": "#/$defs/EdgeBody" } }
},
{
"type": "object",
"title": "EdgeRemove",
"required": ["EdgeRemove"],
"additionalProperties": false,
"properties": { "EdgeRemove": { "$ref": "#/$defs/EdgeBody" } }
},
{
"type": "object",
"title": "QueuePush",
"description": "Op-log delta form for a QueueCell node: append `payload` to the tail. The op-log delta is the incremental counterpart to the storage-snapshot form (cell-model.md § Wire and snapshot shape); a run of same-direction QueuePush ops fuses into one multi-epoch batch Delta (protocol.md § Backpressure & outbox coalescing).",
"required": ["QueuePush"],
"additionalProperties": false,
"properties": { "QueuePush": { "$ref": "#/$defs/CellSetBody" } }
},
{
"type": "object",
"title": "QueuePop",
"description": "Op-log delta form for a QueueCell node: remove the head. Carries no value (the popped element is determined by ordered replay); a re-delivered pop is Ignored by the epoch gap rule. Crosses the wire only when consumption is authoritatively replicated (mirror sync); in the default producer→consumer sync the remote consumer's pops are local. A QueuePop/QueueClose is a fusion boundary (order-sensitive).",
"required": ["QueuePop"],
"additionalProperties": false,
"properties": { "QueuePop": { "$ref": "#/$defs/NodeBody" } }
},
{
"type": "object",
"title": "QueueClose",
"description": "Op-log delta form for a QueueCell node: mark closed (idempotent, terminal — Closed is distinct from Empty). A fusion boundary.",
"required": ["QueueClose"],
"additionalProperties": false,
"properties": { "QueueClose": { "$ref": "#/$defs/NodeBody" } }
}
]
},
"CellSetBody": {
"type": "object",
"required": ["node", "payload"],
"additionalProperties": false,
"properties": {
"node": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"payload": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/IpcValue" }
}
},
"SlotValueBody": {
"type": "object",
"required": ["node", "payload"],
"additionalProperties": false,
"properties": {
"node": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"payload": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/IpcValue" }
}
},
"NodeBody": {
"type": "object",
"required": ["node"],
"additionalProperties": false,
"properties": { "node": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" } }
},
"NodeAddBody": {
"type": "object",
"required": ["node", "type_tag", "state"],
"additionalProperties": false,
"properties": {
"node": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"type_tag": { "type": "string" },
"state": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeState" },
"key": {
"description": "Optional wire-stable keyed address. A conforming ENCODER omits it when absent; an explicit null is also wire-valid and a decoder MUST read both forms as absent (protocol.md § NodeKey, #lzkeynullstrict). This carried a bare NodeKey $ref until that audit, which made a frame the spec now requires decoders to accept schema-invalid.",
"oneOf": [
{ "type": "null" },
{ "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeKey" }
]
}
}
},
"EdgeBody": {
"type": "object",
"required": ["dependent", "dependency"],
"additionalProperties": false,
"properties": {
"dependent": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"dependency": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" }
}
}
}
}
ffi.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/ffi.json",
"title": "FFI Types",
"description": "C ABI types for the lazily FFI boundary",
"$defs": {
"LazilyFfiBytes": {
"type": "object",
"description": "Owned byte buffer crossing the FFI boundary",
"properties": {
"ptr": { "type": "integer", "minimum": 0, "description": "Pointer to byte buffer" },
"len": { "type": "integer", "minimum": 0, "description": "Buffer length in bytes" }
},
"required": ["ptr", "len"],
"additionalProperties": false
},
"LazilyFfiStatus": {
"type": "integer",
"enum": [0, 1, 2, 3, 4, 5],
"description": "FFI operation status code",
"oneOf": [
{ "const": 0, "title": "Ok" },
{ "const": 1, "title": "Empty" },
{ "const": 2, "title": "NullPointer" },
{ "const": 3, "title": "InvalidMessage" },
{ "const": 4, "title": "EncodeFailed" },
{ "const": 5, "title": "Panic" }
]
},
"LazilyFfiMessageKind": {
"type": "integer",
"enum": [0, 1, 2, 3, 4, 5],
"description": "IPC message kind discriminator. 4/5 are the reliable-sync (#lzsync) reverse-channel control-frame variants ResyncRequest/OutboxAck (protocol.md § Reliable Sync).",
"oneOf": [
{ "const": 0, "title": "Unknown" },
{ "const": 1, "title": "Snapshot" },
{ "const": 2, "title": "Delta" },
{ "const": 3, "title": "CrdtSync" },
{ "const": 4, "title": "ResyncRequest" },
{ "const": 5, "title": "OutboxAck" }
]
}
}
}
signaling.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/signaling.json",
"title": "Signaling Protocol",
"description": "WebSocket signaling frames for lazily peer discovery",
"$comment": "Validates both client→server and server→client frames",
"oneOf": [
{
"title": "ClientMessage",
"oneOf": [
{
"type": "object",
"title": "Join",
"properties": {
"type": { "const": "join" },
"peer": { "type": "integer", "minimum": 0, "maximum": 9007199254740991 },
"capabilities": { "type": "array", "items": { "type": "string" } }
},
"required": ["type", "peer"],
"additionalProperties": false
},
{
"type": "object",
"title": "Offer",
"properties": {
"type": { "const": "offer" },
"to": { "type": "integer", "minimum": 0 },
"sdp": { "type": "string" }
},
"required": ["type", "to", "sdp"],
"additionalProperties": false
},
{
"type": "object",
"title": "Answer",
"properties": {
"type": { "const": "answer" },
"to": { "type": "integer", "minimum": 0 },
"sdp": { "type": "string" }
},
"required": ["type", "to", "sdp"],
"additionalProperties": false
},
{
"type": "object",
"title": "Ice",
"properties": {
"type": { "const": "ice" },
"to": { "type": "integer", "minimum": 0 },
"candidate": { "type": "string" }
},
"required": ["type", "to", "candidate"],
"additionalProperties": false
},
{
"type": "object",
"title": "Relay",
"properties": {
"type": { "const": "relay" },
"to": { "type": "integer", "minimum": 0 },
"payload": {}
},
"required": ["type", "to", "payload"],
"additionalProperties": false
},
{
"type": "object",
"title": "Leave",
"properties": {
"type": { "const": "leave" }
},
"required": ["type"],
"additionalProperties": false
}
]
},
{
"title": "ServerMessage",
"oneOf": [
{
"type": "object",
"title": "Welcome",
"properties": {
"type": { "const": "welcome" },
"peer": { "type": "integer", "minimum": 0 },
"peers": {
"type": "array",
"items": { "type": "integer", "minimum": 0 }
}
},
"required": ["type", "peer", "peers"],
"additionalProperties": false
},
{
"type": "object",
"title": "PeerJoined",
"properties": {
"type": { "const": "peer-joined" },
"peer": { "type": "integer", "minimum": 0 }
},
"required": ["type", "peer"],
"additionalProperties": false
},
{
"type": "object",
"title": "PeerLeft",
"properties": {
"type": { "const": "peer-left" },
"peer": { "type": "integer", "minimum": 0 }
},
"required": ["type", "peer"],
"additionalProperties": false
},
{
"type": "object",
"title": "ForwardedOffer",
"properties": {
"type": { "const": "offer" },
"from": { "type": "integer", "minimum": 0 },
"sdp": { "type": "string" }
},
"required": ["type", "from", "sdp"],
"additionalProperties": false
},
{
"type": "object",
"title": "ForwardedAnswer",
"properties": {
"type": { "const": "answer" },
"from": { "type": "integer", "minimum": 0 },
"sdp": { "type": "string" }
},
"required": ["type", "from", "sdp"],
"additionalProperties": false
},
{
"type": "object",
"title": "ForwardedIce",
"properties": {
"type": { "const": "ice" },
"from": { "type": "integer", "minimum": 0 },
"candidate": { "type": "string" }
},
"required": ["type", "from", "candidate"],
"additionalProperties": false
},
{
"type": "object",
"title": "ForwardedRelay",
"properties": {
"type": { "const": "relay" },
"from": { "type": "integer", "minimum": 0 },
"payload": {}
},
"required": ["type", "from", "payload"],
"additionalProperties": false
},
{
"type": "object",
"title": "Error",
"properties": {
"type": { "const": "error" },
"code": { "type": "string" },
"message": { "type": "string" }
},
"required": ["type", "code", "message"],
"additionalProperties": false
}
]
}
]
}
distributed.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/distributed.json",
"title": "Distributed (CRDT) Message",
"description": "CRDT anti-entropy IPC message (protocol.md § Distributed). The CrdtSync message rides the same lazily-ipc transport as Snapshot/Delta as a third IpcMessage variant: the externally-tagged envelope {\"CrdtSync\": {...}}. Supporting distributed/cell-model types are kept under $defs.",
"type": "object",
"required": ["CrdtSync"],
"additionalProperties": false,
"properties": {
"CrdtSync": {
"type": "object",
"description": "Anti-entropy sync frame: the sender advertises its per-peer stamp frontier and ships an op batch.",
"required": ["ops"],
"additionalProperties": false,
"properties": {
"frontier": {
"type": "array",
"items": { "$ref": "#/$defs/StampFrontierEntry" },
"description": "Per-peer highest observed stamp. Retained in full (not permission-filtered) so the receiver can compute a sound causal-stability watermark. Optional under #lzspecfrontiersuppress: omit (or send []) when unchanged since the last accepted frame; the receiver reuses its last-merged frontier."
},
"ops": {
"type": "array",
"items": { "$ref": "#/$defs/CrdtOp" },
"description": "Op batch this frame ships; permission-filtered by omission before serialization."
}
}
}
},
"$defs": {
"NodeId": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeId" },
"PeerId": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/PeerId" },
"NodeKey": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/NodeKey" },
"WireStamp": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/WireStamp" },
"IpcValue": { "$ref": "https://lazily.dev/schemas/defs.json#/$defs/IpcValue" },
"OpKind": {
"type": "string",
"enum": ["read", "write", "trigger_effect"],
"description": "Permission-gated operation kind. The three kinds are gated independently."
},
"RemoteOp": {
"type": "object",
"description": "Gated, serializable unit a peer requests.",
"properties": {
"kind": { "$ref": "#/$defs/OpKind" },
"node": { "$ref": "#/$defs/NodeId" }
},
"required": ["kind", "node"],
"additionalProperties": false
},
"MergeMechanism": {
"type": "string",
"enum": ["crdt", "lww", "ot", "lease", "custom"],
"description": "Convergence mechanism for a multi-write cell. `crdt` is the first normative mechanism (converges without coordination); the rest are reserved extension points (see cell-model.md). Every mechanism MUST be deterministic. An implementation MUST reject an unimplemented mechanism explicitly rather than aliasing it to `crdt`."
},
"CellKind": {
"description": "Static classification of a cell by concurrent-writer count. Single-writer cells take no merge; multi-write cells carry a pluggable merge mechanism. Multi-write is NOT a hardcoded `crdt` kind — see cell-model.md.",
"oneOf": [
{
"type": "object",
"properties": { "kind": { "const": "single_writer" } },
"required": ["kind"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "const": "multi_write" },
"merge": { "$ref": "#/$defs/MergeMechanism" }
},
"required": ["kind", "merge"],
"additionalProperties": false
}
]
},
"CellRegisterType": {
"type": "string",
"enum": ["lww", "mv", "pn-counter"],
"description": "CRDT register type for a `merge: crdt` cell (the value shape within the CRDT mechanism); distinct from MergeMechanism."
},
"StampFrontierEntry": {
"type": "array",
"description": "A (peer, WireStamp) tuple in the per-peer stamp frontier.",
"prefixItems": [
{ "$ref": "#/$defs/PeerId" },
{ "$ref": "#/$defs/WireStamp" }
],
"minItems": 2,
"maxItems": 2,
"items": false
},
"CrdtOp": {
"type": "object",
"description": "One CRDT cell op on the wire (state-based / CvRDT): the converged register/sequence/text state for `node`, tagged with the WireStamp that produced it and an optional wire-stable NodeKey. State-based, idempotent — safe to resend.",
"required": ["node", "key", "stamp", "state"],
"additionalProperties": false,
"properties": {
"node": { "$ref": "#/$defs/NodeId", "description": "Volatile target id; pair with `key` for stable addressing." },
"key": {
"description": "Wire-stable keyed address, or null when unset. Mirrors the lazily-rs derived struct: `key` is always present (null when unset), unlike NodeSnapshot/NodeAdd which omit it.",
"oneOf": [
{ "type": "null" },
{ "$ref": "#/$defs/NodeKey" }
]
},
"stamp": { "$ref": "#/$defs/WireStamp" },
"state": { "$ref": "#/$defs/IpcValue", "description": "The converged CRDT state to merge." }
}
}
}
}
receipts.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/receipts.json",
"title": "Causal Receipts",
"description": "Generic receipt/outcome projection for causally-linked commands or effect requests. This is not a transport ACK plane; observed/accepted are non-terminal, applied/rejected are terminal.",
"type": "object",
"required": ["CausalReceipts"],
"additionalProperties": false,
"properties": {
"CausalReceipts": {
"type": "object",
"required": ["receipts"],
"additionalProperties": false,
"properties": {
"receipts": {
"type": "array",
"items": { "$ref": "#/$defs/CausalReceipt" }
}
}
}
},
"$defs": {
"ReceiptOutcome": {
"type": "string",
"enum": ["observed", "accepted", "applied", "rejected"],
"description": "Outcome vocabulary. observed/accepted are non-terminal; applied/rejected are terminal."
},
"OptionalString": {
"oneOf": [
{ "type": "null" },
{ "type": "string" }
]
},
"CausalReceipt": {
"type": "object",
"required": [
"receipt_id",
"causation_id",
"observer",
"generation",
"outcome",
"reason",
"payload_hash"
],
"additionalProperties": false,
"properties": {
"receipt_id": {
"type": "string",
"minLength": 1,
"description": "Idempotency key for this receipt event."
},
"causation_id": {
"type": "string",
"minLength": 1,
"description": "Stable id of the command, event, or effect request this receipt observes."
},
"observer": {
"type": "string",
"minLength": 1,
"description": "Peer, process, or subsystem that produced the receipt."
},
"generation": {
"type": "integer",
"minimum": 0,
"description": "Producer/editor generation. Consumers discard receipts outside the current generation for the causation id."
},
"outcome": { "$ref": "#/$defs/ReceiptOutcome" },
"reason": { "$ref": "#/$defs/OptionalString" },
"payload_hash": { "$ref": "#/$defs/OptionalString" }
}
}
}
}
message-passing.json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/message-passing.json",
"title": "Command / RPC Message Plane",
"description": "Evented command message plane (command-plane-v1). CommandSubmit/CommandCancel/CommandEvents/CommandProjection are an additive sibling family to Snapshot/Delta/CrdtSync. RPC is a facade over this plane; transport ACKs and non-terminal events are NOT authority. Terminal command outcomes fold through CausalReceipt (receipts.json).",
"oneOf": [
{ "$ref": "#/$defs/CommandSubmitFrame" },
{ "$ref": "#/$defs/CommandCancelFrame" },
{ "$ref": "#/$defs/CommandEventsFrame" },
{ "$ref": "#/$defs/CommandProjectionFrame" }
],
"$defs": {
"CommandId": {
"type": "string",
"minLength": 1,
"description": "Stable, replay-safe id for a command. Dedupe and reconnect projection key off this id."
},
"OptionalString": {
"oneOf": [{ "type": "null" }, { "type": "string" }]
},
"DedupePolicy": {
"type": "string",
"enum": ["none", "same_idempotency_key", "same_command_id"],
"description": "How the admitter collapses concurrent/duplicate submits."
},
"CommandPolicy": {
"type": "object",
"required": ["dedupe", "supersede", "cancel_on_preempt"],
"additionalProperties": false,
"properties": {
"dedupe": { "$ref": "#/$defs/DedupePolicy" },
"supersede": {
"type": "boolean",
"description": "If true, a newer submit with the same idempotency key supersedes an older non-terminal command."
},
"cancel_on_preempt": {
"type": "boolean",
"description": "If true, the admitter may cancel a still-non-terminal command when it is preempted."
}
}
},
"CommandSubmit": {
"type": "object",
"required": [
"command_id",
"causation_id",
"source",
"target",
"namespace",
"name",
"authority_generation",
"idempotency_key",
"deadline_ms",
"policy",
"payload_type",
"payload_hash",
"payload",
"required_features"
],
"additionalProperties": false,
"properties": {
"command_id": { "$ref": "#/$defs/CommandId" },
"causation_id": {
"type": "string",
"minLength": 1,
"description": "Causal parent id (a command id or event id). Self-caused submits set this equal to command_id."
},
"source": {
"type": "string",
"minLength": 1,
"description": "Identity of the submitter (e.g. 'vscode-plugin', 'jetbrains-plugin')."
},
"target": {
"type": "string",
"minLength": 1,
"description": "Identity of the intended handler (e.g. 'project-controller')."
},
"namespace": {
"type": "string",
"minLength": 1,
"description": "Domain namespace that owns the payload schema (e.g. 'agent-doc'). Lazily owns the envelope, not the namespace."
},
"name": {
"type": "string",
"minLength": 1,
"description": "Command name within the namespace (e.g. 'editor_route')."
},
"authority_generation": {
"type": "integer",
"minimum": 0,
"description": "Authority/controller generation. Receipts and events outside this generation are stale and never update the projection."
},
"idempotency_key": {
"type": "string",
"minLength": 1,
"description": "Dedupe/supersede key (e.g. 'project-root:doc:run')."
},
"deadline_ms": {
"type": "integer",
"minimum": 0,
"description": "Deadline in milliseconds. 0 means no deadline."
},
"policy": { "$ref": "#/$defs/CommandPolicy" },
"payload_type": {
"type": "string",
"minLength": 1,
"description": "Fully-qualified domain payload type (e.g. 'agent-doc.editor_route.v1')."
},
"payload_hash": {
"type": "string",
"minLength": 1,
"description": "Content hash of the payload body (e.g. 'sha256:...')."
},
"payload": {
"$ref": "https://lazily.dev/schemas/defs.json#/$defs/IpcValue",
"description": "Domain payload as inline bytes or shared-memory blob reference. Lazily does not interpret the body."
},
"required_features": {
"type": "array",
"items": { "type": "string" },
"description": "Features the target must advertise or the submit fails closed."
}
}
},
"CommandCancel": {
"type": "object",
"required": [
"command_id",
"causation_id",
"source",
"authority_generation",
"reason"
],
"additionalProperties": false,
"properties": {
"command_id": { "$ref": "#/$defs/CommandId" },
"causation_id": {
"type": "string",
"minLength": 1,
"description": "Id of the cancel request itself (for its own receipt/replay)."
},
"source": { "type": "string", "minLength": 1 },
"authority_generation": {
"type": "integer",
"minimum": 0,
"description": "Generation the cancel targets. A stale-generation cancel is ignored."
},
"reason": { "$ref": "#/$defs/OptionalString" }
}
},
"CommandEventKind": {
"type": "string",
"enum": [
"observed",
"accepted",
"started",
"progress",
"cancelled",
"superseded",
"timed_out"
],
"description": "Progress/detail kinds. These are UX/diagnostics only and are NEVER terminal proof; terminal proof folds through CausalReceipt. cancelled/superseded/timed_out are surfaced here for UX but their terminal authority is a matching rejected receipt."
},
"CommandEvent": {
"type": "object",
"required": ["event_id", "command_id", "kind", "generation", "detail"],
"additionalProperties": false,
"properties": {
"event_id": {
"type": "string",
"minLength": 1,
"description": "Idempotency key for this event. Duplicate event_ids are no-ops."
},
"command_id": { "$ref": "#/$defs/CommandId" },
"kind": { "$ref": "#/$defs/CommandEventKind" },
"generation": {
"type": "integer",
"minimum": 0,
"description": "Authority generation. Events outside the current generation for the command are ignored."
},
"detail": {
"$ref": "#/$defs/OptionalString",
"description": "Optional human/diagnostics detail (queue position, retry advice, copied CLI output). Not proof of effect."
}
}
},
"CommandEventsFrame": {
"type": "object",
"required": ["CommandEvents"],
"additionalProperties": false,
"properties": {
"CommandEvents": {
"type": "object",
"required": ["events"],
"additionalProperties": false,
"properties": {
"events": {
"type": "array",
"items": { "$ref": "#/$defs/CommandEvent" }
}
}
}
}
},
"CommandStatus": {
"type": "string",
"enum": [
"submitted",
"accepted",
"running",
"applied",
"rejected",
"cancelled",
"superseded",
"timed_out"
],
"description": "Folded projection status. submitted/accepted/running are non-terminal; applied/rejected/cancelled/superseded/timed_out are terminal and backed by a terminal CausalReceipt."
},
"CommandProjectionEntry": {
"type": "object",
"required": [
"command_id",
"status",
"terminal",
"generation",
"reason",
"terminal_receipt_id",
"last_event_id"
],
"additionalProperties": false,
"properties": {
"command_id": { "$ref": "#/$defs/CommandId" },
"status": { "$ref": "#/$defs/CommandStatus" },
"terminal": {
"type": "boolean",
"description": "True iff a terminal CausalReceipt has folded into this command. accepted/queued/admission never sets terminal."
},
"generation": {
"type": "integer",
"minimum": 0,
"description": "Current authority generation for the command."
},
"reason": {
"$ref": "#/$defs/OptionalString",
"description": "Terminal reason (rejection cause, cancel reason, timeout); null while non-terminal or applied without reason."
},
"terminal_receipt_id": {
"$ref": "#/$defs/OptionalString",
"description": "Receipt id that made the command terminal, or null when non-terminal."
},
"last_event_id": {
"$ref": "#/$defs/OptionalString",
"description": "Last folded event id for incremental resync, or null when none."
}
}
},
"CommandProjectionFrame": {
"type": "object",
"required": ["CommandProjection"],
"additionalProperties": false,
"properties": {
"CommandProjection": {
"type": "object",
"required": ["generation", "commands"],
"additionalProperties": false,
"properties": {
"generation": {
"type": "integer",
"minimum": 0,
"description": "Authority generation this projection image was taken at."
},
"commands": {
"type": "array",
"items": { "$ref": "#/$defs/CommandProjectionEntry" }
}
}
}
}
},
"CommandSubmitFrame": {
"type": "object",
"required": ["CommandSubmit"],
"additionalProperties": false,
"properties": { "CommandSubmit": { "$ref": "#/$defs/CommandSubmit" } }
},
"CommandCancelFrame": {
"type": "object",
"required": ["CommandCancel"],
"additionalProperties": false,
"properties": { "CommandCancel": { "$ref": "#/$defs/CommandCancel" } }
}
}
}
statechart.json
This is a compute schema, not a wire message. It normatively defines the
declarative Harel/SCXML chart form used by conformance fixtures and
cross-language chart definitions. A chart is never serialized as a distinct
wire kind; only its converged active configuration crosses IPC/FFI as an
ordinary cell Payload. See State Charts.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/statechart.json",
"title": "StateChart",
"description": "Declarative form of a lazily state chart (Harel/SCXML subset). This is COMPUTE, not a wire message: a chart is never serialized over IPC/FFI as a distinct type — only its converged active configuration crosses the wire as an ordinary cell Payload. This schema fixes the cross-language declarative form used by conformance fixtures and chart definitions.",
"type": "object",
"required": ["initial", "states"],
"additionalProperties": false,
"properties": {
"initial": {
"description": "Default entry state of the root region. MUST resolve to a leaf by descending compound `initial`s (or, for a parallel root, to one leaf per region).",
"type": "string"
},
"context": {
"description": "Optional extended-state schema: host-resolved caller state over which guard expressions evaluate. Never serialized as part of the active configuration.",
"type": "object"
},
"states": {
"type": "object",
"minProperties": 1,
"additionalProperties": {"$ref": "#/$defs/State"}
}
},
"$defs": {
"State": {
"type": "object",
"additionalProperties": false,
"properties": {
"parent": {
"description": "Parent state id. Exactly one state (the root) has no parent.",
"type": "string"
},
"kind": {
"description": "Authoritative structural kind. If present, it MUST agree with the state's structural fields and child relation; contradictory declarations are malformed. If omitted, it is inferred: `history` when `history` is set; `parallel` when `parallel` is true; `compound` when the state has children; otherwise `atomic`. `final` cannot be inferred.",
"enum": ["atomic", "compound", "parallel", "history", "final"]
},
"parallel": {
"description": "If true, this state is an AND-state: its children are concurrent regions, all of which are active whenever this state is active. Mutually exclusive with `initial`.",
"type": "boolean"
},
"initial": {
"description": "Default child entered when this compound state is entered. Required for compound states; forbidden for parallel states.",
"type": "string"
},
"history": {
"description": "Marks this state as a history pseudo-state. `shallow` records/restores the direct children of the parent region; `deep` records/restores the full active leaf configuration. Resolve to `default` on first entry (no recorded history).",
"enum": ["shallow", "deep"]
},
"default": {
"description": "Default target for a history state when no configuration has been recorded for its region yet.",
"type": "string"
},
"on": {
"description": "Event → transition table. An event maps to a target id or a transition object. Transitions on compound/parallel states are reachable from any active descendant (event bubbling).",
"type": "object",
"additionalProperties": {"$ref": "#/$defs/Transition"}
},
"entry": {
"description": "Ordered actions fired when this state is entered, after its ancestors' entry actions.",
"type": "array",
"items": {"$ref": "#/$defs/Action"}
},
"exit": {
"description": "Ordered actions fired when this state is exited, before its ancestors' exit actions.",
"type": "array",
"items": {"$ref": "#/$defs/Action"}
},
"run": {
"description": "Ongoing (do) actions started on entry and cancelled on exit. Host-managed; not part of conformance replay.",
"type": "array",
"items": {"$ref": "#/$defs/Action"}
}
},
"allOf": [
{
"if": {"required": ["parallel"], "properties": {"parallel": {"const": true}}},
"then": {"not": {"required": ["initial"]}}
},
{
"if": {"required": ["history"]},
"then": {"not": {"anyOf": [{"required": ["initial"]}, {"required": ["parallel"]}]}}
}
]
},
"Transition": {
"description": "A transition. A bare string is shorthand for `{\"target\": <string>}`.",
"oneOf": [
{"type": "string"},
{
"type": "object",
"required": ["target"],
"additionalProperties": false,
"properties": {
"target": {
"description": "Target state id. MAY be compound or parallel; entry descends via `initial` (compound) or enters every region (parallel). MAY target a history state to resume its parent region.",
"type": "string"
},
"guard": {"$ref": "#/$defs/Guard"},
"action": {
"description": "Ordered actions fired after the exit set and before the enter set.",
"type": "array",
"items": {"$ref": "#/$defs/Action"}
},
"internal": {
"description": "If true, an internal transition: the source state is not exited/re-entered even when the target is the source or a descendant. Defaults to false (external).",
"type": "boolean"
}
}
}
]
},
"Guard": {
"description": "A guard predicate. A bare string is a named guard resolved by the caller's guard resolver (fail-closed if absent). An object is an extended-state expression the host evaluates against `context`.",
"oneOf": [
{"type": "string"},
{
"type": "object",
"required": ["expr"],
"additionalProperties": false,
"properties": {
"expr": {"type": "string"}
}
}
]
},
"Action": {
"description": "An action. A bare string is a named action resolved by the caller's action handler. An object carries an action name plus an opaque payload.",
"oneOf": [
{"type": "string"},
{
"type": "object",
"required": ["name"],
"additionalProperties": false,
"properties": {
"name": {"type": "string"},
"payload": {}
}
}
]
}
}
}
stdlib-fixture.schema.json
This compute schema validates the portable standard-library fixture corpus. It is not a wire message. See Portable Standard Library.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://lazily.dev/schemas/stdlib-fixture.schema.json",
"title": "Portable Lazily stdlib conformance fixture",
"type": "object",
"required": [
"fixture_version",
"feature",
"scenario_floor",
"assertion_floor",
"mutation_floor",
"scenarios",
"mutations"
],
"properties": {
"$schema": {
"type": "string",
"const": "https://lazily.dev/schemas/stdlib-fixture.schema.json"
},
"fixture_version": {
"const": 1
},
"feature": {
"enum": [
"stdlib_timer_v1",
"stdlib_timeout_v1",
"stdlib_revision_barrier_v1"
]
},
"scenario_floor": {
"type": "integer",
"minimum": 1
},
"assertion_floor": {
"type": "integer",
"minimum": 1
},
"mutation_floor": {
"type": "integer",
"minimum": 1
},
"scenarios": {
"type": "array",
"minItems": 1
},
"mutations": {
"type": "array",
"minItems": 1,
"items": {
"$ref": "#/$defs/mutation"
}
}
},
"allOf": [
{
"if": {
"properties": {
"feature": {
"const": "stdlib_timer_v1"
}
}
},
"then": {
"properties": {
"scenarios": {
"items": {
"$ref": "#/$defs/timerScenario"
}
}
}
}
},
{
"if": {
"properties": {
"feature": {
"const": "stdlib_timeout_v1"
}
}
},
"then": {
"properties": {
"scenarios": {
"items": {
"$ref": "#/$defs/timeoutScenario"
}
}
}
}
},
{
"if": {
"properties": {
"feature": {
"const": "stdlib_revision_barrier_v1"
}
}
},
"then": {
"properties": {
"scenarios": {
"items": {
"$ref": "#/$defs/barrierScenario"
}
}
}
}
}
],
"additionalProperties": false,
"$defs": {
"u64": {
"type": "integer",
"minimum": 0,
"maximum": 18446744073709551615
},
"nullableU64": {
"oneOf": [
{
"$ref": "#/$defs/u64"
},
{
"type": "null"
}
]
},
"expect": {
"type": "object",
"required": [
"outcome"
],
"properties": {
"outcome": {
"enum": [
"pending",
"fired",
"completed",
"timed_out",
"cancelled",
"unavailable",
"satisfied",
"disposed"
]
},
"deadline": {
"$ref": "#/$defs/nullableU64"
},
"fired_at": {
"$ref": "#/$defs/nullableU64"
},
"value": {
"type": [
"string",
"null"
]
},
"reason": {
"type": [
"string",
"null"
]
},
"revision": {
"$ref": "#/$defs/nullableU64"
},
"generation": {
"$ref": "#/$defs/nullableU64"
},
"operation_calls": {
"type": "integer",
"minimum": 0,
"maximum": 1
},
"cancellation_calls": {
"type": "integer",
"minimum": 0,
"maximum": 1
}
},
"additionalProperties": false
},
"mutation": {
"type": "object",
"required": [
"operator",
"must_fail"
],
"properties": {
"operator": {
"type": "string",
"minLength": 1
},
"must_fail": {
"type": "array",
"minItems": 1,
"items": {
"type": "string",
"minLength": 1
}
}
},
"additionalProperties": false
},
"timerScenario": {
"type": "object",
"required": [
"id",
"steps"
],
"properties": {
"id": {
"type": "string",
"minLength": 1
},
"steps": {
"type": "array",
"minItems": 1,
"items": {
"$ref": "#/$defs/timerStep"
}
}
},
"additionalProperties": false
},
"timerStep": {
"oneOf": [
{
"type": "object",
"required": [
"op",
"now",
"duration",
"expect"
],
"properties": {
"op": {
"const": "start"
},
"now": {
"$ref": "#/$defs/u64"
},
"duration": {
"$ref": "#/$defs/u64"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"now",
"expect"
],
"properties": {
"op": {
"const": "observe"
},
"now": {
"$ref": "#/$defs/u64"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
}
]
},
"timeoutScenario": {
"type": "object",
"required": [
"id",
"steps"
],
"properties": {
"id": {
"type": "string",
"minLength": 1
},
"steps": {
"type": "array",
"minItems": 1,
"items": {
"$ref": "#/$defs/timeoutStep"
}
}
},
"additionalProperties": false
},
"timeoutStep": {
"oneOf": [
{
"type": "object",
"required": [
"op",
"now",
"duration",
"expect"
],
"properties": {
"op": {
"const": "start"
},
"now": {
"$ref": "#/$defs/u64"
},
"duration": {
"$ref": "#/$defs/u64"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"now",
"operation",
"cancellation",
"expect"
],
"properties": {
"op": {
"const": "poll"
},
"now": {
"$ref": "#/$defs/u64"
},
"operation": {
"enum": [
"pending",
"completed",
"unavailable"
]
},
"value": {
"type": "string"
},
"cancellation": {
"enum": [
"pending",
"cancelled",
"unavailable"
]
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
}
]
},
"barrierScenario": {
"type": "object",
"required": [
"id",
"steps"
],
"properties": {
"id": {
"type": "string",
"minLength": 1
},
"steps": {
"type": "array",
"minItems": 1,
"items": {
"$ref": "#/$defs/barrierStep"
}
}
},
"additionalProperties": false
},
"barrierStep": {
"oneOf": [
{
"type": "object",
"required": [
"op",
"revision",
"required_revision",
"deadline",
"expect"
],
"properties": {
"op": {
"const": "start"
},
"revision": {
"$ref": "#/$defs/u64"
},
"required_revision": {
"$ref": "#/$defs/u64"
},
"deadline": {
"$ref": "#/$defs/nullableU64"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"now",
"predicate",
"cancellation",
"expect"
],
"properties": {
"op": {
"const": "observe"
},
"now": {
"$ref": "#/$defs/u64"
},
"predicate": {
"type": "boolean"
},
"cancellation": {
"enum": [
"pending",
"cancelled",
"unavailable"
]
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"now",
"observed_revision",
"predicate",
"expect"
],
"properties": {
"op": {
"const": "register_recheck"
},
"now": {
"$ref": "#/$defs/u64"
},
"observed_revision": {
"$ref": "#/$defs/u64"
},
"predicate": {
"type": "boolean"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"revision",
"predicate",
"expect"
],
"properties": {
"op": {
"const": "advance"
},
"revision": {
"$ref": "#/$defs/u64"
},
"predicate": {
"type": "boolean"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"expect"
],
"properties": {
"op": {
"const": "dispose"
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"op",
"key",
"expect"
],
"properties": {
"op": {
"const": "receipt"
},
"key": {
"type": "string",
"minLength": 1
},
"expect": {
"$ref": "#/$defs/expect"
}
},
"additionalProperties": false
}
]
}
}
}
Lean Formal Model
formal/lean is a small Lean 4 Lake package for the IPC Snapshot/Delta state
machine. It is a spec companion, not an implementation replacement.
The model proves the invariants that are easiest to blur across language bindings:
- delta epochs are strictly sequential (
epoch = base_epoch + 1); - gap, reorder, and restart cases fail closed to snapshot resync;
- equal
setwrites are silent; - equal memo recomputes suppress
slot_valueand downstream invalidation; - eager Signal changes publish concrete
slot_valueops rather than bareinvalidateops for their backing slot; - batch flushes carry a coalesced frontier and advance the IPC epoch once;
- command-plane projection (
LazilyFormal.Command): progress events never complete a command, only a terminal receipt does; stale generations are discarded; duplicate submits are idempotent; a cancel cannot override an applied command; conflicting terminal outcomes fail closed; reconnect projection is fold-equivalent; and an RPCcallcannot resolve before a terminal receipt. This mirrors the standalonelazily-formalmodel.
Run it through the local check target:
make check
Keep the Lean package narrow. JSON Schema, Rust implementation behavior, cross-language conformance fixtures, Loom/thread-safe tests, and live transport validation remain separate verification layers.
PRD: Native Distributed Queue Support
Status: Active — TopicCell semantic core shipped; distributed storage remains post-v1
Created: 2026-07-08
Depends on: v1 QueueCell + QueueStorage adapter seam
Problem
Applications using lazily need distributed queue semantics — producer/consumer across process boundaries, work distribution, event delivery — but face a dilemma:
-
External brokers (Kafka, RabbitMQ, NATS, Redis Streams, SQS) are production-grade but require provisioning and operating a separate service. This is heavy for embedded, edge, small-to-medium-scale, or prototype deployments where the operational overhead of a broker exceeds the application’s complexity budget.
-
CRDT-replicated queues (via lazily’s existing
CrdtPlaneRuntime) converge without coordination but cannot provide the semantics production queues need: destructive pop requires agreement (not merge), FIFO order requires a single sequencer, and exactly-once delivery requires a leader. CRDT is the right tool for collaborative editing (TextCrdt,SeqCrdt) and the wrong tool for queues. See § Background: Why Consensus, Not CRDT.
The embeddability advantage
lazily is embeddable as a library/app. Its advantage: a distributed queue can be built into the application — no external service to provision, no broker to operate, no ops team required — while still providing consensus-based strong consistency. This is the same value proposition as SQLite vs PostgreSQL: trade raw performance and operational features for zero-provisioning embedded simplicity.
Positioning
Use an external broker when one is already available. Established distributed queues have years of production hardening, tooling, client libraries, and operational experience. Until lazily’s native distributed queue reaches feature parity, the external broker is the better choice for production-scale workloads. The v1
QueueStorageadapter seam enables this today — aKafkaStorageorRedisStreamStoragebackend plugs into the same reactive shell.
Use lazily’s native distributed queue when embeddability matters. For embedded, edge, small-to-medium scale, zero-provisioning, or prototype deployments, a library-embedded distributed queue eliminates the operational cost of provisioning and managing a broker. When the application is already shipping lazily, the native queue adds zero new dependencies.
The native distributed queue is not a Kafka replacement. It is the right choice when provisioning a broker is more expensive than the problem it solves.
Target Use Cases
| Use case | Why native (not external broker) |
|---|---|
| Embedded / edge deployment | No ops team, no broker to provision |
| Small-to-medium scale work distribution | Broker overhead exceeds app complexity |
| Prototyping / development | Zero-provisioning; promote to broker later via adapter |
| Single-binary distributed apps | Ship one artifact, not app + broker |
| Applications already using lazily | No new dependency; reuses reactive substrate |
| Partition-tolerant control planes | In-process queue with Raft-level consistency |
Goals
- Consensus-based distributed queue (Raft replicated log), not CRDT.
- Zero external service dependencies — the queue is embedded in the application process; peers are other application instances.
- Builds on v1 — the
QueueStorageadapter seam is the integration point. The native distributed queue is a new backend (RaftQueueStorage), not a new primitive. - Path to parity — phased delivery that converges toward feature-comparability with established distributed queues for the embeddable niche.
- Cross-language — consistent semantics across every lazily binding via the shared conformance fixtures, same as every other lazily primitive.
Non-Goals (v1)
- Not replacing external brokers for production-scale workloads. The adapter seam
(
KafkaStorage,RedisStreamStorage,SqsStorage) is the recommended path when a broker is available. - Not CRDT-based. CRDT is the wrong algebra for destructive pop.
- Not implementing Raft from scratch. Use an existing, proven Raft library per binding (or a Rust core + FFI for other languages).
- Not partitioning / consumer groups / exactly-once transactions in Phase 1. These are Phase 4 parity features.
Background: Why Consensus, Not CRDT
A queue’s defining operation is destructive pop — “I claim this element exclusively; no one else may have it.” This is an agreement problem, not a merge problem.
| Property needed | CRDT provides? | Consensus provides? |
|---|---|---|
| Concurrent writes all survive merge | ✅ | N/A (serialized by leader) |
| Exclusive destructive pop (exactly-once) | ❌ (at-least-once) | ✅ |
| FIFO order matching push order | ❌ (fractional-index, may reorder) | ✅ (log index = total order) |
| Immediate capacity rejection (no overcommit) | ❌ (convergent is_full lags) | ✅ (leader rejects) |
| Single delivery ID for ack/dedup | ❌ | ✅ (leader assigns) |
| No tombstone growth under high churn | ❌ (head-pointer optimization needed) | ✅ (log compaction) |
CRDT is excellent when all operations are commutative and non-destructive (collaborative
text editing, counters, sets, shared maps). Queues are none of these. This is why every
production distributed queue (Kafka, RabbitMQ, NATS, Redis Streams, SQS) uses consensus,
not CRDT — and why lazily uses CRDT for TextCrdt/SeqCrdt but uses consensus for the
native distributed queue.
For the full analysis (CRDT tombstone growth under load, head-pointer vs tombstone GC, resurrection safety), see the design discussion captured in the distributed queue pressure-test (session document).
Architecture
Core insight: the Raft log IS the queue
A replicated log (via Raft) provides everything a distributed queue needs:
- Total order: log entries are committed with monotonically-increasing indices → FIFO is free.
- Leader-based writes: all pushes go through the leader → no concurrent-write conflicts, no fractional-index reordering.
- Durability: committed log entries survive leader failure → queue state is durable.
- GC: log compaction (snapshot + truncate below the lowest consumer cursor) reclaims consumed entries → bounded memory.
The queue is a cursor over the replicated log:
┌──────────────────────────────────────────────────────┐
│ Replicated Log (Raft) │
│ │
│ [Push A] [Push B] [Push C] [Push D] [Push E] ... │
│ 1 2 3 4 5 │
│ │
│ committed ──────────────────────────── ▲ ── tail │
│ consumer cursor │
└──────────────────────────────────────────────────────┘
│
▼
Queue contents = entries (consumer_cursor, tail]
GC: entries ≤ consumer_cursor are compactable
- Push = append
Entry { value }to the Raft log (leader serializes). - Pop = read entry at
consumer_cursor + 1, advance cursor (replicated via Raft state machine). - GC = log compaction below the lowest cursor (or a TTL/expiry floor).
This is architecturally closer to Kafka (log + consumer offset) than to RabbitMQ (AMQP delivery model). The log is append-only; destructive semantics live in the cursor, not in tombstones.
RaftQueueStorage as a QueueStorage backend
The native distributed queue is not a new primitive. It is a QueueStorage backend —
the same adapter seam that v1 defines for VecDequeStorage, KafkaStorage, etc.
┌─────────────────────────────────────────────────┐
│ QueueCell reactive shell │
│ (head/tail/closed version cells, invalidation) │
└───────────────────┬─────────────────────────────┘
│ QueueStorage trait
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
VecDeque RaftQueue KafkaStorage
(local, (embedded (external broker,
default) consensus) via adapter)
This means:
- The reactive shell (invalidation, backpressure, closure) is shared across all backends.
- The consensus logic lives entirely in
RaftQueueStorage. - Users switch between local / native-distributed / external-broker by swapping the storage backend, with no change to the reactive API.
Transport
RaftQueueStorage reuses lazily’s existing transport infrastructure:
- IPC (in-process, cross-thread) — same machine.
- WebSocket — cross-machine, cloud/edge.
- WebRTC — peer-to-peer, NAT traversal.
No new transport layer. The Raft RPCs (RequestVote, AppendEntries, etc.) ride the same
DataChannel abstraction that the CRDT plane and command plane already use.
Consensus implementation
Use an existing, proven Raft library — do not implement Raft from scratch.
| Binding | Candidate |
|---|---|
| lazily-rs | openraft or raft-rs |
| lazily-py | FFI to Rust RaftQueueStorage core |
| lazily-zig | FFI to Rust core, or native Zig Raft if available |
| lazily-js | WASM-compiled Rust core, or WebSocket client to a remote leader |
| lazily-go | Native Go Raft (etcd/raft) — idiomatic for Go |
| lazily-kt | FFI to Rust core, or native JVM Raft (copy-cat) |
The Rust core is the reference implementation; other bindings either FFI to it or use a
native Raft library in their language. The QueueStorage trait guarantees semantic parity
regardless of the underlying Raft implementation.
Relationship to v1
The v1 deliverables are the foundation for native distributed queue support:
| v1 deliverable | Role in the distributed queue |
|---|---|
QueueCell reactive shell | The API surface — unchanged whether storage is local or distributed |
QueueStorage adapter trait | The integration point — RaftQueueStorage is a new backend |
VecDequeStorage default | The local reference; RaftQueueStorage must match its observable FIFO contract |
| FIFO-order spec clause | The cross-backend invariant — consensus-backed or broker-backed, FIFO must hold |
| Closure observable contract | Shared across backends — close semantics are shell-level |
No v1 spec/formal work is blocked by the distributed queue PRD. The adapter seam carries the distributed story: v1 ships the seam; this PRD ships the consensus backend.
Phased Delivery
Phase 0 — v1 foundation (current scope)
- Local
QueueCell(SPSC primitive + MPSC usage rule). QueueStorageadapter trait +VecDequeStoragedefault.TopicCelllocal semantic contract + conformance + Lean reference.WorkQueueCellportable local-authority lifecycle + conformance + Lean safety reference; distributed/HA claim serialization remains a Phase 2 integration.- Reactive shell: closure, bounded/backpressure, ordering contract.
Deliverable: a local queue primitive with a pluggable backend seam, ready for distributed backends.
Phase 1 — Consensus core
RaftQueueStorage: Raft replicated log + consumer cursor.- Single-partition, single-consumer distributed queue.
- Log compaction (GC below consumer cursor).
- Transport over existing
DataChannel(IPC / WebSocket / WebRTC). - Conformance fixtures for distributed FIFO, durability under leader failover, GC safety.
Deliverable: an embeddable distributed queue with strong consistency, no external dependencies. This is the milestone that delivers the PRD’s core value proposition.
Phase 2 — WorkQueueCell (exactly-once handoff)
- Reuse the shipped local
push/claim/ack/nack/reap_expiredlifecycle and cross-language fixtures unchanged. - Leader-based exclusive handoff over the Raft log.
- Pop = advance cursor with ack; unacked entries are redelivered.
- Pending entries list (consumer failure recovery).
- Dead-letter queue (poison-message handling).
Receiptintegration for at-most-once effect authority.
Deliverable: distributed exactly-once assignment authority for the shipped competing-consumer shell — the semantic that CRDT cannot provide and the reason production queues use consensus.
Phase 3 — TopicCell (multi-cursor broadcast)
- Each subscriber maintains its own cursor over the Raft log.
- Cursor persistence (survives subscriber restart).
- Log GC bounded by the slowest subscriber’s cursor.
- Fan-out semantics (one push → all subscribers receive).
Deliverable: pub/sub broadcast — the event-delivery use case.
Status: the storage-independent semantic contract, replay fixtures, and universal Lean proofs shipped in v0.31.0. Wiring those cursors to a Raft-backed durable log remains part of the post-v1 distributed-storage implementation.
Phase 4 — Parity features
| Feature | Parity target | Source |
|---|---|---|
| Partitioning | Key-based routing across multiple Raft groups | Kafka partitions |
| Consumer groups | Shared cursor among group members; rebalance on join/leave | Kafka consumer groups |
| Exactly-once delivery | Transactional consumer (consume + ack + commit in one Raft round) | Kafka transactions |
| Persistence | Durable log (fsync on commit); WAL recovery | Kafka / RabbitMQ durability |
| Visibility timeout / lease | Consumer lease with TTL; redelivery on expiry | SQS |
| Priority | Priority-weighted cursor advancement | RabbitMQ priority queues |
| Monitoring | Depth / lag / throughput / consumer-position metrics | Kafka / RabbitMQ dashboards |
| Flow control | Bounded queue with quota; push rejection with backpressure signal | Kafka quota / RabbitMQ prefetch |
Deliverable: feature-comparability with established distributed queues for the embeddable niche.
Parity Boundary
lazily’s native distributed queue will not match established brokers on:
- Raw throughput — dedicated brokers (Kafka) are optimized for millions of ops/sec with zero-copy kernel-bypass. lazily targets the embeddable niche, not the hyperscale niche.
- Operational tooling — Kafka’s ecosystem (Connect, Streams, Schema Registry, KSQL) is decades of engineering. lazily provides the queue, not the platform.
- Language-specific client libraries — Kafka has clients in 30+ languages. lazily has its own bindings; external-broker integration uses the adapter seam.
The parity target is semantic (FIFO, exactly-once, persistence, ack/nack, dead-letter, consumer groups) and embedding-grade operational (zero-provisioning, in-process, no external services) — not hyperscale performance.
Risks / Open Questions
| Risk | Mitigation |
|---|---|
| Raft library correctness / maturity | Use proven libraries (openraft, etcd/raft); conformance fixtures validate semantics |
| Performance vs dedicated brokers | Explicitly non-goal for hyperscale; target embeddable niche |
| Multi-language Raft divergence | Rust core + FFI for bindings that can’t host a native Raft; conformance fixtures enforce parity |
| Log compaction correctness under failover | Formal model + conformance fixture for GC safety under leader crash |
| Partitioning strategy (Phase 4) | Defer to Phase 4 design; single-partition is sufficient for embeddable niche in Phase 1 |
| When to recommend native vs external broker | Clear decision criteria in docs: scale threshold, existing infrastructure, operational capacity |
| Raft group membership changes | Use the Raft library’s built-in membership change protocol; don’t reinvent |
| Cross-backend snapshot interop | RaftQueueStorage defines its own snapshot format; cross-backend interop requires explicit format agreement (per v1 wire/snapshot clause) |
Decision Criteria: Native vs External Broker
| Criterion | Native (RaftQueueStorage) | External (KafkaStorage etc.) |
|---|---|---|
| No external service to provision | ✅ | ❌ |
| Zero new dependencies (already using lazily) | ✅ | ❌ |
| Embeddable / single-binary deployment | ✅ | ❌ |
| Edge / resource-constrained environment | ✅ | ❌ |
| Production-scale throughput (>100K ops/sec) | ❌ | ✅ |
| Mature operational tooling / monitoring | ❌ | ✅ |
| Rich client ecosystem (30+ languages) | ❌ | ✅ |
| Existing infrastructure / team expertise | ❌ | ✅ |
| Exactly-once transactions (until Phase 4) | ❌ | ✅ |
Rule of thumb: if you already have a broker, use it. If provisioning one is more expensive than the problem, use the native queue.
References
- v1 QueueCell spec — local primitive +
QueueStorageadapter - Cell Model § Merge mechanisms —
leaseandotreserved mechanisms relevant to consensus/authority-based queues - Command / RPC Message Plane —
command-plane-v1transport reused byRaftQueueStorage - Wire Protocol § Distributed — existing distributed plane (CRDT); the consensus plane is its sibling, not its replacement
- Reactive Graph — the shell that wraps every
QueueStoragebackend - Conformance Fixtures — the cross-language parity enforcement layer