Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Durable Effect Sinks (#lzdurablesink)

Lazily fixes one direction of authority between live reactive state and durable storage:

cold durable state ──hydrate once──▶ live Lazily state
                                      │
                                      ▼
                          computed projection or ordered fact stream
                                      │
                                      ▼
                              Effect / AsyncEffect
                                      │
                                      ▼
                             write-only durable sink
                                      │
                                ack / failure
                                      └────────▶ live Lazily state

Invariant. While a Lazily runtime is live, transitions are decided from Lazily state. Durable storage receives a projection or an ordered fact as an effect. A sink MUST NOT reload storage and use it to arbitrate the transition it is currently persisting.

This makes durable storage an effect sink. It does not claim that every effect is durable, nor that durable storage is always eventually consistent. Lazily is not a database framework: applications continue to own serialization, transactions, schema migration, storage selection, and startup recovery. Lazily ships no SQLite, filesystem, or cloud-store adapter as part of this decision — the sink is an application-owned trait (see examples below). An application sink that writes to SQLite, and the bindings’ own SQLite outbox stores, follow the SQLite store contract (#lazilysqlitestoreopen): an explicit 30-second busy timeout, and no deferred read-then-write transaction, so a write contended by another writer waits for the lock instead of failing with database is locked.

Authority rule

  • Live coordination, compare-and-swap, ownership, deduplication, leases, and transition selection happen in Lazily state.
  • Durable I/O runs from an Effect / AsyncEffect, or from a TopicCell / DurableOutbox drain when history must be lossless.
  • A runtime sink is write-only with respect to transition authority. Loading and migration belong to a separate startup hydrator that runs once before the runtime is live, not on the decision seam.
  • Computed values and transition reducers stay pure — a reducer MUST NOT perform I/O or read storage to decide a transition.
  • Success advances a monotone acknowledgement such as durable_through(epoch).
  • Failure stays represented in live state as pending / retrying / backpressured. It MUST NOT trigger a database reload at the decision seam.
  • Values on the existing Ephemeral plane MUST NOT enter a durable sink. Reuse the existing Durable marker — do not invent a second marker hierarchy. (The Ephemeral-never-Durable separation is already pinned by Presence.ephemeral_never_durable.)

Projection vs history — pick the shape before the API

Ordinary Effect / AsyncEffect observe settled reactive state. Lazily already coalesces effect reruns across a batch, so a batch containing A → B → C may persist only C. That is correct for a durable projection (current/recoverable state). Durable history has a different contract: every accepted fact must survive and remain ordered. It uses the existing TopicCell / DurableOutbox family — stable cursor, replay, idempotency key, and monotone acknowledgement — not ordinary effects. Do not modify ordinary effects to retain intermediate values; that would weaken their coalescing semantics and duplicate the ordered-stream primitives.

Before choosing an API, answer two questions:

  1. Is this sink persisting the latest projection or an ordered history?
  2. Must the transition be durable before it is externally visible?

Those two answers fix the shape:

NeedLazily sourceSink contract
Latest recoverable stateComputed read by Effect / AsyncEffectIdempotent upsert of the latest epoch
Every accepted factTopicCell or existing DurableOutboxAppend / replay / ack with a stable cursor
Durable before visibleOrdered fact + application acknowledgement cellVisibility waits for monotone durable_through
Ephemeral statePresence / Ephemeral primitivesPersistence rejected

The separate durable-owner contract applies when accepted transitions must commit before acknowledgement and a fresh runtime must rebuild from the durable image. Its EventHistory and Snapshot modes are authoritative storage contracts. LatestDurableProjectionCore below remains a conflating egress state machine and can advertise only latest_state_only; it cannot certify complete_history.

Latest durable projection core (#lzlatestdurableprojection)

An ordinary Effect coalesces settled graph writes, but by itself it does not own an in-flight claim. If a newer projection appears while the sink is writing, an application-local boolean such as saving cannot tell whether a late success belongs to the current projection, whether it may clear pending work, or whether it came from an actor incarnation that has already disconnected.

LatestDurableProjectionCore<K, V> is the canonical pure state machine for that boundary. It is keyed so unrelated documents, records, or resources may progress independently. The core owns one global sink generation and, for every key, at most one each of:

  • desired: Desired<V> { epoch, value } — the latest projection not currently claimed;
  • inflight: LatestDurableEnvelope<K, V> { generation, key, epoch, value } — the exact projection claimed by the sink Effect; and
  • durable_through: Option<u64> — the monotone frontier known to be represented by durable state.

Epochs are caller-issued and MUST increase monotonically per key. An epoch names one immutable value; reusing an epoch for another value is a conflict. The core does not retain intermediate pending values: accepting a newer desired epoch replaces an older desired, but MUST NOT advance durable_through merely because the older value was superseded.

The normative operations and outcomes are:

OperationOutcomesState transition
upsert_desired(key, epoch, value)Accepted, Unchanged, AlreadyDurable, StaleEpoch, EpochConflictA strictly newer epoch becomes desired, replacing only the older pending desired. Equal epoch/value is idempotent; equal epoch/different value fails closed. An epoch at or below durable_through is already durable, while an epoch older than retained desired/in-flight work is stale.
claim(key, generation)Claimed(envelope), Empty, Busy, StaleGenerationWith the current generation, moves desired to inflight. A key with an in-flight envelope is busy even when it also has a newer desired value.
ack_applied(key, generation, epoch)Advanced { durable_through }, Unchanged, UnknownEpoch, StaleGenerationOnly an exact current-generation in-flight envelope may advance the frontier and clear itself. A duplicate acknowledgement at or below the existing frontier is unchanged. No acknowledgement clears a newer desired.
fail_retryable(key, generation, epoch)Pending, Superseded, UnknownEpoch, StaleGenerationAn exact failure clears inflight. It restores that value as desired when no newer value exists; otherwise the newer desired value remains and the failed older value is reported superseded. The latest value therefore remains retryable.
reconnect(new_generation)Advanced, Unchanged, StaleGenerationA strictly newer generation fences the old sink actor. Each in-flight value is restored to desired unless a newer desired value already supersedes it; all in-flight slots are then cleared. The durable frontier never moves.

ack_applied and fail_retryable carry the key, generation, and epoch from the claimed envelope. A stale-generation or unknown-epoch result MUST leave every entry and every reader version unchanged. In particular, a success from a disconnected sink actor cannot acknowledge work claimed by its replacement.

The graph shell is deliberately a reactive projection, not a second command actor:

Source<K,V,epoch> ─▶ Computed latest projection ─▶ upsert_desired
                                                    │
                                      keyed single-flight Effect
                                                    │ claim envelope
                                                    ▼
                                         application-owned sink
                                                    │
                                  ack_applied / fail_retryable Source
                                                    └──────▶ core readers

There MUST be at most one claimed sink effect per key. Different keys may be in flight concurrently. Failure must not create an immediate reactive spin: a shell re-attempts retained desired work when its retry policy says the sink is ready, when a newer desired projection arrives, or after a reconnect. If a sink adapter also owns an ordered per-key mutation lane (for example an editor document worker), it MUST execute the claimed write on that same lane after previously accepted mutations; creating a separate command actor would reintroduce the race this primitive removes.

This core is for latest recoverable state only. It MUST NOT be used when every accepted intermediate value is a fact that must survive; that remains the TopicCell / DurableOutbox contract.

Generic effect compare-and-swap (#lzeffectcas)

LatestDurableProjectionCore covers a latest-value write whose success is a single durable frontier. A live application often needs the same ownership algebra around an opaque authority token: apply a Git ref update only at one parent, mutate an editor document only at one authority hash, or update a controller row only at one lineage/version. EffectCasCore<K, A, P, R> is the generic protocol for those effects. It composes the latest-projection rules; it does not create an application-local queue or retry loop.

Authority split

Lazily owns:

  • stable effect identity and immutable epoch/plan identity;
  • the authority token observed for the exact attempt;
  • one in-flight attempt per key and actor-generation fencing;
  • latest-wins pending intent, explicit supersession, retry eligibility, and deterministic logical-clock retry times;
  • exact attempt/receipt matching and a monotone applied frontier; and
  • the cursor and receipt lineage of an ordered multi-step effect plan.

The application adapter owns:

  • observe(key), a side-effect-free read of the backend’s opaque authority;
  • one atomic compare_and_apply(attempt) boundary;
  • backend durability and idempotency; and
  • mapping the backend result to exactly one of Applied { receipt, authority }, Superseded { observed }, Retryable(error), or Rejected(error).

The earlier rule that no read/CAS surface reaches a runtime effect means that an application callback MUST NOT reload storage and arbitrate transition policy. It does not forbid the Lazily-owned adapter from observing the token needed by its atomic comparator. The adapter MUST NOT retry, choose a newer desired intent, clear retained work, or advance a frontier. A failed comparison returns Superseded with zero backend mutation; Lazily decides whether a newer desired intent remains eligible.

State and operations

For each key the core retains at most one pending intent and one inflight attempt, plus retry evidence, the monotone applied_through epoch, and the most recent terminal receipt. An immutable intent contains EffectIdentity, epoch, key, initial expected authority, and a non-empty ordered plan. An exact attempt identity contains effect identity, actor generation, epoch, plan-step cursor, and retry-attempt number.

OperationRequired behavior
upsert_desired(intent)Accept only a strictly newer epoch; equal immutable intent is idempotent (Unchanged), equal epoch with different identity/authority/plan is a conflict (EpochConflict), and an epoch at or below applied_through is already applied. The equality is against the ORIGINAL submitted intent (identity, epoch, key, initial expected, plan) retained as pending, in flight, or as the last terminal intent — never against the next step’s expected authority that a completed plan step advanced. Re-sending the same intent mid-plan is therefore Unchanged. Replacing pending work records explicit supersession and retains any completed-step receipts.
claim(key, generation, now)With the current generation and an eligible logical clock, move pending work into the sole in-flight slot. Return Busy, Empty, Waiting { retry_at }, or StaleGeneration without mutation otherwise.
record_applied(attempt, receipt, authority)Match the exact attempt. Retain the receipt; either advance the plan cursor using the returned authority as the next step’s expected token, or complete and monotonically advance applied_through. A duplicate terminal receipt is idempotent. A newer desired intent stops the old plan immediately.
record_superseded(attempt, observed)Match and terminally supersede the attempt without advancing applied_through; the adapter has mutated nothing.
record_retryable(attempt, error, retry_at)Match and restore the same effect identity/cursor with an incremented attempt number and caller-supplied logical retry time, unless a newer desired epoch already supersedes it.
record_rejected(attempt, error)Match and terminally reject the intent. Rejection is permanent and never aliases retry.
reconnect(new_generation)Fence older receipts. Requeue each in-flight cursor with the same effect identity and completed-step lineage unless a newer desired intent supersedes it.

The pure core owns no thread, executor, timer, or I/O. A reactive shell projects pending, in-flight, retry, receipt, and terminal state and invalidates readers only after releasing the core lock. Scheduling observes pending_keys(now) and invokes an adapter outside the lock.

Invariants and restart

  1. At most one adapter call is in flight per key; independent keys may progress concurrently.
  2. A stale generation, unknown attempt, or non-matching receipt changes no state and no reader version.
  3. applied_through is monotone and advances only from an exact terminal apply.
  4. A compare mismatch mutates no backend state and never advances the frontier.
  5. Retry and reconnect preserve effect identity, plan cursor, and completed-step receipts; a newer epoch is the only intent allowed to replace them.
  6. Rejected is terminal and remains observably distinct from Retryable.

Crash recovery hydrates the retained intent, generation, cursor, logical retry time, and receipts before the runtime becomes live. Reconnect then advances the actor generation and replays the same effect identity. A crash MUST NOT mint a new identity for already-retained work.

Heterogeneous effect plans are ordered, not atomic

CAS is atomic only inside one adapter boundary. Lazily cannot make Git, editor state, SQLite, and filesystem cleanup one physical transaction. A heterogeneous plan therefore applies canonical authority first, derives each later step from the prior applied receipt/authority, validates that lineage before every step, and stops on supersession while retaining only unapplied work. Later steps MUST be idempotent and resumable. This is receipted sequencing, never cross-backend atomicity.

The language-neutral corpus at conformance/egress/effect_cas.json pins exact apply and duplicate receipt, zero-mutation mismatch, pending/in-flight supersession, stale receipt fencing, independent keys, reconnect replay, partial-plan resume (including a same-epoch re-send of the original intent while the advanced cursor is pending and while it is in flight, both unchanged), between-step supersession, and rejection-versus-retry.

Application-owned sink trait

The sink is a narrow, write-only trait owned by the application. The reactive graph holds no reference to a query/hydration interface — those live in a separate module used only by the startup hydrator, never passed into the runtime effect. (Phase 3 adds architecture tests that fail if a hot-path actor module imports the persistence query/CAS or file-lock modules.)

The two examples below sketch the trait shape in Rust-flavoured pseudocode. The store API is deliberately minimal — upsert_latest for a projection, append_fact for history — and always carries the epoch so success can advance a monotone durable_through.

Example 1 — claimed current-state projection

A Computed projects the actor’s current state. An Effect reads it and upserts only the settled value into LatestDurableProjectionCore; intermediate batch values are coalesced away by Lazily’s existing effect-batch dedup, and pending values are further conflated by the core. A keyed sink Effect claims the exact envelope it writes. Acknowledgement advances durable_through; a sink failure returns the latest desired value to pending without reloading storage.

#![allow(unused)]
fn main() {
// Application-owned latest-projection sink. It has no authority comparator.
trait ProjectionSink {
    fn upsert_latest(&mut self, epoch: u64, state: &ActorState) -> Result<(), SinkErr>;
}

// Live state. Pure reducer; no I/O.
let live_state: Source<ActorState>  = ctx.source(initial);
let epoch:      Source<u64>          = ctx.source(0);
let projected: Computed<ActorState> = ctx.computed(|c| c.get(live_state).clone());
let durable = LatestDurableProjectionCore::new(/* generation = */ 1);

// Settled projection Effect. The core, not the callback, owns retained intent.
ctx.effect(|c| {
    let s = c.get(projected);
    let e = c.get(epoch);
    durable.upsert_desired(ActorKey, e, s.clone());
});

// Keyed single-flight sink Effect. Real shells schedule retries without spinning.
if let Claimed(envelope) = durable.claim(ActorKey, sink_generation) {
    match sink.upsert_latest(envelope.epoch, &envelope.value) {
        Ok(()) => durable.ack_applied(
            envelope.key, envelope.generation, envelope.epoch,
        ),
        Err(_) => durable.fail_retryable(
            envelope.key, envelope.generation, envelope.epoch,
        ),
    }
}
}

Example 2 — lossless ordered fact sink

Every accepted fact must survive and stay ordered, so the source is a TopicCell (or the existing DurableOutbox), drained by an effect into the application store. Replay from a stable cursor covers every epoch after the durable frontier; duplicate delivery is idempotent by the event_id key. This is the DurableOutbox append-before-send / replay-from-cursor / ack_through contract, applied as a sink.

#![allow(unused)]
fn main() {
trait HistorySink {
    fn append_fact(&mut self, epoch: u64, fact: &Fact) -> Result<(), SinkErr>;
    fn ack_through(&mut self, epoch: u64);
}

let facts: TopicCell<Fact> = ctx.topic();
// Drain: every unacked fact is appended in order; ack advances the cursor.
ctx.effect(|c| {
    while let Some(fact) = facts.read_after(cursor) {
        sink.append_fact(fact.epoch, &fact)?;
        sink.ack_through(fact.epoch);          // cursor advances; GC-safe below it
        cursor = fact.epoch;
    }
});
}

Visibility policy (caller-chosen)

A caller picks an explicit policy per transition; Lazily mandates none:

  • eventual_projection — live state is immediately authoritative; the sink persists in the background and may lag.
  • durable_before_applied — external visibility waits for the durable_through(epoch) acknowledgement before the transition is observed as applied (an ordered fact plus an application acknowledgement cell).
  • ephemeral — no durable sink; the value lives on the Ephemeral plane.

Cold restart

Restart recovery is the hydrator’s job, run once at startup before the runtime is live: load the last acknowledged projection (or replay the ordered fact log up to the durable cursor) into live state, then resume. After hydration, authority is live-only — a sink failure during a live transition never rolls authority backward by rehydrating storage at the decision seam.

Formal backstop

lazily-formal/LazilyFormal/DurableSink.lean and lazily-formal/LazilyFormal/LatestDurableProjection.lean pin the load-bearing invariants: durable_through is monotone; a batched projection persists only the settled epoch (coalescing); an older success cannot erase a newer desired value; failure leaves the latest desired value pending; reconnect fences stale actor acknowledgements; an ordered history replays every epoch past the durable cursor; and a sink failure leaves live authority unchanged (no rehydrate-at-decision- seam). The Ephemeral-never-Durable separation is already proven in Presence.ephemeral_never_durable.