lazily Benchmark Results

Generated benchmark data for the lazily reactive primitives library.

Benchmark Results

Generated for package lazily version 0.56.0.

Environment: rustc 1.97.0 (2d8144b78 2026-07-07) on x86_64-unknown-linux-gnu.

Refresh command:

python3 scripts/update-benchmark-results.py

Regression workflow:

cargo bench --features instrumentation,thread-safe -- --save-baseline before
# apply the performance patch
cargo bench --features instrumentation,thread-safe -- --baseline before
python3 scripts/update-benchmark-results.py --no-run

Regression budgets enforced by python3 scripts/update-benchmark-results.py --check:

Every ceiling is DERIVED from the recorded spread, never hand-typed; refresh the spreads with python3 scripts/update-benchmark-results.py --measure-budget-spread N. A counter with zero spread across idle, loaded and 2-core-pinned runs measures work items rather than interleaving, so it is enforced EXACTLY. A counter whose spread is under half its maximum gets headroom of one full observed range above the observed maximum. A counter whose spread exceeds half its maximum is measuring the scheduler, not the code: it is recorded as an observation and NOT enforced, because a gate that reddens on noise trains everyone to ignore it.

Budgets use lock acquisition counts instead of elapsed wait/hold time. Those counts are only deterministic for the 22 counters classified as such above; 19 of 51 gated counters are scheduling-dominated and carry no regression signal at all.

Synchronization strategy adoption gate:

Candidates do not replace the current strategy before the same run reports throughput, p50/p95 latency, and lock-site budgets for the required 8/16-worker cases.

Required latency evidence uses Criterion sample per-iteration timing.

Watch-item A/B follow-up:

Criterion estimates are local mean wall-clock time per iteration.

Instrumentation snapshots are single local profile runs captured by examples/instrumentation_profile.rs.

ThreadSafe lock attribution for contention profiles:

Scale (≥1M cells) — #lzscalebench

The scale group in the generated section above is a rigorous criterion benchmark over a spreadsheet-shaped graph of N input cells + N formula slots (formula[i] = input[i] + input[i-1]). At the default N = 1_000_000 that is ~2,000,000 reactive nodes. It is gated behind the scale-bench feature so a plain cargo bench skips it; the benchmark generator enables the feature so the group is tracked by make benchmark-check. Run it directly, or at a larger size:

cargo bench --features scale-bench --bench scale
LAZILY_SCALE_N=2000000 cargo bench --features scale-bench --bench scale

What the four cases show at N = 1_000_000 (reference machine below): build constructs 2M nodes (~0.12 s), cold_full_recalc computes every formula from cold (~0.105 s), full_recalc_invalidate_all re-edits every input and recomputes the whole sheet (~0.080 s), and viewport_recalc edits one input and reads only a 1,000-cell viewport — ~3.7 µs, ~21,000× cheaper than a full recalc because the lazy pull-based model leaves off-viewport formulas dirty and never recomputes them (the property a viewport-rendered spreadsheet needs). (build/cold_full_recalc/full_recalc_invalidate_all are unaffected by the v0.22.2 #lzslotfastpath refresh fast path — they are cold/slow-path — so their figures are retained from the original run; only viewport_recalc, which is ~998/1000 cache-hit reads, moved, by the controlled A/B below. The generated scale rows in the table above reflect the latest single criterion run on this host and drift with host load for the allocation-heavy build/cold cases; the curated baseline here is the reference.)

Memory (not captured by criterion): building 2,000,000 nodes uses ~414 MiB RSS, i.e. ~216 B/node, so 1M populated formula cells land in the low hundreds of MiB.

Spreadsheet cell-count context

How the two dominant spreadsheets bound a sheet:

Google Sheets (10M cells) — measured. Modeled as 5,000,000 input cells + 5,000,000 formula cells (= 10M cells) by running the bench at LAZILY_SCALE_N=5000000. Criterion median on the cross-language reference machine (AMD Ryzen 9 9950X3D), pinned to one core (taskset -c 4) and run serially so nothing contends for L3 / memory bandwidth:

So lazily backs a full-capacity Google Sheets workbook: build under a second, full recompute ~0.5 s, and — crucially — viewport recalc stays ~3.8 µs independent of sheet size (it was ~3.7 µs at 1M too), because the lazy pull-based model only recomputes the cells you read. Reproduce: LAZILY_SCALE_N=5000000 cargo bench --features scale-bench --bench scale. Across the three implementations lazily-rs holds the cheapest viewport reads (3.7–3.8 µs); see the cross-language table in lazily-zig’s BENCHMARKS.md for the full head-to-head.

Controlled A/B isolating the v0.22.2 #lzslotfastpath refresh fast path on viewport_recalc (--save-baseline pre_fix, same session, toggling only src/context.rs between 8c64f33 and 1390a6e): 13.78 µs → 4.49 µs, −64.1% (p=0.00) at N = 1_000_000. Only ~2 of the 1,000 viewport cells recompute; the other ~998 are cache-hit slot reads, each now ~7 ns cheaper because refresh_slot early-returns on a clean hit instead of cloning the dependency Vec and walking deps.

Microsoft Excel (17.18B grid) — sparse, not dense. Excel’s 1,048,576 × 16,384 = 17,179,869,184 is the grid capacity, not a populated-cell count. Building all 17.18B cells densely would need ~7 TB at ~216 B/node — infeasible and unrepresentative: real sheets populate a tiny fraction of the grid, and lazily’s storage is a sparse arena (Vec<Option<Node>> with a free-list) that only allocates cells you actually create. The practical limit is therefore populated cells vs. available RAM, not the 17.18B grid. With the flat per-node cost above (~216 B, ~70–100 ns/cell), capacity ≈ available RAM ÷ ~216 B — e.g. this 186 GB host could hold on the order of ~10⁸–10⁹ populated cells, far beyond any realistically-populated Excel sheet. The scale group’s linear scaling (1M → 10M held ~constant per-cell cost) is the evidence that the model extrapolates rather than degrading at spreadsheet capacity.

Cross-library comparison — #lzscalecompare

Head-to-head against leptos_reactive (Leptos 0.6’s fine-grained reactivity) on the identical spreadsheet graph (N input signals + N formula memos, formula[i] = input[i] + input[i-1]), in the same criterion harness on the same host. leptos_reactive is the fair apples-to-apples pick: like lazily it is a lazy, pull-based memo system (a memo recomputes only when read while dirty), so this isolates per-node runtime overhead and the lazy-pull viewport property rather than comparing a pull model against an eager push one. (JS signal libraries — Solid, MobX, Preact Signals — are a different runtime and are excluded; the standard js-reactivity-benchmark / cellx harnesses also measure small/medium graphs, not a 100k-node sheet.)

Measured at N = 100_000 (200,000 nodes/library; leptos is far heavier per node, so this size keeps its wall clock feasible — lazily’s own 1M/10M numbers are above):

† lazily’s viewport_recalc is post-v0.22.2 (#lzslotfastpath). Before that refresh fast path it measured 11.52 µs and leptos led ~1.4× (the original row this table shipped with). The v0.22.2 controlled A/B on this case is 13.78 µs → 4.49 µs, −64.1% (p=0.00) (--save-baseline pre_fix, toggling only src/context.rs). leptos_reactive is an unchanged external library so its 8.22 µs is retained from the original same-host run; a fresh same-session re-measure under load gave ~10.5 µs, i.e. lazily leads by ~1.8–2.3× depending on leptos’s run-to-run variance.

Honest read: lazily now leads all four cases — building the sheet (1.5×), computing it cold (3.6×), recomputing the whole sheet after a full invalidation (2.8×), and the cached-read-dominated viewport read (~1.8×) — driven by its sparse arena + lean single-threaded Context versus leptos’s runtime slotmap and subscriber bookkeeping, plus the v0.22.2 refresh_slot clean-cache-hit fast path that removed the per-read dependency-walk tax on the ~998/1000 viewport cells that are cache hits. The fairness evidence is no longer “leptos wins a case” (it did, before v0.22.2, and that historical result is documented in the footnote above) — it is that leptos’s genuine 30 ms cold recalc proves its memos truly recompute (this is not a straw-man comparison), and that lazily’s viewport lead is a recent code improvement, not an inherent property: the pre-v0.22.2 code lost this case. The shared headline is unchanged: the lazy-pull property both exhibit — a one-input edit + bounded-viewport read is microseconds, ~1000× cheaper than a full recalc, independent of total sheet size — neither library recomputes off-viewport formulas. The defensible claim is now “lazily has materially higher throughput than a comparable native-Rust pull-based reactive system across both whole-graph and incremental-viewport workloads,” not a blanket “fastest reactive library.”

Reproduce (gated behind the scale-compare feature so the comparison dependency is never pulled into normal builds / make check):

cargo bench --features scale-compare --bench scale_compare
LAZILY_SCALE_N=250000 cargo bench --features scale-compare --bench scale_compare

Cross-language comparison (lazily-rs / lazily-cpp / lazily-zig)

Head-to-head on the same spreadsheet-shaped workload (N input cells + N formula slots, formula[i] = input[i] + input[i-1]), measured on x86_64 Linux. lazily-rs uses criterion; lazily-cpp uses its std::chrono harness; lazily-zig uses clock_gettime(.MONOTONIC) for the scale bench. Numbers are the current published results from each repo’s BENCHMARKS.md.

Micro-benchmarks (single-threaded Context unless noted)

† lazily-zig 0.17-dev removed std.time.Timer, so its reactive-core micro-bench is counter-based (deterministic work-counts: allocations, edges, recomputes — not wall-clock). The counters confirm the same zero-work steady state (cached reads = 0 allocs / 0 recomputes) but are not directly comparable on a wall-clock axis. See lazily-zig BENCHMARKS.md.

Scale — 1M rows (~2M cells)

Scale — 10M cells (full Google Sheets workbook capacity)

Honest read: lazily-rs’s monomorphized Rc<T> fast path leads the spreadsheet-scale build wall clock (leanest per-node storage) and — after the v0.22.2 #lzslotfastpath refresh fast path — delivers the cheapest viewport reads of the three (4.5 µs @ 1M, 4.1 µs @ 10M, undercutting lazily-zig’s integer-keyed cache at 6.4/6.6 µs). lazily-cpp’s v0.6.0 SmallAny inline value storage (optimization B) + alloc-free batch bookkeeping (E) flipped the cold-recalc lead: lazily-cpp cold full recalc is now ~3× faster than lazily-rs at both 1M (36 vs 106 ms) and 10M (415 vs 518 ms), and its batch_storms now edges out lazily-rs (1.55 vs 3.1 µs). lazily-cpp’s type-erased SmallFn + SmallVec node layout still wins the high-fan-out micro-benchmarks (fan-out 256, set_cell 512, memo equality) by 16–49× over lazily-rs. The shared headline across all three: they back a full-capacity Google Sheets workbook and all exhibit the lazy-pull viewport property — a one-cell edit + bounded-viewport read stays in the microsecond range, independent of sheet size, because off-viewport formulas are left dirty and never recomputed (~2,000–60,000× cheaper than a full recalc across the three runtimes).

Phase 3 Wire-Format Optimizations (#lzperfaudit)

Three spec-ratified wire wins (#lzspecfrontiersuppress, #lzspecbase64, #lzspecintern), measured by benches/wire_optimizations.rs. Run with:

cargo bench --features json-base64 --bench wire_optimizations

#lzspecfrontiersuppress — optional CrdtSync frontier

Omitting the stamp frontier when unchanged cuts wire size and encode/decode cost:

#lzspecbase64 — base64 byte arrays vs JSON-u8 arrays

Under the json-base64 capability flag, Inline/Payload bytes travel as base64 strings instead of JSON integer arrays:

#lzspecintern — batch string-intern table

Deduplicating repeated type_tag strings into a sidecar intern table (256 nodes, 4 distinct tags):

Savings grow with the node-to-tag ratio (more nodes sharing fewer tags).

Revision engine crossover (#lzspecrevisionengine)

The revision (pull) invalidation engine gives O(1) writes (no dependent cone walk) at the cost of O(changed-subpath) reads. Observable values are provably identical to push mode (get_equiv_push, lazily-formal RevisionEngine.lean).

Benchmark: 10 writes to a source cell with N dependent slots (construction + priming included in each measurement). Run with:

cargo bench --bench revision_engine

The write cost scales linearly with fan-out in push (O(N) dirty walk) but is O(1) in revision (revision bump). The construction+priming overhead (same for both) dilutes the pure write-cost gap; workloads with high write:read ratios and large fan-out benefit most.

Multi-Language

lazily is implemented across three languages with shared semantics:

Related

• lazily-zig — Zig implementation with FFI support
• lazily-py — Python implementation with context-as-dict
• Blog post: Lazily — Reactive Primitives Done Right

License

MIT