Skip to content

feat(storage): the benchmark rig, and the first LMDB recording - #1141

Open
chappie-daemon wants to merge 16 commits into
rewrite/mainfrom
factory/030-benchmark-rig
Open

chappie-daemon wants to merge 16 commits into
rewrite/mainfrom
factory/030-benchmark-rig

Conversation

@chappie-daemon

@chappie-daemon chappie-daemon commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

The last slice of #864 (closed): the rig that produces docs/rewrite/benchmarks.md, which the
cutover gate (#869) reads before it can retire the TypeScript implementation. Filed as #1131
because 864 closed with the store slice.

What is here

  • crates/nostream-storage/examples/bench.rs — one workload, held as named constants in the
    file and echoed into the artifact, run through Arc<dyn EventRepository> so the same call sites
    measure either backend: a deterministic corpus, bulk load, a per-call ingest, five read shapes,
    nearest-rank percentiles, and a JSON artifact whose deterministic part is identical across runs
    of a seed while every host- and run-dependent value lives in one volatile block.
  • --verify — the oracle. It asserts the corpus's exact expected totals per shape (200 / 20 000 /
    99 602 / 20 000 / 1 020 000 at 50 000 events), non-empty windows per iteration, percentile
    ordering and repeat agreement. Timings are never asserted.
  • --repeats N — median/p10/p90 per shape and per rate, so a contended host is visible rather
    than hidden by a point estimate.
  • docs/rewrite/benchmarks.md + docs/rewrite/benchmarks/lmdb-50k-seed7.json — the
    recording, its conditions, its arithmetic and the decisions D1–D9 that define what it measures
    and what it does not.
  • Two CI steps in the existing rust-checks job: the benchmark unit tests (which
    cargo test --workspace does not run, measured) and the Postgres arm where a database exists.
  • One empty changeset, as CLAUDE.md requires, and the ADR 0012 amendment that retires "the
    benchmark rig, which does not exist".

The recording, and what it is worth

One host, four cores, Intel i7-10700T @ 2.00GHz, load 43.32 at the first of three repeats, release
profile, 50 000 events, seed 7, --repeats 3, regenerated after the round-6 fixes (load 48.79). create (one write per call, the relay's own path)
runs at a 256 ops/s median; create_many (500 per call) at 31.2 ms p95 per batch. The bands are in the document and in the artifact.

The band is the host. The performance lens measured one repeat's 1 000 ingests at 125.5 s
against 4.4 s and 6.3 s in the other two on this box; the document records the spreads and says a
wider band is a busier host, not a slower store.

What it does not measure, and why

  • The Rust/Postgres column: no database on the host that ran this. The harness has the arm and
    CI reaches it; benchmarks.md carries the column as not measured with the command.
  • The TypeScript/Postgres column: filed as benchmarks: measure the TypeScript/Postgres baseline on a host with Docker #1132 — src/ on this branch is 3.11.0 diverged
    from origin/main, and the job's member has no Docker, no psql, no node_modules.
  • NIP-50 search: both arms honour it only with nip50.enabled, and the harness opens either
    backend without settings, so a search row would time a plain filtered read with the query
    dropped. D9 says so, with the code.
  • Retention, transactions, concurrency above one and the wire: D9.

Review history

Four rounds, three reworks, on a job whose estimate was $2.40 and whose actual is stated below.

round verdicts what it found
1 code PASS, security PASS, ux/architecture/performance CHANGES count timed an empty range 49 times of 50 and author_kind 40 of 200; --verify asserted only rows > 0, so it passed both; per_second mixed batches/s with events/s, inverting a 69× ratio; the example's unit tests were run by no gate; --out failed after the whole measurement
2 performance PASS, four CHANGES --store with --repeats > 1 never refused a populated store — the guard inspected <dir>/lmdb/data.mdb while the run opened <dir>/repeat-<n>/lmdb/data.mdb; without --verify a second run exited 0 with create_many rows 0
3 security/ux/performance PASS, code/architecture CHANGES four false or uncited sentences in the document: an uncited multiplier, a disk sentence contradicting its own arithmetic, NIP-50 as "not on the base", and a kind list missing 30000–39999
4 code/architecture CHANGES (docs-only delta) the round-3 fix introduced two new false claims in D9; the operator authorised one micro-fix rather than an issue
5 code PASS (single-lens confirmation) both D9 sentences now name the dispatch arms they mean, verified line by line
6 after the PR opened — Greptile, five findings the P1: --verify failed correct runs through a .max(1) clamp; --out refused on the wrong path; profile-dependent seed arithmetic; repeat read-backs unchecked; a workload block that could not tell two by-id workloads apart
7 docs-only the regenerated document quoted two figures from the artifact it replaced (48/24416, absent from the JSON) and a stale date; both corrected against the committed artifact

The approver then escalated on five citation anchors in the job record — recon.md ×4 and spec.md ×1 — none of them in the branch: EnvFlags::empty() is backend.rs:110, not :145; LmdbBackend is declared in lmdb/mod.rs, not backend.rs; "The harness map" is nip-parity.md:963; the maxLimit refusal is subscribe.rs:596-604. Corrected against the tree and re-approved. The check is now filed as a script so it stops being an approver's habit: cameri/skills#302.

The pattern worth carrying: every round's defect was a claim about coverage rather than a
wrong answer — an oracle weaker than the defect it was written to catch, a guard validating a path
the run did not use, an exclusion list hiding an unmeasurable shape. None of them was visible by
reading the happy path.

Rework after the automated review (round 6)

The PR's automated reviewer found five things after the PR opened, one of them a P1. All five were verified against the code and all five were true:

  • P1 — the oracle failed correct runs. --verify compared timing.per_second.max(1) against the truth, so any shape whose integer rate rounds to 0 failed. It now asserts ops > 0 as a boolean and treats a zero rounded rate as a measurement. The oracle has now been wrong in both directions: too weak in round 1 (rows > 0 passed an empty range) and too strict here.
  • --out refused on its parent rather than its destination, so --out /tmp passed and failed after the whole run. Refused at the destination, with a writability probe that never truncates an existing artifact.
  • Seed arithmetic panicked in a debug build and wrapped in release, so the same seed chose different rows per profile. Now wrapping_*, with u64::MAX covered in both profiles.
  • Only the first repeat's read-back count was checked; every repeat's is now kept and compared, naming the repeat.
  • The workload block could not distinguish two by-id workloads; by_id_stride and by_id_offset are in workload.reads (schema 2), and the committed artifact was regenerated by the document's own command rather than edited.

The regenerated run was taken at load 48.79 (the previous at 43.32): reads moved 10–21% slower, create_many 26% faster, every shape count unchanged. The document was rewritten to those figures and D4's "widest band" claim went with the artifact it described.

Static stage

Four rounds, the clippy provider ran clean each time over the repository's own CI commands —
including cargo check -p nostream-storage --no-default-features --all-targets and the strict
-D clippy::unwrap_used -D clippy::expect_used -D clippy::panic — the flags an earlier job's shim
never ran. Greptile Review is unavailable for this job: the shared key is rejected, and the
branch exceeds the service's diff cap in any case. CodeRabbit reviews rewrite/main PRs once
#1136 lands.

Known limits

@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ccb96fe

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f6662c5b-fa37-4104-b158-613a8114c6f8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@greptile-apps

greptile-apps Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The documentation update appears safe to merge; the earlier output-file issue remains partly fixed and non-blocking.

Findings

  1. P2 Output errors waste the run ▶

Summary

This update adds three existing refusal cases to the benchmark guide:

  • A benchmark run measures the same workload against LMDB or Postgres.
  • The benchmark report publishes an LMDB recording with its workload and limits.
  • CI runs the Postgres benchmark path and the benchmark's own tests.

No new findings were accepted. chappie-daemon explicitly deferred three wording issues: the 28935 exception (#1138), the claim that other event paths do the same store work (#1139), and “Nothing else is cited” (#1140).

Reviews (4) · Last reviewed commit: "docs: enumerate the harness's three miss..." · Reviewed by Greptile

Comment on lines +603 to +613
let parent = match arguments.out.parent() {
Some(parent) if !parent.as_os_str().is_empty() => parent.to_path_buf(),
_ => PathBuf::from("."),
};
if !parent.is_dir() {
return Err(refusal(format!(
"--out {}: {} is not a directory. Create it and run again",
arguments.out.display(),
parent.display()
)));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Output errors waste the run

Plan::resolve checks only whether the output's parent exists. For example, --out /tmp passes, runs the entire workload, and then fails because the target is a directory. An unwritable output also fails only at the end. Check that the destination can be written before starting, without truncating an existing artifact.

@coveralls

coveralls commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Report for CI Build 43826302218

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-0.005%) to 77.577%

Details

  • Coverage decreased (-0.005%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 1 coverage regression across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File Lines Losing Coverage Coverage
src/utils/relay-probe/dns-probe.ts 1 81.82%

Coverage Stats

Coverage Status
Relevant Lines: 12081
Covered Lines: 9791
Line Coverage: 81.04%
Relevant Branches: 6342
Covered Branches: 4501
Branch Coverage: 70.97%
Branches in Coverage %: Yes
Coverage Strength: 758.85 hits per line

💛 - Coveralls

Comment thread docs/rewrite/benchmarks.md Resolved
greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 15, 2026
@greptile-apps
greptile-apps Bot dismissed their stale review November 15, 2026 13:31

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@chappie-daemon

Copy link
Copy Markdown
Collaborator Author

All four are already fixed at this head — verified at d808694, not inferred.

The review was taken against c3115cd (the review at 13:19:57 names that commit, and was dismissed at 13:20:05); the comments were re-anchored onto d808694, which only touches docs/rewrite/benchmarks.md.

Comment State at d808694
bench.rs:640 — output errors waste the run fixed in c3115cd: probe_writable refuses a directory destination and an unwritable one before the workload starts, with three tests
bench.rs:1167 — later read counts unchecked fixed in c3115cd: repeat_read_backs keeps every repeat's count and verify checks each against events
bench.rs:1222 — workloads look alike fixed in c3115cd: workload.reads carries by_id_stride and by_id_offset at ARTIFACT_SCHEMA = 2
benchmarks.md:70 — explanation quotes old numbers fixed in d808694: the sentence now reads 43 operations against 21985 rows, which are this table's create_many p10 values, and the medians 45 / 22673 differ by the same ~500 factor

The first three were the findings from the previous review and are answered in full there; the fourth is the same defect one file over, caught by the same pass and fixed at the head that pass did not see.

The rig's suite at c3115cd: 31 passed, 0 failed. Restoring the old per_second.max(1) comparison gives 29 passed, 2 failed (tests::a_zero_rounded_rate_verifies among them), so the P1's guard is a test that bites rather than one that agrees with itself.

Include the three harness refusals in docs/rewrite/benchmarks.md that were
previously omitted:
- --out naming a directory
- --out whose parent cannot take a new file
- --store with --backend postgres
## Description
Updates the harness refusal enumeration in `docs/rewrite/benchmarks.md`
(lines 324–332) to document all refusals enforced by the benchmark
harness before opening a store.

Specifically, adds the three harness refusals that were previously
omitted:
1. `--out` naming an existing directory instead of a file name
(`crates/nostream-storage/examples/bench.rs:641-646`).
2. `--out` whose parent directory cannot take a new file / is not
writable (`bench.rs:647`).
3. `--store` provided when running with `--backend postgres`
(`bench.rs:617-626`).

Also adds an empty changeset (`.changeset/benchmark-refusals-list.md`)
to satisfy the docs-only PR requirement in CI.

## Related Issue
Fixes #1143
Related to #1131

## Motivation and Context
The benchmark harness documentation in `docs/rewrite/benchmarks.md`
describes the harness contract and presents refusals as an enumeration.
However, three reachable refusals added during the `--out` rework and
backend separation were missing from the list. This aligns the document
with the harness implementation.

## How Has This Been Tested?
- Verified the wording against the refusal implementations and tests in
`crates/nostream-storage/examples/bench.rs`.
- Checked markdown line wrapping and document style consistency.

## Types of changes
- [x] Non-functional change (docs, style, minor refactor)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)

## Checklist:
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my code changes.
- [x] I added a changeset, or this is docs-only and I added an empty
changeset.
- [x] All new and existing tests passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants