Skip to content

ci: drop free-disk, fold the benchmark smoke into Test (node), build beacon-mainnet once - #801

Open
MegaRedHand wants to merge 12 commits into
mainfrom
ci/faster-ci-followups
Open

MegaRedHand wants to merge 12 commits into
mainfrom
ci/faster-ci-followups

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

Follows #800, now merged. That PR ported what ethrex already does: main-only caches, the Docker cache in ghcr, RocksDB behind a feature, and skipping docs-only changes. This one adds the three remaining improvements from the same CI-time analysis that ethrex doesn't have.

What Changed

  • No free-disk step in CI (ci.yml).
    • It deleted the runner's preinstalled toolchains and took 1.5–3.5 min in every job (measured).
    • Hosted runners now have a 145 GB root disk. The daily run, which never ran it, ended each test half at 65–70 GB used.
    • The test halves and beacon jobs still print df -h / after their build, so a runner change that leaves too little room would show there.
    • kurtosis_assertoor.yml, which pulls Docker images rather than building Rust, keeps the action. Out of scope here.
  • The benchmark smoke check is an integration test, bin/ethlambda/tests/benchmark_smoke.rs, which Test (node) runs through make test-node. The Benchmark smoke job is gone.
    • That job compiled the whole node a second time, 18–20 min cold, for a run that takes seconds, and held a 1.8 GB cache entry.
    • The test runs the same command line (benchmark synthetic --mock-crypto, 4 validators, 4 warm-up slots, 3 iterations, --format json) and asserts what the old jq -e did: schema_version == 1 and 3 samples.
    • It runs the real binary in its own process (CARGO_BIN_EXE_ethlambda). The benchmark's phase timers read the global metrics registry, which no other test in the process can then touch.
    • Cost: cargo now also builds and links the binary in Test (node). In the smoke job's log, the bin crate was the last unit, about 1.5 min. Building it with the test features is cheap: it reuses the test build's dependency graph, which cargo run could not.
    • Benchmark smoke was not a required check, so no settings change is needed.
    • Lint gains cargo check --locked -p ethlambda. The old job's cargo run was the only PR build of the binary with just the features it ships with. Every other build of it (Lint's workspace check, Test (node)) pulls in the dev-dependencies' features too, tokio's test-util among them, so code that compiled only because of one would pass every PR job and break cargo build --release on main. Its cold cost on CI is not measured yet.
  • make test-beacon-mainnet is one cargo invocation, not three (state-transition's --lib, the spec suite, blockchain's --lib).
    • Cargo resolves features per invocation, over the packages it selects. The separate cargo test -p ethlambda-blockchain therefore rebuilt most of the shared dependency graph, from libc up, for 335 tests that run in 5 s. That rebuild took 11m54s cold before ci: port ethrex's cache, change-detection and RocksDB-feature practices #800 and 2m00s after it, now that RocksDB is gone.
    • The two packages are selected together, with package-qualified features. --lib rides in the same invocation as the spec suite, under BEACON_SPEC_PRESET=mainnet.
    • make test-beacon-minimal selects the same two packages and features, though it runs only the spec binary. Since feat(beacon): select the preset at runtime from PRESET_BASE #851 the two beacon jobs share one beacon-spec cache entry, and only the mainnet job saves it. Selecting ethlambda-state-transition alone resolves libc, bytes, syn, tracing, smallvec and the crate itself with other features (cargo --unit-graph), so the minimal job would rebuild most of the graph on every run. With both packages selected, every unit the minimal job builds is also in the mainnet build.
    • Selecting both also unifies ethlambda-blockchain's dev-dependency features into the spec suite's build. The only ethlambda feature among them is ethlambda-state-transition/test-utils, which only makes test helpers public. The Makefile comment says a feature that changes what the state transition or the crypto does must never be enabled from there.
  • Comments and docs that described the old disk ceiling or the old job (Makefile, CLAUDE.md, docs/benchmarking.md, run-fixture-tests, ci.yml, daily_ci.yml) are updated.
  • Nits from the second review round on ci: port ethrex's cache, change-detection and RocksDB-feature practices #800, moved here so ci: port ethrex's cache, change-detection and RocksDB-feature practices #800's CI and approval stay on the commit that was reviewed:
    • New make inert-paths-check. It fails if an include_str! or include_bytes! names a path the docs-only skip treats as unread (Markdown, docs/, theme/, book.toml, .claude/, LICENSE, .gitignore). make lint runs it, and so does CI's Lint job.
      • It scans every tracked .rs file and reads each macro line plus the two after it, which catches rustfmt's split form. A path assembled further away (concat!) or opened at run time still gets past it; CLAUDE.md says so.
      • It fails, rather than passing an empty scan, if git lists no .rs file (not a checkout, or git refusing the directory). Only source lines are matched (grep -h), not the file names grep prefixes them with; a second scan adds names and line numbers back to print the hits.
      • It passes on the current tree, and fails on planted one-line and split-line violations: checked locally with BSD tools under macOS sh and dash; CI's Lint job is the GNU run.
    • run-beacon-spec-tests validates its preset. Anything but mainnet or minimal now fails up front with a clear error, instead of as an unknown make target. The preset reaches make through the environment.
    • Rustdoc note moved. The note on why RocksDBBackend isn't an intra-doc link no longer renders in the public docs; it's a plain // comment now.

Left alone, and why

  • Test minimal preset. I meant to fold it into the beacon and lint jobs. Two reasons not to:

    • It is a required check, so removing it needs a branch-protection change.
    • It is the only job that runs the --lib tests under the minimal preset. Since feat(beacon): select the preset at runtime from PRESET_BASE #851 the minimal beacon job runs the spec binary alone, which selects its preset at runtime. A unit-test binary never selects one, so preset-minimal (minimal as the default) is still how those tests get minimal. They also run in the dev profile, where overflow checks are on, unlike release-fast.

    Since ci: port ethrex's cache, change-detection and RocksDB-feature practices #800 it takes 4–5 min.

  • cargo-hakari, larger runners, and a lighter merge queue. Each is its own decision, not part of this PR.

Expected effect

These are estimates from #800's cold runs and the per-job minimums of the last 40 runs. Nothing here is measured yet.

  • Runner-minutes per run: down about 30 cold. That's the benchmark job (18–20 min), plus about 2 min of free-disk in each of the six other jobs that ran it, minus the ~1.5 min binary build Test (node) gains and Lint's new binary check (about a minute cold, estimated from local timings).
  • Wall-clock, cold: about the same, ~21–22 min. Test (node) is still the slowest job: it loses free-disk but gains the binary build.
  • Wall-clock, warm: about 2 min less per job from free-disk. Test (node) nets roughly +0, and Beacon spec tests (mainnet) drops by 2 more for the separate blockchain build.
  • Cache budget: one fewer Rust entry (−1.8 GB), and the shared beacon-spec entry holds one dependency graph instead of two. Both make it likelier that main's entries fit in 10 GB and stay warm, which is where ci: port ethrex's cache, change-detection and RocksDB-feature practices #800's larger gain comes from.
  • Until this merges, the beacon jobs on this PR restore main's beacon-spec entry, which was built with the old feature resolution, so they run close to cold. That is expected. The gain shows once main saves the entry again.

Tests Added / Run

  • New: bin/ethlambda/tests/benchmark_smoke.rs, run locally with cargo test --profile release-fast -p ethlambda --test benchmark_smoke.
  • The combined beacon-mainnet command with --no-run builds exactly three test executables: blockchain's lib tests, state-transition's lib tests and beacon_spec_tests. Blockchain's lib reports 335 tests, with 0 ignored under the feature, the same as the separate command's run in CI.
  • After merging feat(beacon): select the preset at runtime from PRESET_BASE #851, cargo --unit-graph on the two merged recipes: mainnet's roots are those same three test targets, minimal's is beacon_spec_tests alone, and every minimal unit is also in the mainnet graph (state-transition alone would have added 9 units with other features). Neither recipe has been run locally; the two beacon jobs are their first run.
  • cargo check --locked -p ethlambda (Lint's new step) is clean on the merged tree.
  • cargo fmt --all -- --check, workspace clippy at -D warnings (which covers the new test), actionlint 1.7.12 with pr_lint_gha.yaml's flags, and gh actions-lock --verify: all clean.

Related Issues / PRs

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran make lint (clippy with -D warnings) — clean
  • Ran make test — only the new integration test locally; CI runs the rest

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll review the PR against the surrounding Makefile, CI, and test code so comments are tied to real behavior, not just the diff. I need the beacon test target, inert-path list, and how the smoke test is wired.The beacon-test invocation and the inert-path grep look like the places a silent skip or a false pass could hide. I'll read those definitions next.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The changes look solid and well-reasoned:

  • Moving benchmark-smoke into bin/ethlambda/tests/benchmark_smoke.rs via CARGO_BIN_EXE_ethlambda avoids redundant compilation and cache overhead while preserving process isolation for the global metrics registry.
  • Merging the ethlambda-blockchain test run into the test-beacon-mainnet invocation with package-qualified features prevents Cargo from resolving a separate feature graph and rebuilding shared dependencies.
  • inert-paths-check and the preset validation in run-beacon-spec-tests are clean safeguards.

LGTM!


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 801: drop free-disk, fold the benchmark smoke into test-node, add inert-paths-check

The changes are well reasoned and well commented. I found no blocking issues, only one merge-time hazard to check. No consensus-critical code is touched; the only Rust change outside the new test is a comment move in crates/storage/src/backend/mod.rs, which is fine.

Things to check before merge

  1. Removed Benchmark smoke job vs. required checks (ci.yml, the deleted benchmark-smoke job).

    • If "Benchmark smoke" is a required status check in branch protection, every PR will wait on it forever after this lands.
    • The same applies to any aggregate "all green" job whose needs: lists benchmark-smoke.
    • The diff shows neither, so please confirm the repo settings are updated in the same change.
  2. BEACON_TEST gains a second package (Makefile, test-beacon-mainnet).

    • --lib and --test beacon_spec_tests now apply to both selected packages.
    • Cargo tolerates a named target existing in only one of them, so this should work.
    • However, anything else in the (truncated) BEACON_TEST definition now also applies to ethlambda-blockchain. That includes a trailing test-name filter or -- <args>, which would silently filter or skip the blockchain lib tests.
    • Please confirm the blockchain test count in the mainnet job log matches what the old separate invocation ran.
  3. serde_json in the integration test (bin/ethlambda/tests/benchmark_smoke.rs:43).

    • Integration tests can use the package's normal dependencies, so this works only if serde_json is already a [dependencies] or [dev-dependencies] entry of bin/ethlambda.
    • No Cargo.toml change is in the diff. Presumably the JSON report already pulls it in, but worth a glance.

Minor suggestions

  1. inert-paths-check misses the most likely multi-line case (Makefile, the grep -rnE ... 'include_(str|bytes)!' line).

    • The comment cites concat!. The more realistic miss is rustfmt splitting a long call:
      #![doc = include_str!(
          "../../../docs/some-long-path.md"
      )]
      The first grep matches the macro line, but the path is on the next line, so the second grep never sees it.
    • Cheap fixes are either of:
      • grep -A2 on the first stage;
      • rg -U 'include_(str|bytes)!\(\s*[^)]*(\.md"|docs/|...)'.
  2. inert-paths-check pattern breadth (INERT_PATH_PATTERN).

    • The pattern is unanchored, so docs/, LICENSE and theme/ match anywhere in the path, e.g. include_bytes!("../fixtures/docs/x.ssz").
    • A false positive fails loudly rather than letting something slip through, so this is acceptable. Mention it in the comment so the next person doesn't "fix" it the wrong way.
  3. grep -r on a missing directory. If tooling/ is ever removed or renamed, grep -r ... tooling prints an error to stderr on every make lint.

    • The || true keeps the check passing, but the noise is confusing.
    • Consider adding 2>/dev/null, or derive the directory list.
  4. Inconsistent input handling (run-fixture-tests/action.yml:121).

    • run-beacon-spec-tests now passes the preset through env, but run-fixture-tests still splices make ${{ inputs.make-target }} into the script.
    • The inputs are workflow-controlled, so this is not exploitable. Still, apply the same env: pattern there for consistency.
  5. No timeout on the smoke test (benchmark_smoke.rs:17).

    • Command::output() blocks indefinitely. If the benchmark ever deadlocks, the Test (node) job hangs until the Actions job timeout.
    • Optional: spawn the process, wait with a deadline, and kill it on expiry.
  6. No disk headroom margin (ci.yml, removal of free-disk).

    • The rationale (145 GB root disk, 65–70 GB used) is documented, and the df -h / steps remain as tripwires.
    • Note that the jobs use vars.ETHLAMBDA_RUNNER. If that variable ever points at a smaller self-hosted or larger-runner image, nothing frees space anymore. The failure mode is the misleading Bus error the comment describes.
    • The comment already covers this; just flagging that the guard is observational only.

Looks good

  • The preset allow-list in run-beacon-spec-tests, with the value read via env rather than interpolated.
  • Moving the benchmark smoke into an integration test. Using CARGO_BIN_EXE_* avoids the feature-resolution rebuild, and running in a separate process keeps the global metrics registry isolated.
  • The assertions (report["schema_version"], 1 via PartialEq<i32> for Value, the samples length) match the old jq -e contract exactly.
  • Merging the blockchain lib tests into the same cargo invocation to avoid a second feature resolution, with package-qualified features.
  • The docs and CLAUDE.md updates are accurate and state the check's limits honestly.

Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The CI changes and benchmark smoke test look correct; no consensus, cryptography, or SSZ implementation changes are present.

make inert-paths-check passed. Full Rust tests could not run because the pinned toolchain requires an update in a read-only location.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found.

  • .github/actions/run-beacon-spec-tests/action.yml: preset validation and environment-based shell usage prevent malformed targets and script injection.
  • Makefile: the consolidated mainnet command preserves both packages’ features while avoiding the duplicate dependency build.
  • bin/ethlambda/tests/benchmark_smoke.rs: correctly uses CARGO_BIN_EXE_ethlambda, checks the exit status, and validates both JSON schema and sample count.

One residual limitation: the inert-paths-check recipe can also miss a conventionally formatted multiline invocation such as include_str!(\n "../README.md"\n), since it only examines lines containing the macro name. This is consistent with the documented line-oriented, best-effort scope, but worth covering with a token-aware check if stronger enforcement is desired.

No consensus-layer code is changed.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The PR is mostly CI/Makefile/docs changes; no consensus-layer code paths are touched, so fork choice, attestations, SSZ, and XMSS are unaffected.

Feedback:

  1. Makefile test-beacon-mainnet recipe (around new line 88):
    $(BEACON_TEST) -p ethlambda-blockchain --features ... combines two packages into one Cargo invocation, but BEACON_TEST already contains --lib --test beacon_spec_tests. Cargo target filters like --lib and --test apply across all selected packages, so verify that ethlambda-blockchain's lib tests still run and that beacon_spec_tests does not cause Cargo to skip or error for a package that likely lacks that integration-test target. If in doubt, keep the two package invocations separate or make target selection package-specific.

  2. Makefile inert-paths-check (around lines 28–31):
    $(INERT_PATH_PATTERN) is expanded unquoted and relies on embedded shell quotes inside the Make variable. It works today but is fragile; a future edit can easily break the shell quoting. It also only scans crates bin tooling, so any root-level Rust source such as build.rs, src/, or root-level tests/ would bypass the check. Consider scanning all workspace Rust files from the repo root while excluding target/, and use a plainly quoted shell variable.

  3. bin/ethlambda/tests/benchmark_smoke.rs:
    The test uses serde_json, but the diff does not add a Cargo.toml change for bin/ethlambda. Verify serde_json is already a normal or dev dependency; otherwise this integration test will not compile. The test itself is reasonable and preserves the old CI smoke contract.

  4. .github/workflows/ci.yml:
    Removing all free-disk steps is fine for the current hosted runner sizes, but if vars.ETHLAMBDA_RUNNER can point to custom self-hosted runners with smaller disks, this may reintroduce disk exhaustion. The retained df -h / reporting is a good mitigation, but it is worth confirming all self-hosted runners have enough space.

The preset validation added in .github/actions/run-beacon-spec-tests/action.yml is a good hardening step.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Review follow-up

Fixed in beef284

  • inert-paths-check misses rustfmt's split form (Claude, Kimi). It now reads each include_str!/include_bytes! line plus the two after it, so include_str!( with the path on the next line is caught.
    • Checked on a planted file: it flags the split docs/....md path, docs/assets/x.png and README.md, and leaves a .txt include alone.
    • It still passes on the tree.
  • The scan misses files outside crates/, bin/ and tooling/ (DeepSeek, Claude). It now scans every tracked .rs file (git ls-files): 393 files, including 2 under shadow/ the old list skipped. That also ends the error output for a missing directory.
    • The pattern is now a plain shell variable instead of a Make variable with embedded quotes.
  • The pattern is unanchored (Claude). That's on purpose, and the comment now says so: a false positive fails loudly, so widen the list rather than anchor it.
  • run-fixture-tests splices its input into the script (Claude). The make target now reaches the script through env, like the beacon preset.
  • The smoke test has no timeout (Claude). It kills the run and fails after 120 s, instead of holding Test (node) until the job's 6-hour default. Output goes to files, so the child can't block on a full pipe while the test polls it. Locally it passes in 0.88 s.

Checked, no change needed

  • Benchmark smoke might be a required check (Claude). It isn't: branch protection requires Lint, Test (consensus), Test (node), Beacon spec tests (mainnet), Beacon spec tests (minimal) and Test minimal preset. No job needs: it.
  • The second package in BEACON_TEST might skip or filter tests (Claude, DeepSeek). It doesn't. BEACON_TEST carries no test filter or trailing args, and this PR's CI ran blockchain 335, state-transition 653 and spec tests 7493, the same counts as the separate commands.
  • serde_json may not be a dependency (Claude, DeepSeek). It already is, in bin/ethlambda's [dependencies].
  • No disk margin if ETHLAMBDA_RUNNER changes (Claude, DeepSeek). That's documented in ci.yml, and the df -h / steps stay as the tripwire. Every job in these runs reported the 145 GB disk.

Grok's review posted only its plan, no findings.

Base automatically changed from ci/ethrex-ci-practices to main November 18, 2026 16:38
…beacon-mainnet once

- No CI job runs the free-disk action any more. It took 1.5-3.5 min per
  job, and hosted runners now have a 145 GB root disk: the daily run,
  which never freed anything, ended each test half at 65-70 GB used. The
  df -h reports after each build stay. kurtosis_assertoor.yml keeps it.
- The benchmark smoke check is an integration test,
  bin/ethlambda/tests/benchmark_smoke.rs, run by Test (node). It drives
  the built binary in its own process with the old job's arguments and
  asserts the same report contract. The Benchmark smoke job, which
  compiled the whole node a second time, and its cache entry are gone.
- make test-beacon-mainnet runs the spec suite and the chain actor's
  fixture tests in one cargo invocation. Two invocations resolved
  features differently and rebuilt most of the shared dependency graph
  for 335 tests that run in 5 s.
- Comments and docs that described the old disk ceiling or the removed
  job are updated.
…, tidy a doc note

Nits from the review round on the base PR, moved here so its CI and
approval stay on the commit that was reviewed:

- make inert-paths-check fails if an include_str! or include_bytes!
  names a path the docs-only skip treats as unread (Markdown, docs/,
  theme/, book.toml, .claude/, LICENSE, .gitignore). make lint runs it
  and so does CI's Lint job. It reads line by line, so a path assembled
  across lines or opened at run time gets past it; CLAUDE.md says so.
- run-beacon-spec-tests rejects a preset other than mainnet or minimal
  up front, and passes it to make through the environment.
- The note on why RocksDBBackend is not an intra-doc link moves out of
  the rendered module docs into a plain comment.
- inert-paths-check scans every tracked .rs file (git ls-files) rather
  than crates/, bin/ and tooling/, which missed shadow/ and printed an
  error for a missing directory. It reads each include_str!/include_bytes!
  line plus the two after it, which catches rustfmt's split form
  (`include_str!(` with the path on the next line). The pattern is a
  plain shell variable, and the comment says why it is unanchored.
- run-fixture-tests passes its make target through the environment, the
  same as the beacon action's preset.
- The benchmark smoke test kills the run and fails after 120 s instead of
  blocking until the job's own timeout, with output going to files so the
  child cannot block on a full pipe while the test polls it.
- Lint now checks `-p ethlambda` on its own. Every other build of the
  binary in CI (Lint's workspace check, Test (node)) unifies the
  dev-dependencies' features in, tokio's `test-util` among them, so the
  removed Benchmark smoke job's `cargo run` was the only PR build of the
  feature set `cargo build --release` ships. About a minute cold, 57
  units, ~64 MB in the lint cache.
- inert-paths-check exits 1 when git lists no `.rs` file (not a
  checkout, or git refusing the directory). It used to read that empty
  scan as a pass.
- Its hits are printed with printf. echo in dash and in macOS sh expands
  backslash escapes, so a `\c` in a hit cut the list short.
- Comments that still said a cold build outgrows the runner's disk, or
  that the test halves are split for disk, now give the current reason.
  CLAUDE.md now says the check does catch rustfmt's split form.
@MegaRedHand
MegaRedHand force-pushed the ci/faster-ci-followups branch from 347b71c to 23787d6 Compare November 18, 2026 16:43
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The CI hardening is solid: preset allowlisting, env-based make invocation, and the shipped-features cargo check close real gaps. A few correctness issues remain, mainly around the new docs-only scanner and the beacon test invocation.

1. inert-paths-check false-fails on context lines. Makefile around the new target (git ls-files -z '*.rs' | xargs -0 grep -HnE -A2 ... | grep -E "$$pattern"). grep -A2 emits context lines that do not contain include_str!/include_bytes!. The second grep matches those too, so a nearby comment, string, or docs/ mention fails the build. Restrict the second filter to lines that still contain the macro (or use a line-window parser that only tests the macro invocation).

2. The “no tracked .rs files” guard does not work. Same recipe: git ls-files --error-unmatch '*.rs'. --error-unmatch takes pathspecs of existing paths, not a glob of tracked files, so this errors even in a normal checkout. Drop it and fail when git ls-files -z '*.rs' produces no output, which is what the comment describes.

3. Pattern and the docs-only list can drift, and the pattern is broader than the comment. The action comment says to keep the scanner in step with the inert list, but the list still lives only in the workflow and the regex is a hand copy. Also \.md" matches any .md string, not only repo-root Markdown, and docs/ is unanchored (fixtures/docs/...). Either generate one list from the other, or document the extra matches as intentional and add a fixture that would have caught Item 1.

4. test-beacon-mainnet feature unification looks incomplete. Makefile test-beacon-mainnet now runs one invocation with -p ethlambda-blockchain --features ethlambda-state-transition/beacon-spec-tests,ethlambda-blockchain/beacon-spec-tests. BEACON_TEST already passes -p ethlambda-state-transition. Package-qualified features are not enabled on packages outside the selected set, so the state-transition feature may not actually turn on, and the old second cargo test -p ethlambda-blockchain is gone. Confirm both --lib suites still run; if not, select both packages explicitly (-p ethlambda-state-transition -p ethlambda-blockchain) and keep the qualified features. test-beacon-minimal was not given the same treatment, so preset jobs can still diverge.

5. Benchmark smoke is weaker than the old CI check, and it is easy to miss. bin/ethlambda/tests/benchmark_smoke.rs only asserts schema_version == 1 and samples.len() == 3. The old jq -e expression required a real array ((.samples | length == 3) is false for a missing/null field). as_array().expect panics instead of failing that assertion, and nothing checks that samples are objects. Also, make test-node uses cargo test --workspace --exclude …, which does not build integration-test binaries unless --tests / --all-targets is set. If TEST has no --tests, this file never runs in Test (node) and the deleted benchmark-smoke job is a coverage regression. Add --tests (or an explicit binary-test invocation) and assert the samples array the way jq did.

6. Timeout reap is fine; the poll is not a problem. The 120s kill plus try_wait loop is appropriate so a hang cannot sit until the job timeout. No change needed there. tempfile must already be a dev-dependency of bin/ethlambda; worth a quick lockfile check since this is a new integration test.

Removing free-disk is consistent with the new disk notes, provided kurtosis_assertoor.yml still has the action it comments on. The RocksDB doc-comment close (//! to //) is correct so cargo doc does not treat the feature-gated link note as crate docs.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks good to merge.

Key improvements verified:

  • CI efficiency & feature coverage: Moving the benchmark smoke check into bin/ethlambda/tests/benchmark_smoke.rs avoids a redundant workspace build and cache entry, while adding cargo check --locked -p ethlambda to the lint job preserves coverage for compiling the binary without dev-dependencies feature unification.
  • Single-invocation Beacon mainnet tests: Merging ethlambda-blockchain into the BEACON_TEST invocation in Makefile (test-beacon-mainnet) prevents Cargo from re-resolving and rebuilding the dependency graph with a different feature set.
  • Docs-only skip enforcement: inert-paths-check in Makefile and CI provides a safeguard against include_str!/include_bytes! referencing files ignored by detect-changes.
  • Action input hardening: Passing inputs.preset and inputs.make-target via environment variables in .github/actions/run-beacon-spec-tests/action.yml and .github/actions/run-fixture-tests/action.yml follows GitHub Actions security best practices.

Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The CI changes and benchmark smoke test look sound; consensus, signature verification, and SSZ logic are unchanged.

make inert-paths-check passes. Rust test compilation could not complete because the pinned libssz Git dependency is unavailable offline.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 801: CI and tooling changes

This PR changes no consensus logic. Most of it is CI hygiene, and the direction is sound: script-injection hardening, dropping a redundant job, and a cheaper benchmark smoke test. A few items need checking before merge.

Correctness / risk

1. Makefile test-beacon-mainnet: merging the two invocations also merges dev-dependency features.

The new rule is:

$(BEACON_TEST) -p ethlambda-blockchain --features ethlambda-state-transition/beacon-spec-tests,ethlambda-blockchain/beacon-spec-tests

Merging the invocations is the point, but it has a consensus-relevant side effect. With resolver 2, cargo test unifies the dev-dependency features of every selected package.

  • If ethlambda-blockchain's [dev-dependencies] turn on anything in ethlambda-state-transition or the crypto crates (a mock-crypto, test-utils or skip-verification style feature), the beacon spec tests will now run with it.
  • That could silently weaken signature or state-transition coverage while the suite still reports green.
  • Please confirm no such feature exists. Better still, add a comment here stating the invariant so a future dev-dependency addition doesn't break it unnoticed.

The multi-package --test beacon_spec_tests selection itself is fine. Cargo only errors if no selected package has that target.

2. Removing free-disk relies on runner disk size with only after-the-fact diagnostics (ci.yml ~L67-80).

  • df -h / prints only after the build, and only in the test and beacon jobs. lint and presets print nothing.
  • If a runner rollout, a fork, or an override of vars.ETHLAMBDA_RUNNER lands on a smaller disk, failures return as the hard-to-diagnose signal 7 [Bus error].
  • Consider a cheap pre-build guard: fail or warn when df --output=avail / is below some threshold, or run free-disk only in that case. That turns a cryptic linker crash into a clear message.

3. bin/ethlambda/tests/benchmark_smoke.rs: check the dev-dependencies.

  • The test uses tempfile and serde_json, but the diff has no Cargo.toml change.
  • serde_json is probably a normal dependency, which is fine.
  • tempfile must be in bin/ethlambda's [dev-dependencies] (and already in Cargo.lock, given --locked). If it is already there, ignore this.

inert-paths-check (Makefile L41-49)

4. The filename is part of the grepped text.

  • grep -H prefixes each line with the path, so any .rs file under a directory named theme/ or docs/, or with LICENSE in its path, fails as soon as it contains an include_*!.
  • The same applies to the -A2 context lines, which can contain unrelated code or comments.
  • You accept loud false positives, but the path-prefix case is easy to avoid. Strip the file:N: / file-N- prefix before matching, e.g. sed -E 's/^[^:]*[:-][0-9]+[:-]//', or match on grep -h output and re-grep for locations.

5. Grep errors are swallowed as a pass.

  • || true also hides grep exit code 2.
  • For example, git ls-files lists a tracked file that has been deleted in the worktree, grep fails to open it, and the check passes.
  • This is minor, but git ls-files -z --cached --exclude-standard doesn't avoid it either. Consider checking PIPESTATUS-like behavior, or using git grep -n -A2 -E 'include_(str|bytes)!' -- '*.rs'. git grep reads the index, avoids xargs, and keeps the error semantics simpler.

6. Unsupported cases.

build.rs scripts that read files, or cargo:rerun-if-changed pointing at inert paths, are another way a build depends on these files. CLAUDE.md covers run-time reads, but it could name build scripts explicitly.

Good

  • Reading inputs.preset and inputs.make-target through env: and validating the preset closes the expression-injection vector.
  • cargo check --locked -p ethlambda correctly restores the "shipped features" coverage that the removed cargo run job used to provide implicitly. Resolver 2 leaves dev-dependency features out for a lib/bin-only check.
  • The smoke test design holds up: files instead of pipes avoid a deadlock, there is a deadline with kill/wait, and process isolation keeps the metrics registry separate. assert_eq!(report["schema_version"], 1) behaves like the old jq check for integer values.
  • The backend/mod.rs change, moving the explanation out of the rendered rustdoc, is a good cleanup.

Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The benchmark test has sound timeout, process cleanup, and output handling, and the combined beacon-test Cargo invocation preserves both packages’ tests while avoiding duplicate dependency builds. No consensus or state-transition logic is changed.

Two non-blocking hardening suggestions:

  • Makefile, inert-paths-check recipe: The scan pipeline ends in || true, so genuine failures from git, xargs, or either grep are indistinguishable from “no matches” and can silently pass the check. This partially undermines the deliberate fail-closed handling for an empty repository. Consider staging the file list/grep output so status 1 can be accepted specifically as “no match,” while other statuses fail. Adding -- before the filenames passed to grep would also handle dash-leading filenames correctly.

  • .github/actions/run-fixture-tests/action.yml, “Run tests” step: make "$MAKE_TARGET" correctly prevents shell injection, but a value beginning with - is still interpreted as a Make option. Current callers use fixed, trusted targets, so this is low risk. For defense in depth, validate against test|test-node or invoke GNU Make as make -- "$MAKE_TARGET".

The new preset validation in run-beacon-spec-tests and the shipped-feature cargo check -p ethlambda are both useful additions.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

No consensus-critical code is changed, so LMD GHOST/attestation/SSZ/XMSS paths are unaffected.

A few concerns:

  • Makefile, test-beacon-mainnet: the new single cargo invocation keeps --test beacon_spec_tests while adding -p ethlambda-blockchain. This likely works because --lib gives ethlambda-blockchain a matching target, but named test filters with multiple -p packages are a cargo subtlety. Please confirm CI actually exercises this exact recipe; if cargo treats the missing beacon_spec_tests target per selected package, it will fail.

  • Makefile, inert-paths-check: hits=$$(... | grep ... || true) masks all pipeline failures, not just the final grep’s “no matches” exit. If git ls-files -z or xargs grep fails after the initial guard, hits is empty and the check passes. Use set -o pipefail or check the pipeline status explicitly so scanning failures are not silently ignored.

  • bin/ethlambda/tests/benchmark_smoke.rs: the new test uses tempfile and serde_json, but no Cargo.toml change is included. Confirm those are already available as dev-dependencies/regular dependencies for the ethlambda package; otherwise cargo check --workspace --all-targets and Test (node) will fail at compile time.

  • Low priority: the timeout path in benchmark_smoke.rs reads only stderr before panicking. Including stdout as well would make diagnosing a hung/malformed benchmark run easier.

Otherwise the workflow changes look clean, and the preset validation/env-var quoting is a good security improvement.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

grep -H put each hit's file name in front of the line the pattern was matched against, so a .rs file under a docs/ or theme/ directory would fail the check for its own path, a false positive that widening the list could not fix. The check now decides on grep -h output, the source lines alone, and scans again with names and line numbers only to print the hits. The pattern goes after --, so a file name starting with - is not read as an option.

The comment now says what the scan skips (a tracked file missing from the worktree, which nothing builds either) and that a line after the macro can match too. CLAUDE.md's docs-only section names build scripts as a third way a build can read an inert path, and says the one build script today reads git and Cargo.lock only.
Selecting ethlambda-blockchain next to ethlambda-state-transition unifies blockchain's dependency and dev-dependency features into the spec suite's build. None enables a feature of an ethlambda crate today; the comment states that a feature changing the state transition or the crypto must never be enabled from there, or the spec suite would run with it and still pass.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Round two (the automated review of 17bee47). Fixed in 1f34b60 and 74f6e75.

Verification: no Rust changes, so make fmt and make lint do not apply. make inert-paths-check passes on the tree under macOS sh and dash, with BSD grep and xargs. It was also run against files planted through git add -N (removed afterwards), with results under the first bullet. GNU grep was not run locally; CI's Lint step on this head is that run. CI on 17bee47 (run 44432097627) answers several points below: Lint ran inert-paths-check green, Test (node) ran benchmark_smoke, and the combined beacon-mainnet command ran 335, 654 and 7493 tests.

Fixed in 1f34b60 and 74f6e75

  • Claude 4a: inert-paths-check matched file names. grep -H put each file's name into the text the pattern was matched against. A .rs file under a docs/ directory would have failed for its own path, and widening the list could not fix that.
    • The check now decides on grep -h output, the source lines alone, and scans again with names and line numbers only to print the hits.
    • A planted zz_plant/docs/c.rs including data.bin passes. One-line and split includes of docs/, README.md and theme/ paths still fail, named by file and line.
    • The pattern now follows -- (Kimi), so a file name starting with - is scanned, not read as an option. Checked with a root-level -dash.rs.
  • Claude 1: one beacon-mainnet invocation unifies blockchain's features into the spec build. Checked: blockchain's dev-dependencies add only tokio features, and nothing it depends on enables a feature of an ethlambda crate. The Makefile comment now states the invariant: a feature that changes the state transition or the crypto must never be enabled from ethlambda-blockchain.
  • Claude 6: build scripts. CLAUDE.md's docs-only section names a build.rs that reads an inert path, or lists it in rerun-if-changed, as a third way in. It also says the one build script today reads git and Cargo.lock only.

Checked, no change needed

  • Claude 5, Kimi 1, DeepSeek 2: || true swallowing errors. It covered only the last grep, whose 1 means no hits. In a checkout, the only error the scan can hit is a tracked file missing from the worktree, which nothing builds either. Failing on it would break make lint for anyone with an uncommitted deletion, so it is skipped, and the comment now says so. pipefail is not POSIX sh, and xargs exits 123 for any grep batch without a match. The || true is gone anyway, since the check is now an if.
  • Claude 4b, Grok 1: context lines. The split form puts the path on a context line, so a comment there can match too. That is the loud false positive round one accepted, and the comment now names it.
  • Grok 2: --error-unmatch. It exits 0 on a checkout, and CI's Lint step ran it green.
  • Grok 3: drift and breadth. Every .md is inert, so \.md" is the rule; docs/ unanchored was settled in round one. Both places carry a keep-in-step note.
  • Grok 4, DeepSeek 1: the combined invocation. Both packages are selected, and the counts match, as in round one. test-beacon-minimal never ran the blockchain suite.
  • Grok 5: the smoke test. cargo test runs integration tests by default, and Test (node) ran this one. A missing samples array fails it as jq -e did.
  • Claude 2: a disk guard. Settled in round one.
  • Claude 3, Grok 6, DeepSeek 3: tempfile, serde_json. tempfile is a dev-dependency of bin/ethlambda, and serde_json a dependency.
  • Kimi 2: make "$MAKE_TARGET". Its only values are literals in this repo's workflows.
  • DeepSeek 4: stdout on timeout. Stdout holds only the final report, so a hung run has nothing there.

Gemini and Codex found nothing to change.

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No actionable findings. The changes affect CI and test infrastructure; consensus logic is unchanged. The benchmark smoke test preserves the previous assertions and adds process timeout handling.

make inert-paths-check passed. Rust validation was blocked because rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks great. The CI optimizations (converting the benchmark smoke check into an integration test to share the test feature graph, combining the test-beacon-mainnet cargo invocations to avoid feature-resolution rebuilds, and dropping the redundant free-disk step on 145 GB runners) and the hardening changes (passing action inputs via environment variables, adding inert-paths-check, and checking ethlambda without dev-dependency feature unification) are all well-reasoned and cleanly implemented.

No issues found.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 801

This is CI and tooling only; no consensus code changes. The direction is sound: moving ${{ inputs.* }} into env vars, turning the benchmark smoke into a real test, and folding the blockchain tests into the spec-suite invocation. A few things to check before merge.

Correctness / risk

  1. Removing the benchmark-smoke job (ci.yml ~L243).

    • Check branch protection and any aggregate/gate job (needs: [...]) for a required check named Benchmark smoke.
    • If it is still listed as required, every PR will sit in "Expected — waiting for status" forever.
    • Nothing in this diff updates such a list, so please confirm out-of-band.
  2. benchmark_smoke.rs dependencies.

    • The test uses tempfile and serde_json, but no bin/ethlambda/Cargo.toml or Cargo.lock change is in the diff.
    • That's fine if both are already deps or dev-deps of the binary; otherwise test-node won't compile. Worth a quick confirm.
  3. benchmark_smoke.rs under a debug build.

    • The Makefile notes release-fast exists partly to avoid stack overflows in signature paths.
    • --mock-crypto probably sidesteps that. Still, a plain cargo test -p ethlambda (dev profile) now runs this test too, and 120 s may be tight or overflow-prone in debug.
    • If debug runs are known to be problematic, consider an early skip or a doc note.
    • Minor: the polling loop is fine and avoids a wait-timeout dep. Using files rather than pipes to avoid pipe-buffer deadlock is the right call.
  4. test-beacon-mainnet merged invocation (Makefile ~L116).

    • --lib --test beacon_spec_tests now applies to both packages. Cargo accepts a --test NAME that exists in only one selected package, so this works.
    • The semantic change is what the new comment describes: blockchain's deps and dev-deps features now unify into the spec-suite build.
    • The comment is a good guard, but it's prose-only. If you want it enforced, a cheap check would help, e.g. cargo tree -e features -p ethlambda-blockchain grepped for ethlambda-crate features in the Lint job.
  5. inert-paths-check coverage gaps beyond those already documented:

    • It doesn't match include!(...) or #[path = "..."]. Both are compile-time reads too, though unlikely to target .md.
    • Adding include! is cheap: 'include(_str|_bytes)?!'.
    • make lint now hard-requires a git checkout (tarball or vendored source can't lint). This is probably acceptable; just noting it.

Shell / Makefile nits

  • scan(): when grep can't open a file, xargs returns 123. That status is swallowed because the last pipeline command's status wins. This matches the documented "skip, don't fail" intent, but it's implicit; a one-word comment (# xargs status ignored) would help.
  • -A2 emits -- group separators into the stream. They're harmless against the current pattern, but if pattern is ever widened, add --no-group-separator (GNU) to keep it robust.
  • run-fixture-tests: make "$MAKE_TARGET" is injection-safe now. A value beginning with - would still be parsed as a make option. Inputs are workflow-controlled, so this is low risk; make -- "$MAKE_TARGET" closes it for free.

CI changes

  • Dropping free-disk is reasonable given the measured 65–70 GB on a 145 GB root.
    • However, lint, presets and tooling don't print df -h /. A vars.ETHLAMBDA_RUNNER pointed at a smaller self-hosted runner would surface there first, as a Bus error from rust-lld.
    • Consider adding the same always() df step to lint, which now builds two tokio feature sets.
  • Cargo check (binary, shipped features) is a good catch. It replaces the coverage the old cargo run gave implicitly.
  • Cosmetic: the ci.yml comment splits the path bin/ethlambda/tests/ benchmark_smoke.rs across a line break, which makes it hard to grep.

Docs

  • The storage/backend/mod.rs move from //! to a // comment is correct: the note no longer renders in rustdoc.
  • The CLAUDE.md and benchmarking.md updates are accurate and consistent with the Makefile.

Overall LGTM once Items 1 and 2 are confirmed.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll inspect the Makefile target wiring and the new smoke test's dependencies so the review isn't just based on the diff hunks.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

No consensus-critical logic is changed in this diff; it is CI/test-harness and Makefile cleanup. The preset/target env indirection in the GitHub Actions is a nice security improvement.

A few issues worth addressing:

  1. Makefile, inert-paths-check: the guard pipeline scan -h | grep -E "$$pattern" > /dev/null does not set pipefail. If git ls-files, xargs, or the first grep fails while producing no matching output, this target can pass silently. Since this check exists specifically to catch CI-skip false negatives, a false negative is worse than a false positive. Add set -o pipefail or capture scan -h output to a temporary file and test the command status explicitly.

  2. Makefile, test-beacon-mainnet: the combined invocation
    $(BEACON_TEST) -p ethlambda-blockchain --features ethlambda-state-transition/beacon-spec-tests,ethlambda-blockchain/beacon-spec-tests
    relies on Cargo applying --lib --test beacon_spec_tests globally across both selected packages. That works only if a missing beacon_spec_tests target in ethlambda-blockchain is tolerated because the package still has a matching --lib target. This is subtle enough that it should either be verified explicitly on CI or replaced with a clearer two-command setup, with a comment explaining why Cargo behaves that way.

  3. bin/ethlambda/tests/benchmark_smoke.rs: the diff does not show tempfile being added as a dev-dependency of bin/ethlambda. If it is not already present, this test will not compile. Also, fs::read_to_string(&stderr_path).unwrap_or_default() silently drops non-UTF-8 stderr on failure; reading bytes and using String::from_utf8_lossy would preserve diagnostic output.

  4. Minor timeout race in the same test: after started.elapsed() > DEADLINE, the process may have exited between the last try_wait and kill, but the code still panics with “benchmark still running”. Re-check try_wait after wait, or use a timeout-wait helper, to avoid a rare flaky failure at the 120s boundary.

The added cargo check -p ethlambda in the Lint job is a useful catch for shipped-features build breakage.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking issues found.

The CI changes look sound:

  • Action inputs are passed through environment variables and quoted, avoiding shell injection.
  • The combined mainnet beacon-test invocation preserves the state-transition spec tests and blockchain library tests. Its feature-unification risk is consensus-relevant, but the limitation is clearly documented and no problematic feature is introduced here.
  • benchmark_smoke.rs avoids pipe deadlocks, enforces a timeout with process cleanup, and strictly validates the JSON report.
  • inert-paths-check fails closed when no Rust sources are found, and its intentional false positives and detection limitations are well documented.

One very minor robustness nit: in bin/ethlambda/tests/benchmark_smoke.rs, around the try_wait().expect(...) call, an unexpected polling error would panic without killing or reaping the child. This is rare and non-blocking, but a cleanup guard would make the timeout guarantee comprehensive.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

Conflicts with #851 (runtime preset), resolved by keeping this branch's
single beacon-mainnet invocation on top of main's runtime-preset layout:

- Makefile: `BEACON_SPEC_TEST` now selects both `ethlambda-state-transition`
  and `ethlambda-blockchain` with the package-qualified features, and
  `test-beacon-mainnet` is that plus `--lib` under `BEACON_SPEC_PRESET=mainnet`.
  `test-beacon-minimal` selects the same two packages too, which goes beyond
  a textual merge: main's two beacon jobs share one cache entry that only the
  mainnet job saves, and a minimal run selecting state-transition alone
  resolves `libc`, `bytes`, `syn`, `tracing`, `smallvec` and state-transition
  itself with other features (cargo --unit-graph), so it would rebuild most of
  the graph every run. With both selected its units are a strict subset of
  the mainnet run's.
- The comment that `ethlambda-blockchain` enables no ethlambda feature is no
  longer true: main's blockchain dev-dependencies enable
  `ethlambda-state-transition/test-utils`, which only makes test helpers
  public. Reworded to say so.
- run-beacon-spec-tests: kept the preset check and main's shared
  `beacon-spec` key with mainnet-only saves; the check's comment no longer
  says the preset names the cache key.
@MegaRedHand
MegaRedHand marked this pull request as ready for review November 21, 2026 06:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant