Repository navigation
ci: drop free-disk, fold the benchmark smoke into Test (node), build beacon-mainnet once - #801
MegaRedHand wants to merge 12 commits into
Conversation
|
/review |
🤖 Grok Code ReviewI'll review the PR against the surrounding Makefile, CI, and test code so comments are tied to real behavior, not just the diff. I need the beacon test target, inert-path list, and how the smoke test is wired.The beacon-test invocation and the inert-path grep look like the places a silent skip or a false pass could hide. I'll read those definitions next. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Gemini Code ReviewThe changes look solid and well-reasoned:
LGTM! Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview of PR 801: drop
|
🤖 Codex Code ReviewNo actionable findings. The CI changes and benchmark smoke test look correct; no consensus, cryptography, or SSZ implementation changes are present.
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Kimi Code ReviewNo blocking correctness or security issues found.
One residual limitation: the No consensus-layer code is changed. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewThe PR is mostly CI/Makefile/docs changes; no consensus-layer code paths are touched, so fork choice, attestations, SSZ, and XMSS are unaffected. Feedback:
The preset validation added in Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
Review follow-upFixed in beef284
Checked, no change needed
Grok's review posted only its plan, no findings. |
…beacon-mainnet once - No CI job runs the free-disk action any more. It took 1.5-3.5 min per job, and hosted runners now have a 145 GB root disk: the daily run, which never freed anything, ended each test half at 65-70 GB used. The df -h reports after each build stay. kurtosis_assertoor.yml keeps it. - The benchmark smoke check is an integration test, bin/ethlambda/tests/benchmark_smoke.rs, run by Test (node). It drives the built binary in its own process with the old job's arguments and asserts the same report contract. The Benchmark smoke job, which compiled the whole node a second time, and its cache entry are gone. - make test-beacon-mainnet runs the spec suite and the chain actor's fixture tests in one cargo invocation. Two invocations resolved features differently and rebuilt most of the shared dependency graph for 335 tests that run in 5 s. - Comments and docs that described the old disk ceiling or the removed job are updated.
…, tidy a doc note Nits from the review round on the base PR, moved here so its CI and approval stay on the commit that was reviewed: - make inert-paths-check fails if an include_str! or include_bytes! names a path the docs-only skip treats as unread (Markdown, docs/, theme/, book.toml, .claude/, LICENSE, .gitignore). make lint runs it and so does CI's Lint job. It reads line by line, so a path assembled across lines or opened at run time gets past it; CLAUDE.md says so. - run-beacon-spec-tests rejects a preset other than mainnet or minimal up front, and passes it to make through the environment. - The note on why RocksDBBackend is not an intra-doc link moves out of the rendered module docs into a plain comment.
- inert-paths-check scans every tracked .rs file (git ls-files) rather than crates/, bin/ and tooling/, which missed shadow/ and printed an error for a missing directory. It reads each include_str!/include_bytes! line plus the two after it, which catches rustfmt's split form (`include_str!(` with the path on the next line). The pattern is a plain shell variable, and the comment says why it is unanchored. - run-fixture-tests passes its make target through the environment, the same as the beacon action's preset. - The benchmark smoke test kills the run and fails after 120 s instead of blocking until the job's own timeout, with output going to files so the child cannot block on a full pipe while the test polls it.
- Lint now checks `-p ethlambda` on its own. Every other build of the binary in CI (Lint's workspace check, Test (node)) unifies the dev-dependencies' features in, tokio's `test-util` among them, so the removed Benchmark smoke job's `cargo run` was the only PR build of the feature set `cargo build --release` ships. About a minute cold, 57 units, ~64 MB in the lint cache. - inert-paths-check exits 1 when git lists no `.rs` file (not a checkout, or git refusing the directory). It used to read that empty scan as a pass. - Its hits are printed with printf. echo in dash and in macOS sh expands backslash escapes, so a `\c` in a hit cut the list short. - Comments that still said a cold build outgrows the runner's disk, or that the test halves are split for disk, now give the current reason. CLAUDE.md now says the check does catch rustfmt's split form.
347b71c to
23787d6
Compare
|
/review |
🤖 Grok Code ReviewThe CI hardening is solid: preset allowlisting, env-based 1. 2. The “no tracked 3. Pattern and the docs-only list can drift, and the pattern is broader than the comment. The action comment says to keep the scanner in step with the 4. 5. Benchmark smoke is weaker than the old CI check, and it is easy to miss. 6. Timeout reap is fine; the poll is not a problem. The 120s kill plus Removing Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Gemini Code ReviewThis PR looks good to merge. Key improvements verified:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Codex Code ReviewNo actionable findings. The CI changes and benchmark smoke test look sound; consensus, signature verification, and SSZ logic are unchanged.
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview of PR 801: CI and tooling changesThis PR changes no consensus logic. Most of it is CI hygiene, and the direction is sound: script-injection hardening, dropping a redundant job, and a cheaper benchmark smoke test. A few items need checking before merge. Correctness / risk1. The new rule is: $(BEACON_TEST) -p ethlambda-blockchain --features ethlambda-state-transition/beacon-spec-tests,ethlambda-blockchain/beacon-spec-testsMerging the invocations is the point, but it has a consensus-relevant side effect. With resolver 2,
The multi-package 2. Removing
3.
|
🤖 Kimi Code ReviewNo blocking correctness or security issues found. The benchmark test has sound timeout, process cleanup, and output handling, and the combined beacon-test Cargo invocation preserves both packages’ tests while avoiding duplicate dependency builds. No consensus or state-transition logic is changed. Two non-blocking hardening suggestions:
The new preset validation in Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
🤖 DeepSeek Code ReviewNo consensus-critical code is changed, so LMD GHOST/attestation/SSZ/XMSS paths are unaffected. A few concerns:
Otherwise the workflow changes look clean, and the preset validation/env-var quoting is a good security improvement. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
grep -H put each hit's file name in front of the line the pattern was matched against, so a .rs file under a docs/ or theme/ directory would fail the check for its own path, a false positive that widening the list could not fix. The check now decides on grep -h output, the source lines alone, and scans again with names and line numbers only to print the hits. The pattern goes after --, so a file name starting with - is not read as an option. The comment now says what the scan skips (a tracked file missing from the worktree, which nothing builds either) and that a line after the macro can match too. CLAUDE.md's docs-only section names build scripts as a third way a build can read an inert path, and says the one build script today reads git and Cargo.lock only.
Selecting ethlambda-blockchain next to ethlambda-state-transition unifies blockchain's dependency and dev-dependency features into the spec suite's build. None enables a feature of an ethlambda crate today; the comment states that a feature changing the state transition or the crypto must never be enabled from there, or the spec suite would run with it and still pass.
|
Round two (the automated review of 17bee47). Fixed in 1f34b60 and 74f6e75. Verification: no Rust changes, so Fixed in 1f34b60 and 74f6e75
Checked, no change needed
Gemini and Codex found nothing to change. |
|
/review |
🤖 Codex Code ReviewNo actionable findings. The changes affect CI and test infrastructure; consensus logic is unchanged. The benchmark smoke test preserves the previous assertions and adds process timeout handling.
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Gemini Code ReviewThis PR looks great. The CI optimizations (converting the benchmark smoke check into an integration test to share the test feature graph, combining the No issues found. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview of PR 801This is CI and tooling only; no consensus code changes. The direction is sound: moving Correctness / risk
Shell / Makefile nits
CI changes
Docs
Overall LGTM once Items 1 and 2 are confirmed. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 Grok Code ReviewI'll inspect the Makefile target wiring and the new smoke test's dependencies so the review isn't just based on the diff hunks. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 DeepSeek Code ReviewNo consensus-critical logic is changed in this diff; it is CI/test-harness and Makefile cleanup. The preset/target env indirection in the GitHub Actions is a nice security improvement. A few issues worth addressing:
The added Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewNo blocking issues found. The CI changes look sound:
One very minor robustness nit: in Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
Conflicts with #851 (runtime preset), resolved by keeping this branch's single beacon-mainnet invocation on top of main's runtime-preset layout: - Makefile: `BEACON_SPEC_TEST` now selects both `ethlambda-state-transition` and `ethlambda-blockchain` with the package-qualified features, and `test-beacon-mainnet` is that plus `--lib` under `BEACON_SPEC_PRESET=mainnet`. `test-beacon-minimal` selects the same two packages too, which goes beyond a textual merge: main's two beacon jobs share one cache entry that only the mainnet job saves, and a minimal run selecting state-transition alone resolves `libc`, `bytes`, `syn`, `tracing`, `smallvec` and state-transition itself with other features (cargo --unit-graph), so it would rebuild most of the graph every run. With both selected its units are a strict subset of the mainnet run's. - The comment that `ethlambda-blockchain` enables no ethlambda feature is no longer true: main's blockchain dev-dependencies enable `ethlambda-state-transition/test-utils`, which only makes test helpers public. Reworded to say so. - run-beacon-spec-tests: kept the preset check and main's shared `beacon-spec` key with mainnet-only saves; the check's comment no longer says the preset names the cache key.
🗒️ Description / Motivation
Follows #800, now merged. That PR ported what ethrex already does: main-only caches, the Docker cache in ghcr, RocksDB behind a feature, and skipping docs-only changes. This one adds the three remaining improvements from the same CI-time analysis that ethrex doesn't have.
What Changed
free-diskstep in CI (ci.yml).df -h /after their build, so a runner change that leaves too little room would show there.kurtosis_assertoor.yml, which pulls Docker images rather than building Rust, keeps the action. Out of scope here.bin/ethlambda/tests/benchmark_smoke.rs, whichTest (node)runs throughmake test-node. TheBenchmark smokejob is gone.benchmark synthetic --mock-crypto, 4 validators, 4 warm-up slots, 3 iterations,--format json) and asserts what the oldjq -edid:schema_version == 1and 3 samples.CARGO_BIN_EXE_ethlambda). The benchmark's phase timers read the global metrics registry, which no other test in the process can then touch.Test (node). In the smoke job's log, the bin crate was the last unit, about 1.5 min. Building it with the test features is cheap: it reuses the test build's dependency graph, whichcargo runcould not.Benchmark smokewas not a required check, so no settings change is needed.cargo check --locked -p ethlambda. The old job'scargo runwas the only PR build of the binary with just the features it ships with. Every other build of it (Lint's workspace check,Test (node)) pulls in the dev-dependencies' features too, tokio'stest-utilamong them, so code that compiled only because of one would pass every PR job and breakcargo build --releaseon main. Its cold cost on CI is not measured yet.make test-beacon-mainnetis one cargo invocation, not three (state-transition's--lib, the spec suite, blockchain's--lib).cargo test -p ethlambda-blockchaintherefore rebuilt most of the shared dependency graph, fromlibcup, for 335 tests that run in 5 s. That rebuild took 11m54s cold before ci: port ethrex's cache, change-detection and RocksDB-feature practices #800 and 2m00s after it, now that RocksDB is gone.--librides in the same invocation as the spec suite, underBEACON_SPEC_PRESET=mainnet.make test-beacon-minimalselects the same two packages and features, though it runs only the spec binary. Since feat(beacon): select the preset at runtime from PRESET_BASE #851 the two beacon jobs share onebeacon-speccache entry, and only the mainnet job saves it. Selectingethlambda-state-transitionalone resolveslibc,bytes,syn,tracing,smallvecand the crate itself with other features (cargo --unit-graph), so the minimal job would rebuild most of the graph on every run. With both packages selected, every unit the minimal job builds is also in the mainnet build.ethlambda-blockchain's dev-dependency features into the spec suite's build. The only ethlambda feature among them isethlambda-state-transition/test-utils, which only makes test helpers public. The Makefile comment says a feature that changes what the state transition or the crypto does must never be enabled from there.Makefile,CLAUDE.md,docs/benchmarking.md,run-fixture-tests,ci.yml,daily_ci.yml) are updated.make inert-paths-check. It fails if aninclude_str!orinclude_bytes!names a path the docs-only skip treats as unread (Markdown,docs/,theme/,book.toml,.claude/,LICENSE,.gitignore).make lintruns it, and so does CI's Lint job..rsfile and reads each macro line plus the two after it, which catches rustfmt's split form. A path assembled further away (concat!) or opened at run time still gets past it; CLAUDE.md says so..rsfile (not a checkout, or git refusing the directory). Only source lines are matched (grep -h), not the file names grep prefixes them with; a second scan adds names and line numbers back to print the hits.shanddash; CI's Lint job is the GNU run.run-beacon-spec-testsvalidates its preset. Anything butmainnetorminimalnow fails up front with a clear error, instead of as an unknown make target. The preset reaches make through the environment.RocksDBBackendisn't an intra-doc link no longer renders in the public docs; it's a plain//comment now.Left alone, and why
Test minimal preset. I meant to fold it into the beacon and lint jobs. Two reasons not to:--libtests under the minimal preset. Since feat(beacon): select the preset at runtime from PRESET_BASE #851 the minimal beacon job runs the spec binary alone, which selects its preset at runtime. A unit-test binary never selects one, sopreset-minimal(minimal as the default) is still how those tests get minimal. They also run in thedevprofile, where overflow checks are on, unlikerelease-fast.Since ci: port ethrex's cache, change-detection and RocksDB-feature practices #800 it takes 4–5 min.
cargo-hakari, larger runners, and a lighter merge queue. Each is its own decision, not part of this PR.Expected effect
These are estimates from #800's cold runs and the per-job minimums of the last 40 runs. Nothing here is measured yet.
free-diskin each of the six other jobs that ran it, minus the ~1.5 min binary buildTest (node)gains and Lint's new binary check (about a minute cold, estimated from local timings).Test (node)is still the slowest job: it losesfree-diskbut gains the binary build.free-disk.Test (node)nets roughly +0, andBeacon spec tests (mainnet)drops by 2 more for the separate blockchain build.beacon-specentry holds one dependency graph instead of two. Both make it likelier that main's entries fit in 10 GB and stay warm, which is where ci: port ethrex's cache, change-detection and RocksDB-feature practices #800's larger gain comes from.beacon-specentry, which was built with the old feature resolution, so they run close to cold. That is expected. The gain shows once main saves the entry again.Tests Added / Run
bin/ethlambda/tests/benchmark_smoke.rs, run locally withcargo test --profile release-fast -p ethlambda --test benchmark_smoke.--no-runbuilds exactly three test executables: blockchain's lib tests, state-transition's lib tests andbeacon_spec_tests. Blockchain's lib reports335 tests, with0ignored under the feature, the same as the separate command's run in CI.cargo --unit-graphon the two merged recipes: mainnet's roots are those same three test targets, minimal's isbeacon_spec_testsalone, and every minimal unit is also in the mainnet graph (state-transition alone would have added 9 units with other features). Neither recipe has been run locally; the two beacon jobs are their first run.cargo check --locked -p ethlambda(Lint's new step) is clean on the merged tree.cargo fmt --all -- --check, workspace clippy at-D warnings(which covers the new test), actionlint 1.7.12 withpr_lint_gha.yaml's flags, andgh actions-lock --verify: all clean.Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test— only the new integration test locally; CI runs the rest