Skip to content

ci: add a Commit-Boost check to the kurtosis battery - #708

Draft
MegaRedHand wants to merge 23 commits into
mainfrom
ci/kurtosis-commit-boost
Draft

MegaRedHand wants to merge 23 commits into
mainfrom
ci/kurtosis-commit-boost

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a Commit-Boost check to the kurtosis battery: ethlambda builds blocks through Commit-Boost's PBS module, the MEV-Boost replacement that speaks the same builder-API.

What it runs

tooling/kurtosis-assertoor/commit_boost_params.yaml:

  • Participants: the base battery's three (lighthouse bootnode; lighthouse with the ethlambda validator client; ethlambda beacon node with the ethlambda validator client).
  • MEV: mev_type: commit-boost. ethereum-package then adds a flashbots relay, an rbuilder with its own lighthouse, and one Commit-Boost PBS sidecar per participant.
  • Transactions: spamoor at a light load (10 transfers and 1 blob transaction a slot). rbuilder submits no bid for an empty block, and spamoor's default load slowed the relay past Commit-Boost's 950 ms getHeader timeout.
  • Images: the relay, builder, PBS and spamoor images are pinned by digest, since the package's defaults are floating tags.
  • No assertoor: its checks still fail on the unrelated GLOAS_FORK_EPOCH spec mismatch (README, "Known failure").

scripts/check-builder-blocks.sh polls Prometheus metrics until all of these hold, or 40 minutes pass:

Service Condition Shows
ethlambda beacon node lean_beacon_builder_registrations_total{outcome="forwarded"} >= 1 registrations reach Commit-Boost
ethlambda beacon node lean_beacon_builder_payload_source_total{source="builder"} >= 1 produceBlockV3 served a builder block
ethlambda VC on ethlambda ethlambda_validator_builder_registrations_sent_total >= 1, ethlambda_validator_blinded_blocks_proposed_total >= 1 the validator client registers and publishes blinded blocks
ethlambda VC on lighthouse ethlambda_validator_blinded_blocks_proposed_total >= 1 the validator client works against a third-party beacon node

The script writes each service's builder metrics and the unmet conditions to the results directory on every exit.

Dispatch only, for now

Depends on

Upstream workarounds

Breakage Worked around in
rbuilder refuses ENR bootnodes fork PR #23, through the pin
Commit-Boost requires SECONDS_PER_SLOT, which the genesis generator no longer writes an explicit chain in mev_params.commit_boost_config (chain_id must be a hex string)

Tested

  • shellcheck, actionlint and yq pass.
  • Local run, passed: an image built from feat(validator): publish proposer preferences (gloas) #707's head (ae1a3d2f) met all five conditions about 12 minutes after the enclave started.
    • The relay delivered a builder block in 14 of the 32 slots of epoch 1.
    • ethlambda's beacon node served 1 builder payload out of 16 proposals.
    • Its validator client proposed 1 blinded block; the one behind lighthouse proposed 3.
  • Earlier local runs, failed: they are why spamoor runs, and at a light load.
    • With no transactions, the relay never held a bid, so every getHeader answered 204.
    • With spamoor's defaults, the validator client behind lighthouse got no bid in 40 minutes.
  • CI: the job itself has not been dispatched.

The builder PR stack (#694 to #705) lets ethlambda's beacon node and
validator client build blocks through a MEV-Boost compatible builder, but
nothing exercises it against a real relay and builder. Commit-Boost's PBS
module is a drop-in MEV-Boost replacement, so a devnet with
`mev_type: commit-boost` covers the whole path.

The check reads the ethlambda services' builder metrics instead of running
assertoor, because the base battery's assertoor tests currently fail on an
unrelated spec-endpoint gap. It passes once registrations are forwarded, the
ethlambda beacon node serves a builder payload, and the ethlambda validator
client proposes blinded blocks both through that node and through lighthouse.

The new `commit-boost` job runs on manual dispatch only: main's image has no
`--builder` or `--builder-proposals` until the stack lands. It has its own
ethereum-package pin, which needs the fork's builder-flags branch pushed. The
relay, builder and PBS images are pinned by digest since the package defaults
are floating tags.
A contributor running the check locally needs to know how to get an image
with the builder flags and that kurtosis will use it without pulling, or
it looks like the run needs a published image. The fork pin also now has a
PR to point at instead of an unpushed branch.
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

Looks good overall. I did not find consensus-layer logic changes here; this PR is CI/tooling-only, so fork choice, attestation processing, state transition, XMSS, and SSZ risks are not directly touched.

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh:52
    value_of() prints with "%d", but Prometheus sample values are numeric strings and may be rendered as floats or scientific notation. In awk, printf "%d" truncates, so values like 0.9 become 0 and 1e+00 handling is awk-dependent. Safer to compare numerically in awk and print with %.0f only if you really want integers, or keep the raw numeric string and compare as a number. This is a correctness risk for the CI gate.

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh:38
    scrape() strips the scheme and then hardcodes "/metrics": curl ... "${addr#*://}/metrics". If kurtosis port print already returns the full metrics URL rather than just scheme://host:port, this builds an invalid URL. The current code assumes a very specific output shape from Kurtosis. I’d either document that contract explicitly or normalize more defensively.

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh:31
    The check hardcodes service names and participant ordering. The README documents that coupling, but it is still brittle: any change in ethereum-package naming or participant order will make the check fail even if builder flow works. At minimum, consider asserting early that the expected services exist and fail with a clearer message.

  • .github/workflows/kurtosis_assertoor.yml:264
    notify only depends on [image, assertoor]. If battery: both and the new commit-boost job fails while assertoor passes, the workflow fails but Slack notification does not mention the failing job. If that is intentional, fine; otherwise notify should either include commit-boost in needs or be renamed/scoped to assertoor-only failure reporting.

  • .github/workflows/kurtosis_assertoor.yml:97
    The assertoor gating expression is correct for the three dispatch modes, but it’s subtle. A short comment here would help maintainability, because future edits could easily invert the logic and unintentionally skip the base battery for battery: both.

  • tooling/kurtosis-assertoor/commit_boost_params.yaml:47
    The comment says the digests are “what the tags pointed to on 2026-10-26”. Pinning by digest is good for supply-chain stability, but this comment will age quickly and can mislead reviewers if the PR is merged later. I’d rephrase to “pinned as of this update” or include how to refresh them, not a date-sensitive claim.

If you want, I can turn the first three points into concrete patch suggestions.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 708: Commit-Boost kurtosis check

This is a CI and tooling change with no consensus code. It looks sound overall. I read the diff but did not run the workflow or the script, so the points below are from reading only.

Good

  • Both the package commit and the relay, builder and PBS images are pinned (the images by digest). The job verifies that HEAD equals the pinned SHA after fetching.
  • The new job is dispatch-only, with a clear note on when to drop that restriction.
  • The base assertoor job's if is safe on non-dispatch events, because the event_name check short-circuits before inputs.battery is read.
  • The script handles set -euo pipefail carefully. The || true guards are in the right places, and the metrics dump runs on every exit through the trap.

Points to consider

  1. Failure notification. notify has needs: [image, assertoor] (the notify job, unchanged in the diff). A commit-boost failure, or a run with battery: commit-boost, will not post to Slack. This is fine while the job is manual. Add it to needs when the job is enabled on push, and check that the notify if copes with a skipped assertoor.
  2. Metric port name. check-builder-blocks.sh assumes the kurtosis port is named metrics for all three services (kurtosis port print "$ENCLAVE" "$1" metrics). If the name is wrong, scrape silently returns nothing. The script then reports "unmet = 0" for 40 minutes rather than naming the cause. Print a distinct message when the port lookup fails, at least on the last iteration.
  3. Hardcoded service names. cl-3-ethlambda-geth, vc-2-geth-lighthouse-ethlambda and vc-3-geth-ethlambda depend on the package's naming and on the participant order. The YAML comment and the README document this. A cheap guard would be to check that the services exist (kurtosis service inspect) up front, so a rename gives an immediate, clear error.
  4. value_of precision. printf "%d" truncates float values. Counters are integers, so this is fine. A gauge-style or 1e+00 exposition would still work in awk, so no change is needed.
  5. Fork pin fragility. The README already notes that the pinned commit can be garbage collected after a squash-merge of feat: wire ethlambda to the external builder ethereum-package#22. A CI failure at git fetch would be the first sign. Consider tagging the commit in the fork to keep it reachable.
  6. Minor. yq is assumed to be on ubuntu-latest, which is true today. The job's timeout-minutes: 90 comfortably covers the 40-minute poll plus startup.

No blocking issues.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR, which adds a Commit-Boost builder test battery to the Kurtosis/Assertoor CI workflow. The changes are primarily in CI configuration, documentation, and a shell test script — no consensus-layer Rust code is modified.

General Assessment

This is a well-structured CI addition with good documentation. The shell script has several issues that could cause flaky or incorrect behavior in CI.


Detailed Findings

1. Shell Script: value_of awk selector parsing is fragile

File: tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh
Lines: 63–69

value_of() {
    awk -v sel="$2" '
        index(sel, "{") { if ($1 == sel) sum += $NF; next }
        { split($1, p, "{"); if (p[1] == sel) sum += $NF }
        END { printf "%d\n", sum }
    ' <<<"$1"
}

Issues:

  • Floating-point output risk: printf "%d\n" with awk's default floating-point arithmetic can produce scientific notation for large values (e.g., 1e+06), causing the subsequent [ "$value" -ge 1 ] test to fail with "integer expression expected" in bash. Prometheus counters can grow large over a long-running test.

  • Label matching with index(sel, "{") is imprecise: A metric name containing { as part of a label value (unlikely but valid in Prometheus text format) would trigger the wrong branch. The selector format is name{labels} — better to use a more robust parse.

Suggested fix:

value_of() {
    awk -v sel="$2" '
        BEGIN {
            # Determine if selector has labels: must start with name{ and end with }
            has_labels = (match(sel, /^{/) == 0 && match(sel, /{.*}$/) > 0)
        }
        has_labels { if ($1 == sel) sum += $NF; next }
        { split($1, p, "{"); if (p[1] == sel) sum += $NF }
        END { printf "%.0f\n", sum + 0 }  # force numeric, round to integer
    ' <<<"$1"
}

Even better, avoid awk floating-point entirely for integer counters:

value_of() {
    local text="$1" sel="$2"
    if [[ "$sel" == *"{"* ]]; then
        grep -F "$sel" <<<"$text" | awk '{sum+=$NF} END {printf "%d\n", sum+0}'
    else
        awk -v name="$sel" '{split($1,p,"{"); if(p[1]==name) sum+=$NF} END {printf "%d\n", sum+0}' <<<"$text"
    fi
}

2. Shell Script: scrape silently ignores failures

File: tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh
Lines: 52–56

scrape() {
    local addr
    addr=$(kurtosis port print "$ENCLAVE" "$1" metrics 2>/dev/null) || return 0
    curl -sf --max-time 10 "${addr#*://}/metrics" 2>/dev/null || true
}

Issue: If kurtosis port print fails (service not yet started), the function returns 0 with empty output. This is intentional for startup polling, but if kurtosis itself fails (e.g., enclave doesn't exist), the script silently continues. Consider distinguishing "service not ready" from "enclave missing" for better debugging.

Minor suggestion: Add a warning when addr is empty but the enclave exists, or at least log kurtosis errors to stderr after initial startup period.


3. Shell Script: Race condition in evaluate with repeated scraping

File: tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh
Lines: 88–99

evaluate() {
    ...
    for entry in "${CONDITIONS[@]}"; do
        IFS='|' read -r svc series desc <<<"$entry"
        value=$(value_of "$(scrape "$svc")" "$series")
        ...
    done
}

Issue: scrape "$svc" is called twice per service per loop iteration — once in evaluate and once in dump_metrics via the EXIT trap. More critically, within evaluate, each condition calls scrape separately, meaning metrics for CL3 are fetched 2 times, VC2 once, VC3 twice — all at different timestamps. A transient network issue could make one call succeed and another fail.

Suggested fix: Cache scrapes per service per evaluation:

evaluate() {
    local entry svc series desc value unmet=0
    local -A metrics
    # Pre-fetch all service metrics once
    for svc in "${SERVICES[@]}"; do
        metrics[$svc]=$(scrape "$svc")
    done
    
    for entry in "${CONDITIONS[@]}"; do
        IFS='|' read -r svc series desc <<<"$entry"
        value=$(value_of "${metrics[$svc]}" "$series")
        ...
    done
}

4. Shell Script: grep '^unmet' after evaluate may operate on stale data

File: tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh
Lines: 113–114, 122

if summary=$(evaluate); then
    ...
fi
echo "$summary"
grep '^unmet' <<<"$summary" > "$OUT/failed.txt" || true

Issue: If evaluate fails (returns 1), summary still contains the output due to set -e not triggering on command substitution failure in the if condition. This is actually correct bash behavior, but subtle. However, if evaluate outputs to stderr or has a pipefail issue, summary could be partial.

More concerning: The finish trap calls dump_metrics which overwrites *.prom files. If the script is interrupted (SIGTERM from CI timeout), the EXIT trap runs, but the failed.txt write at line 122 may not happen if the signal arrives between the loop and that line. The failed.txt should probably be written in the finish trap as well for consistency.


5. Shell Script: Missing kurtosis availability check

File: tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh
Line: 53

The script assumes kurtosis is in PATH. For local runs per the README, this is documented, but a clear error message would help:

command -v kurtosis >/dev/null 2>&1 || { echo "kurtosis not found in PATH" >&2; exit 2; }

6. Workflow: notify job dependency incomplete

File: .github/workflows/kurtosis_assertoor.yml
Line: 264 (original), context around line 170

notify:
    needs: [image, assertoor]

Issue: The notify job only depends on image and assertoor. If commit-boost is run via battery: both or battery: commit-boost, failures in commit-boost will not trigger the Slack notification. The commit-boost job is not in the needs array.

Suggested fix:

notify:
    needs: [image, assertoor, commit-boost]
    if: always() && (needs.assertoor.result == 'failure' || needs.commit-boost.result == 'failure' || needs.image.result == 'failure')

Or adjust the existing if condition to include commit-boost. Without this, the commit-boost job can fail silently in terms of notifications.


7. Workflow: commit-boost job missing outputs

File: .github/workflows/kurtosis_assertoor.yml
Lines: 188–267

The assertoor job defines outputs: passed: ${{ steps.result.outputs.passed }}, but commit-boost has no outputs. This is fine if no downstream job needs the result, but for consistency and future extensibility, consider adding a success/failure output.


8. YAML: commit_boost_params.yaml uses future date

File: tooling/kurtosis-assertoor/commit_boost_params.yaml
Line: 44

# bump them deliberately.
mev_params:
  # ... pin comments say "2026-10-26"

Issue: The comment says "2026-10-26" — this appears to be a future date (assuming current year is 2024 or 2025). If this is intentional (pre-planned pin date), document why. If it's a typo, correct it.


9. Documentation: Fork GC risk well-documented but needs action item

File: tooling/kurtosis-assertoor/README.md
Lines: 143–147

The README correctly identifies that COMMIT_BOOST_PACKAGE_REF may be garbage-collected if the fork PR is squash-merged. However, there's no tracking issue or TODO in the code to re-pin. Consider adding a TODO(#<issue>): comment in the workflow file near the pin, or a scheduled reminder.


10. Security: Docker image digests not fully pinned in workflow

File: .github/workflows/kurtosis_assertoor.yml

The workflow uses actions/checkout@v6 and actions/upload-artifact@v7 — these are version tags, not SHA pins. For a security-critical blockchain project, consider pinning to full commit SHAs for all GitHub Actions to mitigate supply-chain attacks. This is existing practice in the file, not new in this PR, but worth noting.


Positive Observations

  1. Good separation of concerns: The COMMIT_BOOST_PACKAGE_REF is separate from ETHEREUM_PACKAGE_REF, allowing independent updates.

  2. Clear dispatch-only gating: The if: github.event_name == 'workflow_dispatch' condition is well-explained in comments and README, with explicit guidance on when to remove it.

  3. Comprehensive metric coverage: The five conditions cover the full builder block flow: registration forwarding, payload sourcing, validator registration, and blinded block proposals through both native and third-party beacon nodes.

  4. Proper cleanup: Both assertoor and commit-boost jobs use if: always() for enclave removal and artifact upload.

  5. Documentation quality: The README is thorough, including local run instructions, CI usage, and the fork pin rationale.


Summary

Priority Item Location
High Fix notify job to depend on commit-boost .github/workflows/kurtosis_assertoor.yml
Medium Fix awk integer parsing in value_of scripts/check-builder-blocks.sh:63
Medium Cache metrics to avoid race conditions scripts/check-builder-blocks.sh:88
Low Add kurtosis availability check scripts/check-builder-blocks.sh (top)
Low Verify/fix date comment commit_boost_params.yaml:44
Low Add TODO for fork re-pin .github/workflows/kurtosis_assertoor.yml or README

Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

A local run did not start. The newest reth-rbuilder image parses only enode
bootnodes while the package passes ENRs, and Commit-Boost's PBS requires
SECONDS_PER_SLOT, which the genesis generator no longer writes.

The first is fixed in the fork (PR #23), so the package pin moves to that
commit. The second is worked around by giving Commit-Boost an explicit chain
through `commit_boost_config`; its `chain_id` must be a hex string. The README
lists both workarounds so they can be dropped when upstream is fixed, and
explains the harmless "Builder is not answering" logs at startup.
A local run registered every validator but never got a builder block:
rbuilder submits no bid for a block without transactions, and the
enclave had none, so the relay held no bids for any proposer and every
getHeader answered 204. With transactions flowing, the same enclave
served a builder payload within a few slots, and the ethlambda
validator client signed the blinded block that the relay delivered.

spamoor runs with the package's default spammers (transfers and blob
transactions), pinned by digest like the other images. The README also
explains the no_finality fallbacks a run counts before the chain first
finalizes.
With spamoor's defaults (a chain-load group, 200 transfers and 6 blob
transactions a slot) the relay's getHeader slowed to Commit-Boost's
950 ms timeout, so most proposals got no bid, and in 40 minutes the
ethlambda validator client behind lighthouse never received one.

10 transfers and 1 blob transaction a slot keep bids coming without
that slowdown: a local run met every condition about 12 minutes after
the enclave started, and the relay delivered a builder block in 14 of
the 32 slots of epoch 1. The package's own 100 Uniswap swaps a slot,
which it adds whenever a builder runs, stay on; they are not
configurable.
#713 locked every workflow's actions in .github/workflows/actions.lock,
after this job was written. Its checkout@v6 and upload-artifact@v7 no
longer match the v6.1.0 and v7.0.2 the lockfile pins for this workflow,
so verify-lockfile fails and the enforced lockfile refuses to start the
workflow at all (startup_failure on the PR's Kurtosis Assertoor run).

Use the same refs as the assertoor job beside it, including the `$/`
same-repository form #713 gave the free-disk action.
@MegaRedHand
MegaRedHand marked this pull request as draft November 18, 2026 13:27
…s explicit

From the automated review of this PR:

- Counter values are summed and truncated in awk, then printed with %.0f, so a large or exponent-notation value (1e+06) reads whole and a value below 1 never meets a >= 1 condition.
- Each service's metrics port must resolve before polling starts. A renamed service, a reordered participant list or a missing kurtosis/curl now fails at once with exit status 2 and kurtosis's own message, rather than reading as "not met yet" for 40 minutes.
- One evaluation scrapes every service once, so the conditions on one service read the same snapshot.
- A service that does not answer is reported as such, not as a 0.
- failed.txt is rewritten after every evaluation, so a cancelled run still says what was unmet.
- The address kurtosis prints is reduced to host:port, whatever scheme or path surrounds it.

Checked against a stub kurtosis and local metrics servers, under macOS's bash 3.2 and under ubuntu:24.04's bash 5.2 with mawk; shellcheck is clean.
…the Commit-Boost job

notify reports pushes only and the commit-boost job runs on dispatch only, so adding the job to notify's needs today would wire up nothing. The note sits where the event condition is dropped, which is when it starts to matter.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P1] Initialize slashing protection for both validator clients — commit_boost_params.yaml:35 and line 40. Both ethlambda participants omit vc_extra_params: ["--init-slashing-protection"], which the existing battery supplies. Fresh devnet keys have no protection records, so SlashingProtection::open_locked rejects startup. Neither client can register or propose, preventing this check from passing. Add the flag to both participants.

Shell syntax validation passed; the devnet was not run.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

This PR looks good. The workflow conditions, Kurtosis parameters, metric-polling script, and documentation are well-structured and handle failure and cleanup paths cleanly.

A few highlights:

  • .github/workflows/kurtosis_assertoor.yml: The battery input conditions on assertoor and commit-boost properly preserve existing push/PR behavior while allowing isolated or combined manual runs against custom images.
  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh: Failing fast with exit code 2 on missing metrics ports and keeping failed.txt and per-service .prom dumps updated across iterations and EXIT traps will make debugging CI timeouts or cancellations straightforward.
  • tooling/kurtosis-assertoor/commit_boost_params.yaml: Pinning third-party MEV/spammer images by digest and documenting the upstream workarounds (SECONDS_PER_SLOT in commit_boost_config and the ethereum-package ref pin) makes future maintenance clear.

Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 708: Commit-Boost check for ethlambda

This PR only touches CI and tooling; no Rust or consensus code changes. The tooling is well built: it pins images by digest, passes IMAGE through env and strenv (so there is no expression injection), stops early with exit status 2 on setup errors, and rewrites failed.txt after every evaluation. A few issues are worth addressing.

Likely bugs

  1. .github/workflows/kurtosis_assertoor.yml, the "Free up disk space" step: it says uses: $/.github/actions/free-disk.

    • uses: does not evaluate expressions.
    • A local action path must start with ./, so this will fail to resolve.
    • It should almost certainly be uses: ./.github/actions/free-disk, matching the assertoor job. If this is just how the diff was rendered, ignore this item.
  2. Same file, the "Start the devnet" step: kurtosis run ... | tee ... hides a failed kurtosis run.

    • A run: step without an explicit shell: runs under bash -e {0}, which has no pipefail. The step's status is therefore tee's status.
    • A failed devnet start then moves on to check-builder-blocks.sh. That script exits 2 with "no metrics port", which points at service naming instead of the real cause.
    • Fix: add shell: bash to the step, which gives -eo pipefail, or start the script with set -o pipefail.

Robustness of check-builder-blocks.sh

  1. value_of (around lines 99–105): labelled selectors only match the exact name{labels} token.

    • Suppose lean_beacon_builder_payload_source_total later gains a second label (say {source="builder",fork="fulu"}), or the exporter orders labels differently.
    • The condition would then quietly read 0 until the 40-minute deadline, instead of failing clearly.
    • Consider matching the metric name plus "label set contains outcome="forwarded"", for example by testing index($1, "{" lbl) || index($1, "," lbl) on the name-stripped part.
  2. Same function: $NF assumes the exposition line has no timestamp.

    • The Prometheus text format allows name{..} value timestamp; with a timestamp, $NF is the timestamp and the sum is meaningless.
    • The Rust prometheus crate does not emit timestamps, so this is fine today.
    • $2 would be the safer field, since metric names and label values here contain no spaces.
  3. Deadline accounting (lines 167–180): minor. The deadline is checked only after a full evaluate, and each curl can take up to 10 s. Worst-case overrun is therefore about 30 s, which is fine, but it is not quite "for up to the timeout it is given".

Workflow and configuration

  1. COMMIT_BOOST_PACKAGE_REF points at a commit from an unmerged PR.

    • The README already notes the garbage-collection risk.
    • Separately, git fetch --depth 1 <sha> works only if the commit is reachable from a ref in lambdaclass/ethereum-package. It must sit on a branch of that repo, not on a contributor's fork.
    • Worth confirming, or a dispatch could fail at the test "$(git rev-parse HEAD)" = ... line.
  2. commit_boost_params.yaml: the explicit chain duplicates the network parameters (genesis_fork_version, chain_id, fulu_fork_slot).

    • The header comment flags this, and I checked that 0x301824 equals 3151908.
    • This drift risk deserves a pointer in network_params.yaml as well, so whoever edits the base battery sees it.
  3. Nits:

    • The spamoor digest comment says 2026-10-27, while the mev_params comment says the images were pinned on 2026-10-26. Unify the dates, or say they were pinned separately.
    • The Kurtosis apt source uses [trusted=yes], which skips signature verification. This was presumably copied from the existing job; it is acceptable for CI, but worth knowing.
    • Job timeout: 90 minutes covers devnet startup, the 40-minute poll and the dump. That is adequate, but leaves little margin if image pulls are slow.

Documentation

  • The README is thorough, and the TODO for wiring commit-boost into notify is clearly recorded in both the workflow and the README.

Summary: fix Item 1 and Item 2 before merging. Item 3 is a worthwhile hardening so that a metric gaining a label shows up as a clear failure rather than a 40-minute timeout. The rest are optional.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

I'll compare the new Commit-Boost job and check script with the existing Assertoor workflow so the review is grounded in how this repo actually runs Kurtosis.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

Review summary: this is CI/testing tooling only, no consensus-layer or Rust code. Overall the approach is reasonable, but there are a few concrete issues.

  • .github/workflows/kurtosis_assertoor.yml, commit-boost job, Free up disk space step:
    uses: $/.github/actions/free-disk is invalid. Local actions are referenced as ./.github/actions/free-disk, not $/.... This will fail the job before it starts.

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh, value_of function:
    Both sum += $NF lines should be sum += $2. Prometheus text exposition may include an optional trailing timestamp, so the metric value is the second whitespace-separated field, not always the last. Using $NF would read the timestamp when present and can cause conditions to pass or fail spuriously.

  • .github/workflows/kurtosis_assertoor.yml, commit-boost job, Start the devnet step:
    kurtosis run ... | tee "$RUNNER_TEMP/kurtosis_output.txt" runs under the default bash -e without pipefail, so a non-zero exit from kurtosis run is masked by tee succeeding. Add set -o pipefail to the step shell or explicitly check ${PIPESTATUS[0]}.

  • .github/workflows/kurtosis_assertoor.yml, commit-boost job:
    Verify the action refs actions/checkout@v6.1.0 and actions/upload-artifact@v7.0.2 are valid. As of common public releases these versions are not current, so this may be another typo. Also confirm yq is installed on the runner for the image-rewrite step, or add a setup step; the new job depends on yq without installing it.

No consensus-layer correctness, SSZ, XMSS, or Rust memory-safety concerns in this PR.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The workflow gating handles all three battery choices correctly, and the pinned package commit/container digests, always() cleanup, and diagnostic artifacts are good practices.

Minor robustness suggestion:

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh, metrics_address() around lines 76–81: consider rejecting empty or malformed successful output from kurtosis port print. Currently, an empty address passes the setup check and is subsequently treated as an unreachable service, resulting in a 40-minute timeout rather than the intended immediate setup failure with exit status 2.

The dispatch-only limitation means this does not yet provide push/PR regression coverage, but that is clearly documented, including the required follow-up to enable the job and update Slack notification dependencies.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

…-slashing-protection

Since #734 the ethlambda validator client refuses keys with no slashing-protection record, so both ethlambda VCs exited at startup and kurtosis rolled them back. network_params.yaml has passed the flag since then; this file now does the same, for the same reason: genesis keys in a fresh directory have never signed.
The step piped kurtosis run into tee under bash -e without pipefail, so it took tee's status: a Starlark error passed the step and surfaced only as missing services in the next one.
@MegaRedHand
MegaRedHand marked this pull request as ready for review November 18, 2026 19:30
@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

PR 708 review: Commit-Boost check (CI and tooling only)

This PR touches no Rust code, so the consensus-layer items (fork choice, state transition, XMSS, SSZ) don't apply. The change is well documented, and the shell script is careful: it uses pipefail, takes one snapshot per evaluation, fails fast with exit 2 on setup errors, and rewrites failed.txt after every evaluation. One item likely blocks the job; the rest are small.

Likely blocker

  1. .github/workflows/kurtosis_assertoor.yml, commit-boost job, "Free up disk space" step
    uses: $/.github/actions/free-disk
    This is not a valid uses: reference. A local action must be written as ./.github/actions/free-disk, and uses: does not support expressions. The workflow will either fail to parse or fail at this step. Please match whatever the assertoor job uses.

Workflow

  1. "Dump the enclave" step: cp "$RUNNER_TEMP/args.yaml" ... has no || true, unlike the two lines after it.

    • If "Point the args at the image under test" failed before writing the file, the cp aborts the step.
    • The kurtosis enclave dump then never runs, which is the case where you most want the dump.
    • Fix: add || true to that line.
  2. The COMMIT_BOOST_PACKAGE_REF pin is a commit from an unmerged PR branch.

    • The README already covers the garbage-collection risk.
    • Consider also adding a TODO next to the env var, so it is visible to anyone touching the workflow, not just README readers.
  3. Input handling is good. inputs.image flows through env: and strenv() and is never interpolated into run:, which avoids script injection.

scripts/check-builder-blocks.sh

  1. value_of (lines ~99–105) matches labelled selectors as an exact name{labels} token. This is brittle in a few ways:

    • It breaks if the metric gains another label, e.g. {outcome="forwarded",relay="..."}.
    • It breaks if the exporter orders labels differently.
    • A label value containing a space would split $1.
    • $NF assumes there is no trailing timestamp in the exposition line. The Rust prometheus crate does not emit one, but $2 is safer if the first field is guaranteed to have no spaces.

    A more robust approach is to match on the metric name plus the presence of the label pair, e.g. index($1, "outcome=\"forwarded\""). This is optional, given that the metrics are under our control.

  2. TIMEOUT_MIN is not validated. It is used directly in $((SECONDS + TIMEOUT_MIN * 60)). A quick [[ $TIMEOUT_MIN =~ ^[0-9]+$ ]] || exit 2 would make a typo fail clearly instead of producing a strange deadline.

  3. summary=$(evaluate) && status=0 is correct but subtle. status is never reset to 1, and that is fine only because a success breaks the loop immediately. A short comment, or if summary=$(evaluate); then status=0; break; fi, would make the intent explicit.

  4. Nit: the EXIT trap scrapes on the setup-failure path. On the exit-2 "missing metrics port" path, finish → dump_metrics still tries to scrape services already known to be missing. This is harmless (|| true), but it adds one more kurtosis port print failure per missing service to the logs.

Docs and params

  1. Inconsistent pin dates in commit_boost_params.yaml.

    • The spamoor digest says "2026-10-27".
    • The mev_params block says the tags were pinned "on 2026-10-26".
    • Please unify them, or note that they were pinned separately.
  2. Missing year in the README. The "Upstream workarounds" table says "since its 08-05 refactor".

  3. Values checked. chain_id = "0x301824" is correct for 3151908, and 0x10000038 matches the kurtosis default genesis fork version.

Apart from Item 1, this looks good to merge.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Verify builder blocks reach the chain before passing — check-builder-blocks.sh:45–46. blinded_blocks_proposed_total increments even for Published::BroadcastNotImported (crates/validator/src/proposal.rs:860–880), while payload_source_total{source="builder"} only confirms bid selection. All conditions can therefore pass while builder proposals fail import. Add a check that a builder-backed proposal from each tested validator path appears on-chain.

Shell syntax validation passed. No Rust or consensus implementation changes are included.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

Looks good to me! The workflow wiring, Kurtosis parameters (with digest-pinned upstream images), metric polling script, and README documentation are all clean and well-structured.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

The PR is entirely CI/tooling (workflow, YAML params, bash check script, documentation). No Rust protocol code is touched, so consensus-critical correctness is not affected. The bash script is generally well-written and robust. Below are the specific issues found.


.github/workflows/kurtosis_assertoor.yml

  • Bug – In the new commit-boost job, the "Free up disk space" step uses:

    uses: $/.github/actions/free-disk

    The $ is invalid. It should be ./.github/actions/free-disk (or an explicit action reference). As written, the workflow will fail at parse time or when attempting to resolve the action.

  • Potential version issue – The steps use actions/checkout@v6.1.0 and actions/upload-artifact@v7.0.2. As of current GitHub Actions releases, checkout is at v4 and upload-artifact at v4 (v3 also exists). Versions v6 and v7 do not exist. Unless this repository uses a private fork or these are future versions, these pins will cause the job to fail. Verify and use available versions.

  • Missing dependency – The "Point the args at the image under test" step uses yq -i. The commit-boost job runs on ubuntu-latest, which does not have yq pre-installed by default. The assertoor job may install it elsewhere, but that is not shown in the diff. If yq is not already present in the runner environment (e.g., a custom runner), the step will fail. Ensure yq is installed before use, or use a pre-built action (e.g., mikefarah/yq).


tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh

  • The script correctly uses set -euo pipefail, handles command failures with || return 0 / || true where appropriate, and avoids command injection by quoting all variables.
  • The value_of function uses int(sum) to truncate counter values to integers, which is safe for the >= 1 checks. The comment about %.0f and exponent notation is accurate.
  • The initial metrics-port check is a good idea to fail fast on a setup problem rather than waiting for a timeout.
  • No correctness or security problems found in the script.

tooling/kurtosis-assertoor/commit_boost_params.yaml

  • The configuration appears correct: participant ordering matches the hardcoded service names in check-builder-blocks.sh, the mev_params pin images by digest, and the commit_boost_config template uses the expected variables ({{ .Timestamp }}, {{ .Port }}, {{ .Relays }}).
  • The chain_id is correctly provided as a hex string ("0x301824"), as noted in the comment.
  • No issues found.

tooling/kurtosis-assertoor/README.md

  • Documentation is clear and detailed. No issues.

Overall: The tooling addition is well-structured and the check script is robust. The primary blocker is the invalid $ in the workflow’s uses field, and the action version pins need to be verified. The missing yq dependency may also cause runtime failure.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The Commit-Boost wiring is careful (digest pins, fail-fast on bad service names, pipefail on kurtosis run, unmet conditions rewritten to failed.txt). Two bugs will make the new job fail before it tests the builder path.

  1. yq is not installed in the commit-boost job. Point the args at the image under test runs yq -i, but that job only installs kurtosis. ubuntu-latest does not ship yq. The step fails before the devnet starts. Install the same yq the assertoor job uses, before that step.

  2. Metrics scrapes drop the URL scheme, which current curl rejects. metrics_address strips http:// and scrape calls curl with host:port/metrics:

metrics_address() {
    local addr
    addr=$(kurtosis port print "$ENCLAVE" "$1" metrics) || return
    addr="${addr#*://}"
    printf '%s\n' "${addr%%/*}"
}
...
    curl -sf --max-time 10 "$addr/metrics" 2>/dev/null || true

Curl's parser treats the colon as a scheme (Protocol "127.0.0.1" not supported / URL rejected). The port check still passes, every scrape is empty, and the loop reports "service did not answer" until the 40 minute deadline. Use http://${addr}/metrics. A lenient local curl (macOS) can hide this; GitHub's Ubuntu curl will not.

Also worth fixing while you are there:

  • 40 minute wait vs the no-finality window is tight. README says the beacon node skips the builder from epoch 4 until first finalization (mainnet preset: 32×12 s). Registrations can pass in an epoch; payload_source{source="builder"} and both blinded_blocks_proposed_total series cannot. If finality slips past ~epoch 5, this flakes, and kurtosis startup plus 40 minutes sits close to the job's timeout-minutes: 90. Confirm against a real run, or start the clock only after finality and give the proposer slots more room.
  • value_of uses $NF. A Prometheus timestamp suffix would be treated as the sample and a >= 1 check would pass on the timestamp. These series are counters, so take the value field (second token), not the last.

Automated review by Grok (xAI) · grok-4.7 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness or security issues found. The workflow dispatch conditions correctly handle assertoor, commit-boost, and both; the package revision and container images are pinned; and the polling script handles scrape failures, setup errors, and timeouts carefully.

One non-blocking diagnostics improvement:

  • tooling/kurtosis-assertoor/scripts/check-builder-blocks.sh:111-117 — dump_metrics() truncates each output file before a fresh scrape. If a service becomes unreachable between the final evaluation and the EXIT trap, previously collected metrics are replaced with an empty file, potentially removing useful failure evidence. Consider writing to a temporary file and replacing the artifact only after a non-empty scrape, or preserving the final snapshot alongside the exit-time scrape.

Otherwise, the changes look well documented and appropriately isolated from the existing Assertoor battery.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@MegaRedHand
MegaRedHand marked this pull request as draft November 20, 2026 13:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant