Skip to content

ethmonitor: detect stalled newHeads streams and fall back to polling - #226

Merged
taylanpince merged 2 commits into
masterfrom
ethmonitor-stream-health-check
Nov 9, 2026
Merged

taylanpince merged 2 commits into
masterfrom
ethmonitor-stream-health-check

Conversation

@ScreamingHawk

Copy link
Copy Markdown
Contributor

Summary

A newHeads subscription can silently serve stale heads: no error arrives on the subscription, while the node's HTTP api keeps advancing. This is what left the node-gateway's Base head 23 minutes behind during INC-280 while every endpoint health probe stayed green.

In streaming mode, the listener now tracks the timestamp of the last received head and drops the subscription, counting it toward the existing switch to polling, when it is older than StreamHealthCheckStaleness (default 60s). Seeded on connect so a stream that never delivers a head is caught too. NewMonitor rejects StreamHealthCheckStaleness <= StreamHealthCheckInterval.

Implements the streamHealthCheck TODO that was already in listenNewHead.

Test Plan

  • TestMonitorStreamHealthCheckFallsBackToPolling: stalled stream switches to polling, head keeps advancing, alert raised
  • TestMonitorStreamHealthCheckFreshStreamKeepsStreaming: control, no stall alerts on a live stream
  • TestMonitorStreamHealthCheckRecoversToStreaming: after the stall clears, polling retries streaming and catches back up
  • full ./ethmonitor/ suite passes

@ScreamingHawk
ScreamingHawk force-pushed the ethmonitor-stream-health-check branch from ddfd377 to f89c4b1 Compare November 5, 2026 21:19
@ScreamingHawk
ScreamingHawk marked this pull request as ready for review November 5, 2026 21:32
@ScreamingHawk
ScreamingHawk requested a review from a team November 5, 2026 21:32
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T21:06:05.836899Z 120bcd8 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@taylanpince

Copy link
Copy Markdown
Contributor

Findings from Ada's review of f89c4b1:

  • Arbitrum Nova would repeatedly reconnect on a healthy, idle chain. Nova produces blocks only when there is activity. In the sampled public RPC history, 18 of the last 59 block gaps exceeded 2 minutes; the longest was 3412s. The gateway head matched the public head, and Nova had streamingMode: true. With the 60s threshold and the timer seeded on connect, an idle Nova stream would be dropped and alert roughly every 70s. Three drops switch it to polling, with streaming retried after 4 minutes, so the cycle repeats while the chain is quiet.
  • The check is enabled by default. DefaultOptions sets StreamHealthCheckInterval: 10s and StreamHealthCheckStaleness: 60s (ethmonitor/ethmonitor.go:35-36). node-gateway starts from ethmonitor.DefaultOptions (netmon/monitor.go:119) and does not override these fields, so an ethkit upgrade enables it automatically. Callers constructing Options from scratch get it disabled. Other consumers such as relayer and indexer still need checking.
  • An existing counter-reset bug compounds this. streamingErrLastTime is set just before goto reconnect, so the elapsed-time check is effectively always zero. StreamingErrorResetInterval never resets the counter, and drops accumulate even when far apart.

Suggested fix: when the stream appears stale, fetch the node's latest block over HTTP and only drop the subscription if the node is ahead of the stream. This distinguishes an idle chain from a stalled subscription and follows the deleted TODO.

The incident attribution also needs revisiting. In the 19:30–21:10 UTC window on Nov 5, the reviewed node-gateway logs showed no Base (8453) stream errors; the only ethmonitor stream errors were context canceled during shutdown on other chains. The incident evidence points to the monitor failing to keep up, with prefetch resolving it. If fresh heads continue arriving while processing lags, this check would correctly remain quiet, but would not catch that failure. The monitor-head check in node-gateway#334 addresses that case.

Before node-gateway adopts the release containing this change, either add the HTTP-head comparison or explicitly disable the check with StreamHealthCheckInterval: 0 / configure suitable per-network staleness.

Validation reported by Ada: go test -race ./ethmonitor/ passed, including the five new StreamHealthCheck tests.

@ScreamingHawk
ScreamingHawk force-pushed the ethmonitor-stream-health-check branch from f89c4b1 to 37182e4 Compare November 8, 2026 19:01

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37182e4130

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ethmonitor/ethmonitor.go Resolved
Comment thread ethmonitor/ethmonitor.go Resolved
A subscription can silently serve stale heads: no error arrives on
sub.Err() while the node's http api keeps advancing. The listener tracks
the chain timestamp of the last received head and, when it exceeds
StreamHealthCheckStaleness, fetches the node's head over http before
dropping the subscription: only if the node is ahead of the stream, by
number (missed heads) or by timestamp (stale head data). A quiet chain
is not a stalled stream, so idle chains like Arbitrum Nova keep
streaming. Drops count toward StreamingErrNumToSwitchToPolling, and the
error count now actually resets after StreamingErrorResetInterval.
@ScreamingHawk
ScreamingHawk force-pushed the ethmonitor-stream-health-check branch from 37182e4 to f39d9c8 Compare November 8, 2026 20:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f39d9c89cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ethmonitor/ethmonitor.go Resolved
The staleness probe runs synchronously in the loop that receives
newHeads, so heads delivered while it is in flight are parked on the
channel and invisible to the comparison, while the http response can
already include them. An idle chain waking up during a probe would read
as a missed head and the healthy subscription got dropped. Drain the
pending heads through the same processing as the receive case before
comparing.
@ScreamingHawk

Copy link
Copy Markdown
Contributor Author

All three addressed in f39d9c8.

The check now fetches the node's head over http on a staleness trip and only drops the subscription if the node is ahead by number or timestamp, so idle chains like Nova stay streaming. The counter reset is fixed too (the elapsed-time check was always zero before).

Kept it on by default since the http comparison removes the Nova churn case, so node-gateway doesn't need to opt out on upgrade. Say the word if you'd still rather it's off by default and configured per network.

@taylanpince
taylanpince merged commit 7b6bd8c into master Nov 9, 2026
14 checks passed
@taylanpince
taylanpince deleted the ethmonitor-stream-health-check branch November 9, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants