Repository navigation
ethmonitor: detect stalled newHeads streams and fall back to polling - #226
Conversation
ddfd377 to
f89c4b1
Compare
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Findings from Ada's review of
Suggested fix: when the stream appears stale, fetch the node's latest block over HTTP and only drop the subscription if the node is ahead of the stream. This distinguishes an idle chain from a stalled subscription and follows the deleted TODO. The incident attribution also needs revisiting. In the 19:30–21:10 UTC window on Nov 5, the reviewed node-gateway logs showed no Base (8453) stream errors; the only ethmonitor stream errors were Before node-gateway adopts the release containing this change, either add the HTTP-head comparison or explicitly disable the check with Validation reported by Ada: |
f89c4b1 to
37182e4
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37182e4130
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A subscription can silently serve stale heads: no error arrives on sub.Err() while the node's http api keeps advancing. The listener tracks the chain timestamp of the last received head and, when it exceeds StreamHealthCheckStaleness, fetches the node's head over http before dropping the subscription: only if the node is ahead of the stream, by number (missed heads) or by timestamp (stale head data). A quiet chain is not a stalled stream, so idle chains like Arbitrum Nova keep streaming. Drops count toward StreamingErrNumToSwitchToPolling, and the error count now actually resets after StreamingErrorResetInterval.
37182e4 to
f39d9c8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f39d9c89cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The staleness probe runs synchronously in the loop that receives newHeads, so heads delivered while it is in flight are parked on the channel and invisible to the comparison, while the http response can already include them. An idle chain waking up during a probe would read as a missed head and the healthy subscription got dropped. Drain the pending heads through the same processing as the receive case before comparing.
|
All three addressed in f39d9c8. The check now fetches the node's head over http on a staleness trip and only drops the subscription if the node is ahead by number or timestamp, so idle chains like Nova stay streaming. The counter reset is fixed too (the elapsed-time check was always zero before). Kept it on by default since the http comparison removes the Nova churn case, so node-gateway doesn't need to opt out on upgrade. Say the word if you'd still rather it's off by default and configured per network. |
Summary
A newHeads subscription can silently serve stale heads: no error arrives on the subscription, while the node's HTTP api keeps advancing. This is what left the node-gateway's Base head 23 minutes behind during INC-280 while every endpoint health probe stayed green.
In streaming mode, the listener now tracks the timestamp of the last received head and drops the subscription, counting it toward the existing switch to polling, when it is older than
StreamHealthCheckStaleness(default 60s). Seeded on connect so a stream that never delivers a head is caught too.NewMonitorrejectsStreamHealthCheckStaleness <= StreamHealthCheckInterval.Implements the streamHealthCheck TODO that was already in
listenNewHead.Test Plan
TestMonitorStreamHealthCheckFallsBackToPolling: stalled stream switches to polling, head keeps advancing, alert raisedTestMonitorStreamHealthCheckFreshStreamKeepsStreaming: control, no stall alerts on a live streamTestMonitorStreamHealthCheckRecoversToStreaming: after the stall clears, polling retries streaming and catches back up./ethmonitor/suite passes