feat(p2p): warn periodically while node has zero connected peers - #25183
Merged
Conversation
This was referenced Aug 11, 2026
A validator could run for hours with a dead libp2p stack - zero peers and zero gossip - without anything in its logs flagging the condition. The failure was only reconstructable after the fact from the absence of activity, rather than from any explicit signal. The peer manager heartbeat now tracks consecutive zero-peer heartbeats. After 3 of them (so startup and transient dips stay quiet) it logs a warning that the node has no connected peers and can neither gossip nor propagate txs, with the zero-peer heartbeat count, the heartbeat interval, time since a peer was last connected, and the cached peer count. The warning then repeats about once a minute, derived from peerCheckIntervalMS, instead of on every heartbeat. When peers come back, a single info log reports connectivity restored and the new peer count. Connected peers are counted via libp2p getPeers(), matching what getP2PConnectivity reports. A peer count gauge already existed (PEER_MANAGER_PEER_COUNT, recorded in discover()), so no metric was added - this change is logs only. Part of A-1701.
spalladino
force-pushed
the
spl/p2p-zero-peers-warning
branch
from
August 12, 2026 01:31
762b15b to
6fd4f4d
Compare
spalladino
changed the base branch from
spl/sendtx-reject-no-peers
to
merge-train/spartan-v5
August 12, 2026 01:31
spalladino
added
port-to-next
fcarreiro
approved these changes
Aug 12, 2026
fcarreiro
reviewed
Aug 12, 2026
| * and then repeat roughly once a minute instead of on every heartbeat. Recovery is logged once. | ||
| */ | ||
| private checkPeerConnectivity() { | ||
| const connectedPeerCount = this.libP2PNode.getPeers().length; |
Contributor
There was a problem hiding this comment.
In a later PR you do this.peerManager.getPeers().filter(peer => peer.status === 'connected').length; and filter them. Why not here?
Contributor
Author
There was a problem hiding this comment.
They count the same set: PeerManager.getPeers() without includePending returns exactly the libp2p connected list with status hard-coded to 'connected', so the filter in the other PR is a no-op. Here we're inside PeerManager itself, so we just read libP2PNode.getPeers() directly, which only returns actively connected peers.
Contributor
Author
There was a problem hiding this comment.
Apparently it's not needed. See response above.
spalladino
added a commit
that referenced
this pull request
Aug 12, 2026
When the p2p client starts while there are blocks to sync, the libp2p service start is deferred to `startServiceIfSynched`, which runs inside the block stream event handler. That handler catches and only logs errors, so a TCP bind failure (e.g. `ERR_NO_VALID_ADDRESSES` after a restart) was swallowed. On top of that, the sync promise awaited by node creation was resolved before the service was started, so startup succeeded regardless and the node kept running with a dead p2p stack and zero peers. The service is now started before the sync promise settles, and a failure logs at fatal level and rejects the stored sync promise (rather than throwing, which the block stream would eat) so node creation fails and the process exits, letting the orchestrator restart it. Part of A-1701. --- Related PRs from the same incident (all independent, all targeting `merge-train/spartan-v5`): #25177 (fail startup when the p2p service fails to start), #25185 (connectivity signal + slasher/proposer/health/sendTx gates), #25183 (periodic zero-peers warning).
spalladino
added a commit
that referenced
this pull request
Aug 14, 2026
…tivity (#25185) Gates p2p-dependent operations on actual gossip connectivity, so a node running with a dead libp2p stack (zero peers) can no longer file false slashing accusations, burn its proposer slots, or silently blackhole txs. Follows an incident where a validator whose TCP listener failed to bind kept running peerless and voted `DATA_WITHHOLDING` against entire committees. One commit per concern (squashed on merge): 1. **`feat(p2p): expose p2p connectivity`** — new `getP2PConnectivity(): { enabled, connectedPeers }` on the p2p service, client, and node RPC API. The dummy (p2p-disabled) implementation reports `enabled: false`, so sandbox and single-node setups are vacuously healthy for every gate below. 2. **`fix(slasher): do not file data-withholding offenses while peerless`** — the data-withholding watcher infers offenses from the *absence* of txs in the local pool, which is invalid evidence without gossip connectivity. It now permanently skips (never backfills) slots probed while p2p is enabled with zero peers — after reconnecting, mined txs may already be evicted from the pool, so a late probe would still false-positive. 3. **`fix(sequencer): skip proposing when node has no connected peers`** — a peerless proposer broadcasts into the void (burns the slot) or, with enough own committee seats, publishes a checkpoint whose data nobody has. It now skips building below `minPeersToPropose` (default 1, `SEQ_MIN_PEERS_TO_PROPOSE`, 0 disables) while keeping L1 duties (votes, prune, invalidation) via the existing fallback. 4. **`feat(node): report per-component health on GET /status`** — `/status` now returns `{ ok, components: { p2p: { healthy, enabled, connectedPeers }, ... } }`. The `p2pHealthMinPeers` floor (`P2P_HEALTH_MIN_PEERS`) defaults to 0 (report-only) so fresh-network bootstrap nodes aren't failed by default; operators opt in. 5. **`fix(node): reject sendTx when node has no peers`** — instead of accepting a tx and gossiping it to nobody, the node rejects immediately so callers can retry against a healthy node. 6. **`test: keep single-node mock-gossip e2e working`** — `registerPhantomGossipPeer` test helper so the one single-node e2e test on the in-memory mock gossip bus presents one connected peer to the gates. 7. **`fix(slasher): do not file data-withholding offenses when p2p is disabled`** — a node with no p2p stack at all sees strictly less than a peerless one, so it must not slash either. The runtime gate drops `enabled` (any node with zero peers skips), and the watcher isn't constructed when `p2pEnabled` is false. Unlike the other gates, this one infers guilt from *absence*, which is never valid without gossip; the propose/`/status`/`sendTx` gates keep treating `enabled: false` as vacuously fine because they gate the ability to *act*, which a deliberately p2p-less node has by design. Strongly suggest to **review commit by commit**. Related PRs (independent, same incident): #25177 (fail node startup when the p2p service fails to start), #25183 (periodic zero-peers warning). Part of A-1701.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A validator could run for hours with a dead libp2p stack - zero peers and zero
gossip - without anything in its logs flagging the condition. The failure was
only reconstructable after the fact from the absence of activity, rather than
from any explicit signal.
The peer manager heartbeat now tracks consecutive zero-peer heartbeats. After 3
of them (so startup and transient dips stay quiet) it logs a warning that the
node has no connected peers and can neither gossip nor propagate txs, with the
zero-peer heartbeat count, the heartbeat interval, time since a peer was last
connected, and the cached peer count. The warning then repeats about once a
minute, derived from peerCheckIntervalMS, instead of on every heartbeat. When
peers come back, a single info log reports connectivity restored and the new
peer count.
Connected peers are counted via libp2p getPeers(), matching what
getP2PConnectivity reports. A peer count gauge already existed
(PEER_MANAGER_PEER_COUNT, recorded in discover()), so no metric was added -
this change is logs only.
Part of A-1701.
Related PRs from the same incident (all independent, all targeting
merge-train/spartan-v5): #25177 (fail startup when the p2p service fails to start), #25185 (connectivity signal + slasher/proposer/health/sendTx gates), #25183 (periodic zero-peers warning).