Consensus Reactor

Consensus Reactor defines a reactor for the consensus service. It contains the ConsensusState service that manages the state of the Tendermint consensus internal state machine. When Consensus Reactor is started, it starts Broadcast Routine which starts ConsensusState service. Furthermore, for each peer that is added to the Consensus Reactor, it creates (and manages) the known peer state (that is used extensively in gossip routines) and starts the following three routines for the peer p: Gossip Data Routine, Gossip Votes Routine and QueryMaj23Routine. Finally, Consensus Reactor is responsible for decoding messages received from a peer and for adequate processing of the message depending on its type and content. The processing normally consists of updating the known peer state and for some messages (ProposalMessage, BlockPartMessage and VoteMessage) also forwarding message to ConsensusState module for further processing. In the following text we specify the core functionality of those separate unit of executions that are part of the Consensus Reactor.

ConsensusState service

Consensus State handles execution of the Tendermint BFT consensus algorithm. It processes votes and proposals, and upon reaching agreement, commits blocks to the chain and executes them against the application. The internal state machine receives input from peers, the internal validator and from a timer.

Inside Consensus State we have the following units of execution: Timeout Ticker and Receive Routine. Timeout Ticker is a timer that schedules timeouts conditional on the height/round/step that are processed by the Receive Routine.

Receive Routine of the ConsensusState service

Receive Routine of the ConsensusState handles messages which may cause internal consensus state transitions. It is the only routine that updates RoundState that contains internal consensus state. Updates (state transitions) happen on timeouts, complete proposals, and 2/3 majorities. It receives messages from peers, internal validators and from Timeout Ticker and invokes the corresponding handlers, potentially updating the RoundState. The details of the protocol (together with formal proofs of correctness) implemented by the Receive Routine are discussed in separate document. For understanding of this document it is sufficient to understand that the Receive Routine manages and updates RoundState data structure that is then extensively used by the gossip routines to determine what information should be sent to peer processes.

Round State

RoundState defines the internal consensus state. It contains height, round, round step, a current validator set, a proposal and proposal block for the current round, locked round and block (if some block is being locked), set of received votes and last commit and last validators set.

  1. type RoundState struct {
  2.     Height             int64
  3.     Round              int
  4.     Step               RoundStepType
  5.     Validators         ValidatorSet
  6.     Proposal           Proposal
  7.     ProposalBlock      Block
  8.     ProposalBlockParts PartSet
  9.     LockedRound        int
  10.     LockedBlock        Block
  11.     LockedBlockParts   PartSet
  12.     Votes              HeightVoteSet
  13.     LastCommit         VoteSet
  14.     LastValidators     ValidatorSet
  15. }

Internally, consensus will run as a state machine with the following states:

  • RoundStepNewHeight
  • RoundStepNewRound
  • RoundStepPropose
  • RoundStepProposeWait
  • RoundStepPrevote
  • RoundStepPrevoteWait
  • RoundStepPrecommit
  • RoundStepPrecommitWait
  • RoundStepCommit

Peer Round State

Peer round state contains the known state of a peer. It is being updated by the Receive routine of Consensus Reactor and by the gossip routines upon sending a message to the peer.

  1. type PeerRoundState struct {
  2.     Height                   int64               // Height peer is at
  3.     Round                    int                 // Round peer is at, -1 if unknown.
  4.     Step                     RoundStepType       // Step peer is at
  5.     Proposal                 bool                // True if peer has proposal for this round
  6.     ProposalBlockPartsHeader PartSetHeader
  7.     ProposalBlockParts       BitArray
  8.     ProposalPOLRound         int                 // Proposal's POL round. -1 if none.
  9.     ProposalPOL              BitArray            // nil until ProposalPOLMessage received.
  10.     Prevotes                 BitArray            // All votes peer has for this round
  11.     Precommits               BitArray            // All precommits peer has for this round
  12.     LastCommitRound          int                 // Round of commit for last height. -1 if none.
  13.     LastCommit               BitArray            // All commit precommits of commit for last height.
  14.     CatchupCommitRound       int                 // Round that we have commit for. Not necessarily unique. -1 if none.
  15.     CatchupCommit            BitArray            // All commit precommits peer has for this height & CatchupCommitRound
  16. }

Receive method of Consensus reactor

The entry point of the Consensus reactor is a receive method. When a message is received from a peer p, normally the peer round state is updated correspondingly, and some messages are passed for further processing, for example to ConsensusState service. We now specify the processing of messages in the receive method of Consensus reactor for each message type. In the following message handler, rs and prs denote RoundState and PeerRoundState, respectively.

NewRoundStepMessage handler

  1. handleMessage(msg):
  2.     if msg is from smaller height/round/step then return
  3.     // Just remember these values.
  4.     prsHeight = prs.Height
  5.     prsRound = prs.Round
  6.     prsCatchupCommitRound = prs.CatchupCommitRound
  7.     prsCatchupCommit = prs.CatchupCommit
  9.     Update prs with values from msg
  10.     if prs.Height or prs.Round has been updated then
  11.         reset Proposal related fields of the peer state
  12.     if prs.Round has been updated and msg.Round == prsCatchupCommitRound then
  13.         prs.Precommits = psCatchupCommit
  14.     if prs.Height has been updated then
  15.         if prsHeight+1 == msg.Height && prsRound == msg.LastCommitRound then
  16.             prs.LastCommitRound = msg.LastCommitRound
  17.             prs.LastCommit = prs.Precommits
  18.         } else {
  19.             prs.LastCommitRound = msg.LastCommitRound
  20.             prs.LastCommit = nil
  21.         }
  22.         Reset prs.CatchupCommitRound and prs.CatchupCommit

NewValidBlockMessage handler

  1. handleMessage(msg):
  2.     if prs.Height != msg.Height then return
  4.     if prs.Round != msg.Round && !msg.IsCommit then return
  6.     prs.ProposalBlockPartsHeader = msg.BlockPartsHeader
  7.     prs.ProposalBlockParts = msg.BlockParts

HasVoteMessage handler

  1. handleMessage(msg):
  2.     if prs.Height == msg.Height then
  3.         prs.setHasVote(msg.Height, msg.Round, msg.Type, msg.Index)

VoteSetMaj23Message handler

  1. handleMessage(msg):
  2.     if prs.Height == msg.Height then
  3.         Record in rs that a peer claim to have ⅔ majority for msg.BlockID
  4.         Send VoteSetBitsMessage showing votes node has for that BlockId

ProposalMessage handler

  1. handleMessage(msg):
  2.     if prs.Height != msg.Height || prs.Round != msg.Round || prs.Proposal then return
  3.     prs.Proposal = true
  4.     if prs.ProposalBlockParts == empty set then // otherwise it is set in NewValidBlockMessage handler
  5.       prs.ProposalBlockPartsHeader = msg.BlockPartsHeader
  6.     prs.ProposalPOLRound = msg.POLRound
  7.     prs.ProposalPOL = nil
  8.     Send msg through internal peerMsgQueue to ConsensusState service

ProposalPOLMessage handler

  1. handleMessage(msg):
  2.     if prs.Height != msg.Height or prs.ProposalPOLRound != msg.ProposalPOLRound then return
  3.     prs.ProposalPOL = msg.ProposalPOL

BlockPartMessage handler

  1. handleMessage(msg):
  2.     if prs.Height != msg.Height || prs.Round != msg.Round then return
  3.     Record in prs that peer has block part msg.Part.Index
  4.     Send msg trough internal peerMsgQueue to ConsensusState service

VoteMessage handler

  1. handleMessage(msg):
  2.     Record in prs that a peer knows vote with index msg.vote.ValidatorIndex for particular height and round
  3.     Send msg trough internal peerMsgQueue to ConsensusState service

VoteSetBitsMessage handler

  1. handleMessage(msg):
  2.     Update prs for the bit-array of votes peer claims to have for the msg.BlockID

Gossip Data Routine

It is used to send the following messages to the peer: BlockPartMessage, ProposalMessage and ProposalPOLMessage on the DataChannel. The gossip data routine is based on the local RoundState (rs) and the known PeerRoundState (prs). The routine repeats forever the logic shown below:

  1. 1a) if rs.ProposalBlockPartsHeader == prs.ProposalBlockPartsHeader and the peer does not have all the proposal parts then
  2.         Part = pick a random proposal block part the peer does not have
  3.         Send BlockPartMessage(rs.Height, rs.Round, Part) to the peer on the DataChannel
  4.         if send returns true, record that the peer knows the corresponding block Part
  5.         Continue
  7. 1b) if (0 < prs.Height) and (prs.Height < rs.Height) then
  8.         help peer catch up using gossipDataForCatchup function
  9.         Continue
  11. 1c) if (rs.Height != prs.Height) or (rs.Round != prs.Round) then
  12.         Sleep PeerGossipSleepDuration
  13.         Continue
  15. //  at this point rs.Height == prs.Height and rs.Round == prs.Round
  16. 1d) if (rs.Proposal != nil and !prs.Proposal) then
  17.         Send ProposalMessage(rs.Proposal) to the peer
  18.         if send returns true, record that the peer knows Proposal
  19.         if 0 <= rs.Proposal.POLRound then
  20.         polRound = rs.Proposal.POLRound
  21.         prevotesBitArray = rs.Votes.Prevotes(polRound).BitArray()
  22.         Send ProposalPOLMessage(rs.Height, polRound, prevotesBitArray)
  23.         Continue
  25. 2)  Sleep PeerGossipSleepDuration

Gossip Data For Catchup

This function is responsible for helping peer catch up if it is at the smaller height (prs.Height < rs.Height). The function executes the following logic:

  1. if peer does not have all block parts for prs.ProposalBlockPart then
  2.     blockMeta =  Load Block Metadata for height prs.Height from blockStore
  3.     if (!blockMeta.BlockID.PartsHeader == prs.ProposalBlockPartsHeader) then
  4.         Sleep PeerGossipSleepDuration
  5.     return
  6.     Part = pick a random proposal block part the peer does not have
  7.     Send BlockPartMessage(prs.Height, prs.Round, Part) to the peer on the DataChannel
  8.     if send returns true, record that the peer knows the corresponding block Part
  9.     return
  10. else Sleep PeerGossipSleepDuration

Gossip Votes Routine

It is used to send the following message: VoteMessage on the VoteChannel. The gossip votes routine is based on the local RoundState (rs) and the known PeerRoundState (prs). The routine repeats forever the logic shown below:

  1. 1a) if rs.Height == prs.Height then
  2.         if prs.Step == RoundStepNewHeight then
  3.             vote = random vote from rs.LastCommit the peer does not have
  4.             Send VoteMessage(vote) to the peer
  5.             if send returns true, continue
  7.         if prs.Step <= RoundStepPrevote and prs.Round != -1 and prs.Round <= rs.Round then
  8.             Prevotes = rs.Votes.Prevotes(prs.Round)
  9.             vote = random vote from Prevotes the peer does not have
  10.             Send VoteMessage(vote) to the peer
  11.             if send returns true, continue
  13.         if prs.Step <= RoundStepPrecommit and prs.Round != -1 and prs.Round <= rs.Round then
  14.              Precommits = rs.Votes.Precommits(prs.Round)
  15.             vote = random vote from Precommits the peer does not have
  16.             Send VoteMessage(vote) to the peer
  17.             if send returns true, continue
  19.         if prs.ProposalPOLRound != -1 then
  20.             PolPrevotes = rs.Votes.Prevotes(prs.ProposalPOLRound)
  21.             vote = random vote from PolPrevotes the peer does not have
  22.             Send VoteMessage(vote) to the peer
  23.             if send returns true, continue
  25. 1b)  if prs.Height != 0 and rs.Height == prs.Height+1 then
  26.         vote = random vote from rs.LastCommit peer does not have
  27.         Send VoteMessage(vote) to the peer
  28.         if send returns true, continue
  30. 1c)  if prs.Height != 0 and rs.Height >= prs.Height+2 then
  31.         Commit = get commit from BlockStore for prs.Height
  32.         vote = random vote from Commit the peer does not have
  33.         Send VoteMessage(vote) to the peer
  34.         if send returns true, continue
  36. 2)   Sleep PeerGossipSleepDuration

QueryMaj23Routine

It is used to send the following message: VoteSetMaj23Message. VoteSetMaj23Message is sent to indicate that a given BlockID has seen +2/3 votes. This routine is based on the local RoundState (rs) and the known PeerRoundState (prs). The routine repeats forever the logic shown below.

  1. 1a) if rs.Height == prs.Height then
  2.         Prevotes = rs.Votes.Prevotes(prs.Round)
  3.         if there is a ⅔ majority for some blockId in Prevotes then
  4.         m = VoteSetMaj23Message(prs.Height, prs.Round, Prevote, blockId)
  5.         Send m to peer
  6.         Sleep PeerQueryMaj23SleepDuration
  8. 1b) if rs.Height == prs.Height then
  9.         Precommits = rs.Votes.Precommits(prs.Round)
  10.         if there is a ⅔ majority for some blockId in Precommits then
  11.         m = VoteSetMaj23Message(prs.Height,prs.Round,Precommit,blockId)
  12.         Send m to peer
  13.         Sleep PeerQueryMaj23SleepDuration
  15. 1c) if rs.Height == prs.Height and prs.ProposalPOLRound >= 0 then
  16.         Prevotes = rs.Votes.Prevotes(prs.ProposalPOLRound)
  17.         if there is a ⅔ majority for some blockId in Prevotes then
  18.         m = VoteSetMaj23Message(prs.Height,prs.ProposalPOLRound,Prevotes,blockId)
  19.         Send m to peer
  20.         Sleep PeerQueryMaj23SleepDuration
  22. 1d) if prs.CatchupCommitRound != -1 and 0 < prs.Height and
  23.         prs.Height <= blockStore.Height() then
  24.         Commit = LoadCommit(prs.Height)
  25.         m = VoteSetMaj23Message(prs.Height,Commit.Round,Precommit,Commit.blockId)
  26.         Send m to peer
  27.         Sleep PeerQueryMaj23SleepDuration
  29. 2)  Sleep PeerQueryMaj23SleepDuration

Broadcast routine

The Broadcast routine subscribes to an internal event bus to receive new round steps and votes messages, and broadcasts messages to peers upon receiving those events. It broadcasts NewRoundStepMessage or CommitStepMessage upon new round state event. Note that broadcasting these messages does not depend on the PeerRoundState; it is sent on the StateChannel. Upon receiving VoteMessage it broadcasts HasVoteMessage message to its peers on the StateChannel.

Channels

Defines 4 channels: state, data, vote and vote_set_bits. Each channel has SendQueueCapacity and RecvBufferCapacity and RecvMessageCapacity set to maxMsgSize.

Sending incorrectly encoded data will result in stopping the peer.