failed to consensus: unable to read new message from the message queue

[xingxinglian]

[Subject of the issue]

Description

I deploy polygon-edge for 3 validator nodes, use pos consensus.
after run more than 20 hours, the system can't generate blocks. after analysis the log, I found the three node all have something wrong with the "unable to read new message from the message queue: timeout expired=5m0s". then I restart the proposal validator node, the system run ok.

Your environment

ubuntu 22.04
polygon edge develop branch

Steps to reproduce

run 20 hours, all nodes in the same machine

Expected behaviour

Is there any optimize for the system self-cure replace restart nodes ?

Actual behaviour

the system can't generate blocks

Logs

2022-06-01T12:21:21.232Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:28:40.733Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:33:41.151Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:38:41.436Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:43:41.703Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:48:41.956Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:53:22.251Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAkuV6dvZgwUhwd9f7QmBh2Yyd1b2WgweiJ22RcSLqstkzV
2022-06-01T12:53:27.815Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAkuV6dvZgwUhwd9f7QmBh2Yyd1b2WgweiJ22RcSLqstkzV
2022-06-01T12:53:42.381Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=5m0s
2022-06-01T12:53:43.033Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-01T12:53:43.033Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=50794 round=73
2022-06-01T12:53:43.033Z [INFO]  polygon.consensus.ibft: current snapshot: validators=3
2022-06-01T12:53:43.033Z [INFO]  polygon.consensus.ibft.acceptState: we are the proposer: block=50794
2022-06-01T12:53:46.812Z [INFO]  polygon.consensus.ibft: state change: new=ValidateState
2022-06-01T12:53:47.433Z [INFO]  polygon.consensus.ibft: state change: new=CommitState
2022-06-01T12:53:47.433Z [INFO]  polygon.blockchain: write block: num=50793 parent=0x3f1d5c6a0ac597bc55c70d9bb3fb3d085e3e91880b919dfd06cd674a281ad3a1
2022-06-01T12:53:47.435Z [INFO]  polygon.blockchain: new block: number=50793 hash=0x15c9ceb72381fd0c62e5fc44fb82525a843509adf9a4dae0f3871240b61282fd txns=0 generation_time_in_seconds=2
2022-06-01T12:53:47.435Z [INFO]  polygon.consensus.ibft: block committed: sequence=50794 hash=0x15c9ceb72381fd0c62e5fc44fb82525a843509adf9a4dae0f3871240b61282fd validators=3 rounds=73 committed=3
2022-06-01T12:53:47.439Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-01T12:53:47.439Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=50794 round=1
2022-06-01T12:53:47.439Z [INFO]  polygon.consensus.ibft: current snapshot: validators=3
2022-06-01T12:53:47.439Z [INFO]  polygon.consensus.ibft.acceptState: we are the proposer: block=50794
2022-06-01T12:53:47.440Z [INFO]  polygon.consensus.ibft: executed txns: failed =0 successful=0 remaining in pool=0
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAkuV6dvZgwUhwd9f7QmBh2Yyd1b2WgweiJ22RcSLqstkzV
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAm2xjh3zJ3vc2gEZRQtWp9V6DYZKRUVxKWzUVJoXVAKQjN
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAmBma6STKWCwVi3znTAYhVDv9d4dv3uFYGgtxaaoQzsTER
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAm6qozetaiNQzr7PVC2TgVpSNbF3NeYYwsAPaNp4TdnPoa
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAm8cLdMikNtX2hAqUKvqkeBHdrLCJ6y9SciJD8u2BdJMYm
2022-06-01T12:53:47.458Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAmSF2bywVXVNxsmzoQWsHX17EeEbMmFWUCYg4VZiiPvQc8
2022-06-01T12:53:47.457Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAmDHxTENhfnv2dYbMDKSKBEXzFjwpm4H15AxU4oUQunxS7
2022-06-01T12:53:47.458Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAm5CCs3jedQMXces8pRcdCAwNGAd6XhVknoc4gdE4QYLJp
2022-06-01T12:53:47.458Z [INFO]  polygon.network: Peer disconnected: id=16Uiu2HAmPBCAzF6H66VnnpzneM6FBj1S1q7qoo1RNCwDzbcN6xG8
2022-06-01T12:53:47.723Z [INFO]  polygon: Data dir: path=/data/etm3test/data/node7
2022-06-01T12:53:47.776Z [INFO]  polygon.blockchain: Current header: hash=0x15c9ceb72381fd0c62e5fc44fb82525a843509adf9a4dae0f3871240b61282fd number=50793
2022-06-01T12:53:47.776Z [INFO]  polygon.blockchain: genesis: hash=0x3dab7cca87683489491589093774adfd59cf15a441359a32655af47895cce4b5
2022-06-01T12:53:48.390Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:30002
2022-06-01T12:53:48.566Z [INFO]  polygon.consensus.ibft: validator key: addr=0xb1bb5DE2EA0F96ebeCF7f2d3A31E5f2B787E46A9
2022-06-01T12:53:48.566Z [INFO]  polygon: GRPC server running: addr=0.0.0.0:30000
2022-06-01T12:53:48.566Z [INFO]  polygon.network: LibP2P server running: addr=/ip4/172.93.0.213/tcp/30001/p2p/16Uiu2HAmK2QW6pecRJapfDY441scfT6gR4sg9wUgoZrQnoCq52wZ
2022-06-01T12:53:48.566Z [INFO]  polygon.consensus.ibft: state change: new=SyncState
2022-06-01T12:53:48.566Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-01T12:53:48.566Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=50794 round=1

Proposed solution

maybe in the case , the validators should step in sync state

[zivkovicmilos]

Hey @xingxinglian,

Which commit hash are you running on from the develop branch?

[xingxinglian]

@zivkovicmilos I use the commit 8b85d68 , May 20

[zivkovicmilos]

@xingxinglian

Got it, are you able to reproduce this problem with the latest commit?

We've added mechanisms in place that will prevent this kind of situation from occurring:

06dc155

[xingxinglian]

@zivkovicmilos got it. I will pull the newest code to try it. And I want to know to what's mechanisms to prevent; I already read the new code , but can't get how to prevent.

[zivkovicmilos]

Hey @xingxinglian,

There are several things that prevent it, actually.

One of the most obvious ones is that we've added improved block verification that will not cause your network to get confused when irregular blocks are proposed. This has been resolved and is live on the develop branch.

The other major thing that caused this sort of behavior is bad unlocking mechanisms that are in place in the current IBFT implementation. We've since resolved this in an open PR.

Let me know if you still experience these problems after you've pulled the latest develop changes 🙏

Also, please make sure your nodes are actually online and connected.

[xingxinglian]

@zivkovicmilos thanks, I already pull code and run online. If still occur the problem , I'll tell you.

[spunkyshooter]

@zivkovicmilos Facing same issue I tried with latest release (v0.4.1) which has commit 06dc155 and even tried to setup freshly.

Here are my working logs for TESTING:

export GOPROXY=direct
go install github.com/0xPolygon/polygon-edge@v0.4.1 #earlier tried with @develop, had the same issue

$ polygon-edge secrets init --data-dir test-chain-1

[SECRETS INIT]
Public key (address) = 0x27cA47569ab07408dd17Ef4C86D3aD7552f928DE
Node ID              = 16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd


$ polygon-edge secrets init --data-dir test-chain-2

[SECRETS INIT]
Public key (address) = 0xD3FE2496eA30862303B30726ACFD69F96B9011F7
Node ID              = 16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay


$ polygon-edge secrets init --data-dir test-chain-3

[SECRETS INIT]
Public key (address) = 0xCD29703Cae7096ddDd3FfB1592ED103b7939F8e6
Node ID              = 16Uiu2HAkzQHFomADnWfhJTBV1RnECCTTqJiqawixjqEQcK2SUpY1


$ polygon-edge secrets init --data-dir test-chain-4

[SECRETS INIT]
Public key (address) = 0xa1523656555B1c9F833B789764864E56aF08A631
Node ID              = 16Uiu2HAmNK1HC8cVziUWd7sjt3rmxGPHh1iyPUdmhXxSvvC8dunS

$ polygon-edge genesis --consensus ibft --ibft-validators-prefix-path test-chain- --bootnode /ip4/127.0.0.1/tcp/10001/p2p/16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd --bootnode /ip4/127.0.0.1/tcp/20001/p2p/16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay 


$ polygon-edge server --data-dir ./test-chain-1 --chain genesis.json --grpc-address :10000 --libp2p :10001 --jsonrpc :10002 --seal
$ polygon-edge server --data-dir ./test-chain-2 --chain genesis.json --grpc-address :20000 --libp2p :20001 --jsonrpc :20002 --seal


Logs:

NODE 1:

$ polygon-edge server --data-dir ./test-chain-1 --chain genesis.json --grpc-address :10000 --libp2p :10001 --jsonrpc :10002 --seal

2022-06-09T05:08:17.495Z [INFO]  polygon: Data dir: path=./test-chain-1
2022-06-09T05:08:17.528Z [INFO]  polygon.blockchain: genesis: hash=0x200fa3735f1ab4101ffbd00fb246f6af97fd8432cc54cdae0239316e8a29bd1d
2022-06-09T05:08:17.528Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:10002
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft: validator key: addr=0x27cA47569ab07408dd17Ef4C86D3aD7552f928DE
2022-06-09T05:08:17.529Z [INFO]  polygon: GRPC server running: addr=127.0.0.1:10000
2022-06-09T05:08:17.529Z [INFO]  polygon.network: LibP2P server running: addr=/ip4/127.0.0.1/tcp/10001/p2p/16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd
2022-06-09T05:08:17.529Z [INFO]  polygon.network: Omitting bootnode with same ID as host: id=16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft: state change: new=SyncState
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=1 round=1
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft.acceptState: we are the proposer: block=1
2022-06-09T05:08:17.529Z [INFO]  polygon.consensus.ibft: executed txns: failed =0 successful=0 remaining in pool=0
2022-06-09T05:08:17.530Z [INFO]  polygon.consensus.ibft: build block: number=1 txns=0
2022-06-09T05:08:17.532Z [INFO]  polygon.consensus.ibft: state change: new=ValidateState
2022-06-09T05:08:27.533Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=10s
2022-06-09T05:08:27.533Z [INFO]  polygon.consensus.ibft: state change: new=RoundChangeState
2022-06-09T05:08:39.535Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=12s
2022-06-09T05:08:53.536Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=14s
2022-06-09T05:09:11.537Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=18s
2022-06-09T05:09:17.530Z [ERROR] polygon.network.discovery: Unable to execute bootnode peer discovery, %w: EXTRA_VALUE_AT_END="unable to create new discovery client connection, peer disconnected before the discovery client was initialized"
2022-06-09T05:09:35.118Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay
2022-06-09T05:09:37.539Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=26s
2022-06-09T05:10:19.540Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=42s
2022-06-09T05:11:33.540Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=1m14s


NODE 2:

$ polygon-edge server --data-dir ./test-chain-2 --chain genesis.json --grpc-address :20000 --libp2p :20001 --jsonrpc :20002 --seal

2022-06-09T05:09:35.077Z [INFO]  polygon: Data dir: path=./test-chain-2
2022-06-09T05:09:35.113Z [INFO]  polygon.blockchain: genesis: hash=0x200fa3735f1ab4101ffbd00fb246f6af97fd8432cc54cdae0239316e8a29bd1d
2022-06-09T05:09:35.114Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:20002
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft: validator key: addr=0xD3FE2496eA30862303B30726ACFD69F96B9011F7
2022-06-09T05:09:35.114Z [INFO]  polygon: GRPC server running: addr=127.0.0.1:20000
2022-06-09T05:09:35.114Z [INFO]  polygon.network: LibP2P server running: addr=/ip4/127.0.0.1/tcp/20001/p2p/16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft: state change: new=SyncState
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=1 round=1
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
2022-06-09T05:09:35.114Z [INFO]  polygon.consensus.ibft: proposer calculated: proposer=0x27cA47569ab07408dd17Ef4C86D3aD7552f928DE block=1
2022-06-09T05:09:35.114Z [INFO]  polygon.network: Omitting bootnode with same ID as host: id=16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay
2022-06-09T05:09:35.118Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd
2022-06-09T05:09:45.114Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=10s
2022-06-09T05:09:45.114Z [INFO]  polygon.consensus.ibft: state change: new=RoundChangeState
2022-06-09T05:09:57.116Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=12s
2022-06-09T05:10:11.117Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=14s
2022-06-09T05:10:37.542Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=18s
2022-06-09T05:11:03.543Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=26s
2022-06-09T05:12:15.542Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=42s
2022-06-09T05:13:29.544Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=1m14s

[ZeljkoBenovic]

Hi @spunkyshooter ,

Are you running only 2 nodes out of 4 ?
Can we get the logs from all nodes ?

Thanks

[spunkyshooter]

Yes, I ran only 2 out of 4 nodes and noticed the issue.
I tried now with all 4 nodes, seems to be working fine after some wait time. Thanks

$ polygon-edge server --data-dir ./test-chain-1 --chain genesis.json --grpc-address :10000 --libp2p :10001 --jsonrpc :10002 --seal

2022-06-09T10:16:36.917Z [INFO]  polygon: Data dir: path=./test-chain-1
2022-06-09T10:16:36.954Z [INFO]  polygon.blockchain: Current header: hash=0x200fa3735f1ab4101ffbd00fb246f6af97fd8432cc54cdae0239316e8a29bd1d number=0
2022-06-09T10:16:36.954Z [INFO]  polygon.blockchain: genesis: hash=0x200fa3735f1ab4101ffbd00fb246f6af97fd8432cc54cdae0239316e8a29bd1d
2022-06-09T10:16:36.955Z [INFO]  polygon.jsonrpc: http server started: addr=0.0.0.0:10002
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: validator key: addr=0x27cA47569ab07408dd17Ef4C86D3aD7552f928DE
2022-06-09T10:16:36.955Z [INFO]  polygon: GRPC server running: addr=127.0.0.1:10000
2022-06-09T10:16:36.955Z [INFO]  polygon.network: LibP2P server running: addr=/ip4/127.0.0.1/tcp/10001/p2p/16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: state change: new=SyncState
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=1 round=1
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft.acceptState: we are the proposer: block=1
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: executed txns: failed =0 successful=0 remaining in pool=0
2022-06-09T10:16:36.955Z [INFO]  polygon.network: Omitting bootnode with same ID as host: id=16Uiu2HAm8phRNSM2q5ocJGhN6GLbtAbkcHj5foCVBvco76HqesLd
2022-06-09T10:16:36.955Z [INFO]  polygon.consensus.ibft: build block: number=1 txns=0
2022-06-09T10:16:36.957Z [INFO]  polygon.consensus.ibft: state change: new=ValidateState
2022-06-09T10:16:46.958Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=10s
2022-06-09T10:16:46.958Z [INFO]  polygon.consensus.ibft: state change: new=RoundChangeState
2022-06-09T10:16:55.247Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAkxDrmdxg8s5fUy47eDX6RRoAZhZzyVFCqzsFd2eurwNay
2022-06-09T10:16:58.959Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=12s
2022-06-09T10:17:12.961Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=14s
2022-06-09T10:17:30.963Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=18s
2022-06-09T10:17:39.497Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAkzQHFomADnWfhJTBV1RnECCTTqJiqawixjqEQcK2SUpY1
2022-06-09T10:17:56.964Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=26s
2022-06-09T10:18:20.209Z [INFO]  polygon.network: Peer connected: id=16Uiu2HAmNK1HC8cVziUWd7sjt3rmxGPHh1iyPUdmhXxSvvC8dunS
2022-06-09T10:18:38.965Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=42s
2022-06-09T10:19:52.967Z [INFO]  polygon.consensus.ibft: unable to read new message from the message queue: timeout expired=1m14s
2022-06-09T10:19:52.967Z [INFO]  polygon.consensus.ibft: state change: new=SyncState
2022-06-09T10:19:52.969Z [INFO]  polygon.blockchain: write block: num=1 parent=0x200fa3735f1ab4101ffbd00fb246f6af97fd8432cc54cdae0239316e8a29bd1d
2022-06-09T10:19:52.969Z [INFO]  polygon.blockchain: new block: number=1 hash=0x117756ab789a5ac7965582bdbecf025293f519238cfa81a17c8ba8933a142d16 txns=0 generation_time_in_seconds=1654769962
2022-06-09T10:19:52.971Z [INFO]  polygon.blockchain: write block: num=2 parent=0x117756ab789a5ac7965582bdbecf025293f519238cfa81a17c8ba8933a142d16
2022-06-09T10:19:52.971Z [INFO]  polygon.blockchain: new block: number=2 hash=0x23763477a78b0eb660e0da083fb67906d57c0b235247f69117d567d59fb5ff65 txns=0 generation_time_in_seconds=2
2022-06-09T10:19:52.973Z [INFO]  polygon.blockchain: write block: num=3 parent=0x23763477a78b0eb660e0da083fb67906d57c0b235247f69117d567d59fb5ff65
2022-06-09T10:19:52.973Z [INFO]  polygon.blockchain: new block: number=3 hash=0x0afc76c60fb424f076a303ccda9c3594a01cc366ffb4f6d1b332fb61e6777010 txns=0 generation_time_in_seconds=2
2022-06-09T10:19:52.974Z [INFO]  polygon.blockchain: write block: num=4 parent=0x0afc76c60fb424f076a303ccda9c3594a01cc366ffb4f6d1b332fb61e6777010
2022-06-09T10:19:52.975Z [INFO]  polygon.blockchain: new block: number=4 hash=0xd70d335df3e9903916bac4b68edac68ed994af8389943c511044496bee84417d txns=0 generation_time_in_seconds=10
2022-06-09T10:19:52.976Z [INFO]  polygon.blockchain: write block: num=5 parent=0xd70d335df3e9903916bac4b68edac68ed994af8389943c511044496bee84417d
2022-06-09T10:19:52.976Z [INFO]  polygon.blockchain: new block: number=5 hash=0xe07f8d50176b5041efa89912ac6ba0ba5094d1b1f321bb259bf3d861f7fd42ad txns=0 generation_time_in_seconds=2
2022-06-09T10:19:52.978Z [INFO]  polygon.blockchain: write block: num=6 parent=0xe07f8d50176b5041efa89912ac6ba0ba5094d1b1f321bb259bf3d861f7fd42ad
2022-06-09T10:19:52.978Z [INFO]  polygon.blockchain: new block: number=6 hash=0x8a581bafdd6bb46ff70f4c6e40de21553713f49688ff065ce3d0f64657918f60 txns=0 generation_time_in_seconds=2
2022-06-09T10:19:52.979Z [INFO]  polygon.blockchain: write block: num=7 parent=0x8a581bafdd6bb46ff70f4c6e40de21553713f49688ff065ce3d0f64657918f60
2022-06-09T10:19:52.979Z [INFO]  polygon.blockchain: new block: number=7 hash=0x75f40d561ca1d48b4f3acebfdc07c4deca38ba000ceaefb5d407b16dea4d7cad txns=0 generation_time_in_seconds=10
2022-06-09T10:19:52.981Z [INFO]  polygon.blockchain: write block: num=8 parent=0x75f40d561ca1d48b4f3acebfdc07c4deca38ba000ceaefb5d407b16dea4d7cad
2022-06-09T10:19:52.981Z [INFO]  polygon.blockchain: new block: number=8 hash=0x0056b72309a965d492b61c4f55f10347a58d4940fab08bc3db87ac0666da5124 txns=0 generation_time_in_seconds=2
2022-06-09T10:19:52.981Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState
2022-06-09T10:19:52.981Z [INFO]  polygon.consensus.ibft.acceptState: Accept state: sequence=9 round=1
2022-06-09T10:19:52.981Z [INFO]  polygon.consensus.ibft: current snapshot: validators=4 votes=0
2022-06-09T10:19:52.981Z [INFO]  polygon.consensus.ibft: proposer calculated: proposer=0xa1523656555B1c9F833B789764864E56aF08A631 block=9
2022-06-09T10:19:54.003Z [INFO]  polygon.consensus.ibft: state change: new=ValidateState
2022-06-09T10:19:54.007Z [INFO]  polygon.consensus.ibft: state change: new=CommitState
2022-06-09T10:19:54.008Z [INFO]  polygon.blockchain: write block: num=9 parent=0x0056b72309a965d492b61c4f55f10347a58d4940fab08bc3db87ac0666da5124
2022-06-09T10:19:54.009Z [INFO]  polygon.blockchain: new block: number=9 hash=0xd9e22d6e6bef8559785bd4733ec7de9dcad606de71f4a14d6f1f9857fae0f276 txns=0 generation_time_in_seconds=2
2022-06-09T10:19:54.009Z [INFO]  polygon.consensus.ibft: block committed: sequence=9 hash=0xd9e22d6e6bef8559785bd4733ec7de9dcad606de71f4a14d6f1f9857fae0f276 validators=4 rounds=1 committed=3
2022-06-09T10:19:54.009Z [INFO]  polygon.consensus.ibft: state change: new=AcceptState

[ZeljkoBenovic]

@spunkyshooter Great 🥇

There needs to be at least 4 validators active at minimum in order for the blockchain network to run properly.
If there are less than 4 ( actually there can be 3 validators per IBFT spec ) there will be RoundChangeState in the logs and depending on the number of active validators the chain won't start / will halt .

I'll close this issue for now. Feel free to open a new issue if you face any more problems.