Node shutdown when fetching blocks over RPC

[joepetrowski]

When fetching blocks from the sidecar, a fully synced node sometimes crashes after sending a few thousand blocks.

2020-02-19 20:07:17          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:16          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:15          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:14          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:13          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:12          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:11          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:10          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:09          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:08          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'connection failed'
2020-02-19 20:07:07.479  DEBUG libp2p_tcp  Dropped TCP connection to V4(114.55.243.192:30333)
2020-02-19 20:07:07.318 http.worker00 DEBUG tokio_reactor  dropping I/O source: 1
2020-02-19 20:07:07.319 async-std/net DEBUG libp2p_tcp  Dropped TCP connection to V4(18.191.164.60:30333)
2020-02-19 20:07:07.318 http.worker00 DEBUG tokio_reactor  dropping I/O source: 0
2020-02-19 20:07:07.318 http.worker00 DEBUG tokio_reactor::registration  scheduling Read for: 0
2020-02-19 20:07:07.319 http.worker20 DEBUG tokio_reactor  dropping I/O source: 0
2020-02-19 20:07:07          API-WS: disconnected from ws://127.0.0.1:9944 code: '1006' reason: 'Connection dropped by remote peer.'
2020-02-19 20:07:07.317 http.worker20 DEBUG tokio_reactor::registration  scheduling Read for: 0
2020-02-19 20:07:07.317 http.worker30 DEBUG tokio_reactor::registration  scheduling Read for: 0
2020-02-19 20:07:07.317 http.worker10 DEBUG tokio_reactor::registration  scheduling Read for: 0
2020-02-19 20:07:07.317 http.worker10 DEBUG tokio_reactor  dropping I/O source: 0
2020-02-19 20:07:07.317 main DEBUG service  Substrate service shutdown
2020-02-19 20:07:07.318 http.worker30 DEBUG tokio_reactor  dropping I/O source: 1
2020-02-19 20:07:07.317 http.worker30 DEBUG tokio_reactor  dropping I/O source: 0
2020-02-19 20:07:07.310 main-tokio- DEBUG libp2p_core::upgrade::apply  Successfully applied negotiated protocol
2020-02-19 20:07:07.310 main-tokio- DEBUG multistream_select::dialer_select  Dialer: Proposed protocol: /yamux/1.0.0
2020-02-19 20:07:07.308 main-tokio- DEBUG afg  Completed round 3856, state = State { prevote_ghost: Some((0x8fde89e3989f88b5b11b51f4fab805fa75e326e5946baae88d89f06ac75820be, 1111542)), finalized: Some((0x8fde89e3989f88b5b11b51f4fab805fa75e326e5946baae88d89f06ac75820be, 1111542)), estimate: Some((0x8fde89e3989f88b5b11b51f4fab805fa75e326e5946baae88d89f06ac75820be, 1111542)), completable: true }, step = None
2020-02-19 20:07:07.308 main-tokio- DEBUG afg  Round 3856: prevotes: 142/107/160 weight, 142/160 actual

[bkchr]

Where is the panic? The log does not contain any information about a panic?

[NikVolf]

Where is the panic? The log does not contain any information about a panic?

2020-02-19 20:07:07.317 main DEBUG service Substrate service shutdown

Not panic though, but still

[bkchr]

I saw this, but there still could have been a panic. (Tokio catches panics)