Node crashes randomly

[avinashbo]

System information

Geth version: 1.1.2
OS & Version: Linux/k8s
Commit hash : (if develop)

Expected behaviour

Node runs normally and remains stable as long as there is no resource crunch

Actual behaviour

A syncing node randomly crashes with the log below

INFO [10-27|16:01:16.364] Unindexed transactions                   blocks=1   txs=315   tail=9,789,321 elapsed=6.422ms
INFO [10-27|16:01:22.456] Imported new chain segment               blocks=1   txs=720   mgas=85.596   elapsed=6.097s      mgasps=14.038   number=12,139,321 hash=ffeb7b..1506a2 dirty=3.27GiB
INFO [10-27|16:01:22.471] Unindexed transactions                   blocks=1   txs=281   tail=9,789,322 elapsed=14.334ms
INFO [10-27|16:01:22.964] Re-queue blocks                          number=12,139,322 hash=0x648363ab7ba5377b7685a09d371d8a5a548a11c5b84ea900f82a1abec152993c
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xf624b5]

goroutine 694852697 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestorBinarySearch(0xc00000c960, 0xc11b469e00, 0x0, 0xb93b3c, 0xb7dba9, 0xb7dba9, 0x0, 0x1b00940)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:988 +0x6d5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc00000c960, 0xc11b469e00, 0xc8a9414d80, 0x0, 0x0, 0x0)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:840 +0x3a5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc00000c960, 0xc11b469e00, 0x71824af7c6a90451, 0x947d740f6b4b75ad, 0xe3ebf62177694f07, 0xf48dd5ad452663e1, 0xc8b3a11460, 0x0, 0x0)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:497 +0x516
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc00000c960, 0xc465451fc0, 0x40, 0x71824af7c6a90451, 0x947d740f6b4b75ad, 0xe3ebf62177694f07, 0xf48dd5ad452663e1, 0xc8b3a11460, 0x0, 0x0, ...)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:453 +0x3b1
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc00000c960, 0xc465451fc0, 0x40, 0x71824af7c6a90451, 0x947d740f6b4b75ad, 0xe3ebf62177694f07, 0xf48dd5ad452663e1, 0xc8b3a11460, 0x0, 0x14f1880, ...)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:348 +0x8c
github.com/ethereum/go-ethereum/eth.(*handler).doSync(0xc001f91b00, 0xc2dc1ff180, 0x0, 0x486f81)
	github.com/ethereum/go-ethereum/eth/sync.go:325 +0x125
github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync.func1(0xc001fad140, 0xc2dc1ff180)
	github.com/ethereum/go-ethereum/eth/sync.go:301 +0x38
created by github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync
	github.com/ethereum/go-ethereum/eth/sync.go:301 +0x76
$ 

Steps to reproduce the behaviour

Yet to be consistently reproduced

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.

[easeev]

One more crash

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xf624b5]

goroutine 189847802 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestorBinarySearch(0xc00000c960, 0xca230d1950, 0x0, 0xbe387b, 0xbcd8a4, 0xbcd8a4, 0x0, 0x1b00940)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:988 +0x6d5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc00000c960, 0xca230d1950, 0xc2e9246240, 0x0, 0x0, 0x0)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:840 +0x3a5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc00000c960, 0xca230d1950, 0xdf8f8e43e462c74f, 0x90aee429bf5bda43, 0x1cf9abe7285bacb6, 0xa0b27d51e91741b8, 0xc7a7fc8120, 0x0, 0x0)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:497 +0x516
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc00000c960, 0xcaa43c8500, 0x40, 0xdf8f8e43e462c74f, 0x90aee429bf5bda43, 0x1cf9abe7285bacb6, 0xa0b27d51e91741b8, 0xc7a7fc8120, 0x0, 0x0, ...)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:453 +0x3b1
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc00000c960, 0xcaa43c8500, 0x40, 0xdf8f8e43e462c74f, 0x90aee429bf5bda43, 0x1cf9abe7285bacb6, 0xa0b27d51e91741b8, 0xc7a7fc8120, 0xc300000000, 0x1b33600, ...)
	github.com/ethereum/go-ethereum/eth/downloader/downloader.go:348 +0x8c
github.com/ethereum/go-ethereum/eth.(*handler).doSync(0xc059e90400, 0xc0428fc300, 0x1, 0x0)
	github.com/ethereum/go-ethereum/eth/sync.go:325 +0x125
github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync.func1(0xc059e893b0, 0xc0428fc300)
	github.com/ethereum/go-ethereum/eth/sync.go:301 +0x38
created by github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync
	github.com/ethereum/go-ethereum/eth/sync.go:301 +0x76

[keefel]

Fixed by #409