Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geth 1.8.2 in docker : fatal error: runtime: out of memory #16377

Closed
sulliwane opened this issue Mar 23, 2018 · 24 comments
Closed

Geth 1.8.2 in docker : fatal error: runtime: out of memory #16377

sulliwane opened this issue Mar 23, 2018 · 24 comments

Comments

@sulliwane
Copy link

sulliwane commented Mar 23, 2018

Hi there,

I get a fatal error: runtime: out of memory after updating geth to 1.8.2 in my docker-compose file. These two screenshots are separated by 1:30 minute (attached is video)
geth-out-of-memory.zip :

screenshot from 2018-03-23 11-18-00
After 1'30'' one can see that geth is constantly eating up memory
screenshot from 2018-03-23 11-18-16

These are the cli args used to start geth : geth --syncmode "fast" --testnet --ws --wsaddr "0.0.0.0" --wsorigins "*" --rpcvhosts my.domain --cache 512

Here is my server RAM:

less /proc/meminfo
MemTotal:        4045092 kB

Is that related to #16244, #16243 and #16174 ?

Many thanks for your help!

System information

Geth version: 1.8.2
OS & Version: Linux Docker

Backtrace

2018-03-23T08:46:59.051466662Z INFO [03-23|08:46:58] Generating ethash verification cache     epoch=101 percentage=88 elapsed=2m33.666s
2018-03-23T08:46:59.110963632Z INFO [03-23|08:46:59] Imported new state entries               count=555  elapsed=48.329ms  processed=3136088 pending=1406  retry=0   duplicate=0 unexpected=0
2018-03-23T08:46:59.110981772Z INFO [03-23|08:46:59] Imported new block headers               count=0    elapsed=67.024ms  number=2996204 hash=6010cc…4adc93 ignored=192
2018-03-23T08:46:59.114108529Z INFO [03-23|08:46:59] Imported new block receipts              count=213  elapsed=96.369ms  number=2994949 hash=e228fa…95c5ba size=667.78kB ignored=0
2018-03-23T08:46:59.125402457Z INFO [03-23|08:46:59] Imported new block headers               count=0    elapsed=6.390ms   number=2996396 hash=c260de…cbb623 ignored=192
2018-03-23T08:47:02.157967836Z INFO [03-23|08:47:01] Generating ethash verification cache     epoch=101 percentage=88 elapsed=2m36.802s
2018-03-23T08:47:02.456907921Z INFO [03-23|08:47:02] Imported new block headers               count=0    elapsed=91.459ms  number=2996588 hash=573842…dfd9ef ignored=192
2018-03-23T08:47:02.480951206Z INFO [03-23|08:47:02] Imported new block receipts              count=207  elapsed=108.148ms number=2995156 hash=3f1189…10ca8e size=831.33kB ignored=0
2018-03-23T08:47:05.249866518Z INFO [03-23|08:47:05] Generating ethash verification cache     epoch=101 percentage=88 elapsed=2m39.975s
2018-03-23T08:47:05.782409059Z INFO [03-23|08:47:05] Imported new block headers               count=0    elapsed=158.628ms number=2996780 hash=26e9c4…9159ad ignored=192
2018-03-23T08:47:05.849579741Z INFO [03-23|08:47:05] Imported new block headers               count=0    elapsed=28.674ms  number=2996972 hash=1330d6…1b885a ignored=192
2018-03-23T08:47:05.944288540Z INFO [03-23|08:47:05] Imported new block receipts              count=205  elapsed=97.359ms  number=2995361 hash=f9fc57…1277a2 size=984.12kB ignored=0
2018-03-23T08:47:08.583671237Z INFO [03-23|08:47:08] Generating ethash verification cache     epoch=101 percentage=88 elapsed=2m43.079s
2018-03-23T08:47:10.850788138Z INFO [03-23|08:47:10] Imported new block headers               count=0    elapsed=222.984ms number=2997164 hash=5ef424…021eb8 ignored=192
2018-03-23T08:47:11.534812147Z fatal error: runtime: out of memory
2018-03-23T08:47:11.598728309Z 
2018-03-23T08:47:11.598741686Z runtime stack:
2018-03-23T08:47:11.598745665Z runtime.throw(0xf8b55f, 0x16)
2018-03-23T08:47:11.598749468Z 	/usr/local/go/src/runtime/panic.go:605 +0x95
2018-03-23T08:47:11.598753798Z runtime.sysMap(0xc47f2f0000, 0xc000000, 0xc4200ec700, 0x1bbe078)
2018-03-23T08:47:11.598757763Z 	/usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
2018-03-23T08:47:11.598761524Z runtime.(*mheap).sysAlloc(0x1ba4700, 0xc000000, 0x7f681174f868)
2018-03-23T08:47:11.598765551Z 	/usr/local/go/src/runtime/malloc.go:470 +0xd7
2018-03-23T08:47:11.598769054Z runtime.(*mheap).grow(0x1ba4700, 0x6000, 0x0)
2018-03-23T08:47:11.598772527Z 	/usr/local/go/src/runtime/mheap.go:887 +0x60
2018-03-23T08:47:11.598776043Z runtime.(*mheap).allocSpanLocked(0x1ba4700, 0x6000, 0x1bbe088, 0xc4219a1200)
2018-03-23T08:47:11.598779731Z 	/usr/local/go/src/runtime/mheap.go:800 +0x334
2018-03-23T08:47:11.598783253Z runtime.(*mheap).alloc_m(0x1ba4700, 0x6000, 0x430101, 0x1b932d8)
2018-03-23T08:47:11.598787700Z 	/usr/local/go/src/runtime/mheap.go:666 +0x118
2018-03-23T08:47:11.598791588Z runtime.(*mheap).alloc.func1()
2018-03-23T08:47:11.598795029Z 	/usr/local/go/src/runtime/mheap.go:733 +0x4d
2018-03-23T08:47:11.598798424Z runtime.systemstack(0x7f681174f950)
2018-03-23T08:47:11.598801807Z 	/usr/local/go/src/runtime/asm_amd64.s:360 +0xab
2018-03-23T08:47:11.598805406Z runtime.(*mheap).alloc(0x1ba4700, 0x6000, 0x7f6811010101, 0x4513a5)
2018-03-23T08:47:11.598808910Z 	/usr/local/go/src/runtime/mheap.go:732 +0xa1
2018-03-23T08:47:11.598812325Z runtime.largeAlloc(0xc000000, 0xc433800101, 0x10000c47f1837a8)
2018-03-23T08:47:11.598815858Z 	/usr/local/go/src/runtime/malloc.go:827 +0x98
2018-03-23T08:47:11.598819313Z runtime.mallocgc.func1()
2018-03-23T08:47:11.598822687Z 	/usr/local/go/src/runtime/malloc.go:722 +0x46
2018-03-23T08:47:11.598826031Z runtime.systemstack(0xc42001f300)
2018-03-23T08:47:11.598829425Z 	/usr/local/go/src/runtime/asm_amd64.s:344 +0x79
2018-03-23T08:47:11.598832824Z runtime.mstart()
2018-03-23T08:47:11.598836129Z 	/usr/local/go/src/runtime/proc.go:1135
2018-03-23T08:47:11.598839521Z 
2018-03-23T08:47:11.598842756Z goroutine 51229 [running]:
2018-03-23T08:47:11.598846156Z runtime.systemstack_switch()
2018-03-23T08:47:11.598849658Z 	/usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc47f17eaa8 sp=0xc47f17eaa0 pc=0x46ef00
2018-03-23T08:47:11.598853232Z runtime.mallocgc(0xc000000, 0xdeec80, 0xdeadbe01, 0xc461dc8600)
2018-03-23T08:47:11.598856702Z 	/usr/local/go/src/runtime/malloc.go:721 +0x7ae fp=0xc47f17eb50 sp=0xc47f17eaa8 pc=0x42566e
2018-03-23T08:47:11.604469618Z runtime.makeslice(0xdeec80, 0x0, 0xc000000, 0x7f6811ace000, 0xc45930c000, 0xc456b6cd01)
2018-03-23T08:47:11.604479143Z 	/usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc47f17eb80 sp=0xc47f17eb50 pc=0x4586b7
2018-03-23T08:47:11.604483047Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/memdb.New(0x1834820, 0xc42049e6c0, 0xc000000, 0x0)
2018-03-23T08:47:11.630941869Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/memdb/memdb.go:470 +0xfc fp=0xc47f17ec20 sp=0xc47f17eb80 pc=0x65160c
2018-03-23T08:47:11.630972668Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolGet(0xc420234680, 0x1d347, 0x0)
2018-03-23T08:47:11.630983313Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:90 +0xb5 fp=0xc47f17ec68 sp=0xc47f17ec20 pc=0x67cf45
2018-03-23T08:47:11.630992059Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).newMem(0xc420234680, 0x1d347, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.631004268Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:147 +0x23b fp=0xc47f17ed18 sp=0xc47f17ec68 pc=0x67d4cb
2018-03-23T08:47:11.631013168Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).rotateMem(0xc420234680, 0x1d347, 0x0, 0xc47f17eef0, 0xc46c9c60f0, 0x7f6810aad730)
2018-03-23T08:47:11.631021638Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:46 +0x81 fp=0xc47f17ed70 sp=0xc47f17ed18 pc=0x680c11
2018-03-23T08:47:11.631029990Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).flush.func1(0xbea54da3adac6900)
2018-03-23T08:47:11.631038726Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:102 +0x2db fp=0xc47f17ee10 sp=0xc47f17ed70 pc=0x6972fb
2018-03-23T08:47:11.631047108Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).flush(0xc420234680, 0x1d347, 0xc431089300, 0x11397, 0x0, 0x0)
2018-03-23T08:47:11.631055541Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:114 +0x171 fp=0xc47f17eee0 sp=0xc47f17ee10 pc=0x680ee1
2018-03-23T08:47:11.631063888Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked(0xc420234680, 0xc42379da00, 0x0, 0x1, 0x0, 0x0)
2018-03-23T08:47:11.631072320Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:153 +0x6c fp=0xc47f17f088 sp=0xc47f17eee0 pc=0x6811dc
2018-03-23T08:47:11.631092190Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Write(0xc420234680, 0xc42379da00, 0x0, 0xc4562d98a8, 0xc47f17f218)
2018-03-23T08:47:11.631096697Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:313 +0x308 fp=0xc47f17f328 sp=0xc47f17f088 pc=0x681e68
2018-03-23T08:47:11.631100461Z github.com/ethereum/go-ethereum/ethdb.(*ldbBatch).Write(0xc4603e27e0, 0x1add7, 0xc42e47ebd0)
2018-03-23T08:47:11.631111976Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:293 +0x40 fp=0xc47f17f360 sp=0xc47f17f328 pc=0x69d960
2018-03-23T08:47:11.631118552Z github.com/ethereum/go-ethereum/core.(*BlockChain).InsertReceiptChain(0xc4205a0000, 0xc432679100, 0xd5, 0xd5, 0xc424d34000, 0xd5, 0xd5, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.631122339Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:811 +0x12cc fp=0xc47f17fa60 sp=0xc47f17f360 pc=0x876fdc
2018-03-23T08:47:11.631126004Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).commitFastSyncData(0xc4200edb00, 0xc420e14800, 0xd5, 0x100, 0xc469921ab0, 0xc420e14800, 0xd5)
2018-03-23T08:47:11.631129630Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1489 +0x63d fp=0xc47f17fd70 sp=0xc47f17fa60 pc=0xa49c9d
2018-03-23T08:47:11.631133330Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processFastSyncContent(0xc4200edb00, 0xc44d3d1680, 0x0, 0x0)
2018-03-23T08:47:11.631136944Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1408 +0x32e fp=0xc47f17ff60 sp=0xc47f17fd70 pc=0xa489ae
2018-03-23T08:47:11.631140569Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func7(0x8d84999300000008, 0x10086a0)
2018-03-23T08:47:11.631144150Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:469 +0x33 fp=0xc47f17ff90 sp=0xc47f17ff60 pc=0xa5ba53
2018-03-23T08:47:11.631147895Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc425a5ca60, 0xc44c67e600, 0xc473704680)
2018-03-23T08:47:11.631151468Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x51 fp=0xc47f17ffc8 sp=0xc47f17ff90 pc=0xa5bad1
2018-03-23T08:47:11.631155145Z runtime.goexit()
2018-03-23T08:47:11.631158561Z 	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc47f17ffd0 sp=0xc47f17ffc8 pc=0x471b31
2018-03-23T08:47:11.631162248Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
2018-03-23T08:47:11.631166351Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xc7
2018-03-23T08:47:11.631170057Z 
2018-03-23T08:47:11.631173347Z goroutine 1 [chan receive, 25 minutes]:
2018-03-23T08:47:11.631176907Z github.com/ethereum/go-ethereum/node.(*Node).Wait(0xc4201d4000)
2018-03-23T08:47:11.631180376Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:539 +0x7f
2018-03-23T08:47:11.631183936Z main.geth(0xc420216b00, 0x1006748, 0xb2d05e00)
2018-03-23T08:47:11.631187344Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:220 +0x51
2018-03-23T08:47:11.631191267Z github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0xe077e0, 0x1007c08, 0xc420216b00, 0xc420216b00, 0xc420497f40)
2018-03-23T08:47:11.631194961Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xd2
2018-03-23T08:47:11.631201780Z github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc420234ea0, 0xc4200100a0, 0xa, 0xa, 0x0, 0x0)
2018-03-23T08:47:11.631205539Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x635
2018-03-23T08:47:11.631209078Z main.main()
2018-03-23T08:47:11.631212488Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:208 +0x55
2018-03-23T08:47:11.631216011Z 
2018-03-23T08:47:11.631219308Z goroutine 6 [chan receive, 25 minutes]:
2018-03-23T08:47:11.631222748Z github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc42005c1e0, 0xc42005c120)
2018-03-23T08:47:11.631226339Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:36 +0xc0
2018-03-23T08:47:11.631230076Z created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
2018-03-23T08:47:11.631233610Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:29 +0x149
2018-03-23T08:47:11.631237187Z 
2018-03-23T08:47:11.631240555Z goroutine 7 [chan receive, 25 minutes]:
2018-03-23T08:47:11.631244014Z github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc42005c1e0, 0xc42005c180)
2018-03-23T08:47:11.631247641Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:81 +0x88
2018-03-23T08:47:11.631251259Z created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
2018-03-23T08:47:11.631254761Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:30 +0x178
2018-03-23T08:47:11.631258393Z 
2018-03-23T08:47:11.631261837Z goroutine 8 [syscall, 25 minutes]:
2018-03-23T08:47:11.631265342Z os/signal.signal_recv(0x0)
2018-03-23T08:47:11.631268816Z 	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
2018-03-23T08:47:11.631273917Z os/signal.loop()
2018-03-23T08:47:11.631277352Z 	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
2018-03-23T08:47:11.631280852Z created by os/signal.init.0
2018-03-23T08:47:11.631284261Z 	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
2018-03-23T08:47:11.631287718Z 
2018-03-23T08:47:11.631290956Z goroutine 10 [select, 25 minutes]:
2018-03-23T08:47:11.631294336Z github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).loop(0xc4204445c0)
2018-03-23T08:47:11.631297833Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:94 +0x511
2018-03-23T08:47:11.631301410Z created by github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).start
2018-03-23T08:47:11.631309537Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:52 +0x5d
2018-03-23T08:47:11.631313312Z 
2018-03-23T08:47:11.631316636Z goroutine 21 [syscall, 25 minutes]:
2018-03-23T08:47:11.631320000Z syscall.Syscall6(0xe8, 0x6, 0xc42002576c, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.631335238Z 	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
2018-03-23T08:47:11.631338557Z github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix.EpollWait(0x6, 0xc42002576c, 0x1, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.631342026Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1529 +0x77
2018-03-23T08:47:11.631345512Z github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).loop(0xc4200a4000, 0xc4200661e0)
2018-03-23T08:47:11.631348854Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:194 +0x8b
2018-03-23T08:47:11.631352307Z created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
2018-03-23T08:47:11.631355636Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:134 +0x13a
2018-03-23T08:47:11.631359116Z 
2018-03-23T08:47:11.631362227Z goroutine 22 [chan receive, 25 minutes]:
2018-03-23T08:47:11.631365522Z github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc4200a4000, 0xc4200661e0)
2018-03-23T08:47:11.631368923Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xc0
2018-03-23T08:47:11.631372398Z created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
2018-03-23T08:47:11.631376350Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x18d
2018-03-23T08:47:11.631379882Z 
2018-03-23T08:47:11.631383143Z goroutine 23 [chan receive, 25 minutes]:
2018-03-23T08:47:11.631386413Z github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).send(0xc4200a4000, 0xc4200661e0)
2018-03-23T08:47:11.631389793Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:254 +0xc0
2018-03-23T08:47:11.631393343Z created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*inotify).lazyinit
2018-03-23T08:47:11.631396863Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_inotify.go:137 +0x18d
2018-03-23T08:47:11.631400413Z 
2018-03-23T08:47:11.631403677Z goroutine 11 [select]:
2018-03-23T08:47:11.631407039Z github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).updater(0xc42009a000)
2018-03-23T08:47:11.631410457Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:203 +0x104
2018-03-23T08:47:11.631417052Z created by github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).Subscribe
2018-03-23T08:47:11.631420534Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:190 +0x12c
2018-03-23T08:47:11.631424266Z 
2018-03-23T08:47:11.631427420Z goroutine 12 [sleep]:
2018-03-23T08:47:11.631430672Z time.Sleep(0x3b9aca00)
2018-03-23T08:47:11.631434002Z 	/usr/local/go/src/runtime/time.go:65 +0x130
2018-03-23T08:47:11.631437330Z github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc4201d2000)
2018-03-23T08:47:11.631440780Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:224 +0x3e
2018-03-23T08:47:11.631444228Z created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe
2018-03-23T08:47:11.631447557Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:213 +0x12c
2018-03-23T08:47:11.631451162Z 
2018-03-23T08:47:11.631454372Z goroutine 13 [sleep]:
2018-03-23T08:47:11.631457579Z time.Sleep(0x3b9aca00)
2018-03-23T08:47:11.631460824Z 	/usr/local/go/src/runtime/time.go:65 +0x130
2018-03-23T08:47:11.631464137Z github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc4201d2140)
2018-03-23T08:47:11.631467487Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:224 +0x3e
2018-03-23T08:47:11.631470863Z created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe
2018-03-23T08:47:11.631474204Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:213 +0x12c
2018-03-23T08:47:11.631477662Z 
2018-03-23T08:47:11.631492476Z goroutine 14 [select, 25 minutes]:
2018-03-23T08:47:11.647125205Z github.com/ethereum/go-ethereum/accounts.(*Manager).update(0xc4201a65b0)
2018-03-23T08:47:11.647139408Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:95 +0x1d6
2018-03-23T08:47:11.647144036Z created by github.com/ethereum/go-ethereum/accounts.NewManager
2018-03-23T08:47:11.647386885Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:68 +0x5a7
2018-03-23T08:47:11.647399812Z 
2018-03-23T08:47:11.647403957Z goroutine 15 [select]:
2018-03-23T08:47:11.647407514Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc420292620)
2018-03-23T08:47:11.647411249Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x172
2018-03-23T08:47:11.647415090Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
2018-03-23T08:47:11.647425802Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x195
2018-03-23T08:47:11.647457288Z 
2018-03-23T08:47:11.647462723Z goroutine 24 [select]:
2018-03-23T08:47:11.647466192Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc420234680)
2018-03-23T08:47:11.647469803Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x119
2018-03-23T08:47:11.647473583Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.647477196Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:141 +0x4bd
2018-03-23T08:47:11.647480786Z 
2018-03-23T08:47:11.647484128Z goroutine 25 [select]:
2018-03-23T08:47:11.647487464Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc420234680)
2018-03-23T08:47:11.647491224Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x13b
2018-03-23T08:47:11.647494840Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.647498391Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x4df
2018-03-23T08:47:11.647503144Z 
2018-03-23T08:47:11.647506445Z goroutine 26 [select]:
2018-03-23T08:47:11.647509732Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc420234680)
2018-03-23T08:47:11.647513261Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x2c8
2018-03-23T08:47:11.647516910Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.647521542Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x65e
2018-03-23T08:47:11.647525226Z 
2018-03-23T08:47:11.647528478Z goroutine 27 [select]:
2018-03-23T08:47:11.647531848Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc420234680)
2018-03-23T08:47:11.647535409Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x192
2018-03-23T08:47:11.647539034Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.647542541Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x680
2018-03-23T08:47:11.647546156Z 
2018-03-23T08:47:11.647549441Z goroutine 28 [select, 24 minutes]:
2018-03-23T08:47:11.655849497Z github.com/ethereum/go-ethereum/core.(*ChainIndexer).updateLoop(0xc4201b23c0)
2018-03-23T08:47:11.655867808Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/chain_indexer.go:271 +0x176
2018-03-23T08:47:11.655872452Z created by github.com/ethereum/go-ethereum/core.NewChainIndexer
2018-03-23T08:47:11.655876118Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/chain_indexer.go:107 +0x25e
2018-03-23T08:47:11.655879763Z 
2018-03-23T08:47:11.655883315Z goroutine 29 [select]:
2018-03-23T08:47:11.655886892Z github.com/ethereum/go-ethereum/core.(*BlockChain).update(0xc4205a0000)
2018-03-23T08:47:11.655890450Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1376 +0x141
2018-03-23T08:47:11.655894047Z created by github.com/ethereum/go-ethereum/core.NewBlockChain
2018-03-23T08:47:11.655897532Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:193 +0x1162
2018-03-23T08:47:11.655901085Z 
2018-03-23T08:47:11.655904458Z goroutine 30 [select, 25 minutes]:
2018-03-23T08:47:11.655907930Z github.com/ethereum/go-ethereum/core.(*ChainIndexer).eventLoop(0xc4201b23c0, 0xc4201dd8c0, 0xc4200e91a0, 0x18391a0, 0xc420244520)
2018-03-23T08:47:11.655959523Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/chain_indexer.go:189 +0x261
2018-03-23T08:47:11.655976211Z created by github.com/ethereum/go-ethereum/core.(*ChainIndexer).Start
2018-03-23T08:47:11.655980889Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/chain_indexer.go:132 +0xcf
2018-03-23T08:47:11.655984628Z 
2018-03-23T08:47:11.655989315Z goroutine 31 [select]:
2018-03-23T08:47:11.657213444Z github.com/ethereum/go-ethereum/core.(*TxPool).loop(0xc4201e7dc0)
2018-03-23T08:47:11.657223079Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:283 +0x367
2018-03-23T08:47:11.657226827Z created by github.com/ethereum/go-ethereum/core.NewTxPool
2018-03-23T08:47:11.657230480Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:255 +0x5fe
2018-03-23T08:47:11.657234019Z 
2018-03-23T08:47:11.657237294Z goroutine 32 [select]:
2018-03-23T08:47:11.657240640Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).qosTuner(0xc4200edb00)
2018-03-23T08:47:11.657244254Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1569 +0x352
2018-03-23T08:47:11.657247783Z created by github.com/ethereum/go-ethereum/eth/downloader.New
2018-03-23T08:47:11.657251297Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:230 +0xa78
2018-03-23T08:47:11.657254918Z 
2018-03-23T08:47:11.657258148Z goroutine 33 [select]:
2018-03-23T08:47:11.657261518Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync(0xc4200edb00, 0xc469921ab0, 0x0)
2018-03-23T08:47:11.657282554Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:125 +0x463
2018-03-23T08:47:11.657287331Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).stateFetcher(0xc4200edb00)
2018-03-23T08:47:11.657290949Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:79 +0x47
2018-03-23T08:47:11.657294477Z created by github.com/ethereum/go-ethereum/eth/downloader.New
2018-03-23T08:47:11.657297879Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:231 +0xa9d
2018-03-23T08:47:11.657301712Z 
2018-03-23T08:47:11.657304985Z goroutine 34 [select, 25 minutes]:
2018-03-23T08:47:11.669823783Z github.com/ethereum/go-ethereum/miner.(*worker).update(0xc4201d2640)
2018-03-23T08:47:11.669842114Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:250 +0x309
2018-03-23T08:47:11.669847036Z created by github.com/ethereum/go-ethereum/miner.newWorker
2018-03-23T08:47:11.669850696Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:153 +0x532
2018-03-23T08:47:11.669854474Z 
2018-03-23T08:47:11.669857876Z goroutine 35 [chan receive, 25 minutes]:
2018-03-23T08:47:11.669861390Z github.com/ethereum/go-ethereum/miner.(*worker).wait(0xc4201d2640)
2018-03-23T08:47:11.669865087Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:293 +0x34e
2018-03-23T08:47:11.669868751Z created by github.com/ethereum/go-ethereum/miner.newWorker
2018-03-23T08:47:11.669872192Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:155 +0x554
2018-03-23T08:47:11.669875735Z 
2018-03-23T08:47:11.669879176Z goroutine 37 [select, 11 minutes]:
2018-03-23T08:47:11.669882621Z github.com/ethereum/go-ethereum/p2p/nat.Map(0x1841560, 0xc4201699c0, 0xc42056e780, 0xf78481, 0x3, 0x765f, 0x765f, 0xf86cbb, 0x12)
2018-03-23T08:47:11.669886448Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4cb
2018-03-23T08:47:11.669890044Z created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
2018-03-23T08:47:11.669893456Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:435 +0x1007
2018-03-23T08:47:11.669896978Z 
2018-03-23T08:47:11.669900375Z goroutine 66 [select]:
2018-03-23T08:47:11.669903888Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4202929a0)
2018-03-23T08:47:11.669907464Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x172
2018-03-23T08:47:11.669911140Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
2018-03-23T08:47:11.669914693Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x195
2018-03-23T08:47:11.669938327Z 
2018-03-23T08:47:11.669943072Z goroutine 69 [select]:
2018-03-23T08:47:11.669946542Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc420234820)
2018-03-23T08:47:11.669950192Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x13b
2018-03-23T08:47:11.669953902Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.669958168Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x4df
2018-03-23T08:47:11.669961827Z 
2018-03-23T08:47:11.669971288Z goroutine 68 [select, 25 minutes]:
2018-03-23T08:47:11.669975723Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc420234820)
2018-03-23T08:47:11.669979243Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x119
2018-03-23T08:47:11.669982943Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.669986452Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:141 +0x4bd
2018-03-23T08:47:11.669990044Z 
2018-03-23T08:47:11.669993301Z goroutine 53295 [select]:
2018-03-23T08:47:11.669996822Z github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders.func2(0xc471e88a20, 0xc442ef4600, 0xc0, 0xc0, 0xc42f00f030, 0xc46e012060, 0xc471e81800, 0xc0, 0xc0, 0xc459694a80)
2018-03-23T08:47:11.670000585Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:131 +0x193
2018-03-23T08:47:11.670004163Z created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders
2018-03-23T08:47:11.670008340Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:123 +0x3b8
2018-03-23T08:47:11.670011972Z 
2018-03-23T08:47:11.670015225Z goroutine 70 [select, 25 minutes]:
2018-03-23T08:47:11.670018560Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc420234820)
2018-03-23T08:47:11.670022100Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x2c8
2018-03-23T08:47:11.670025898Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.670029470Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x65e
2018-03-23T08:47:11.670033116Z 
2018-03-23T08:47:11.670036530Z goroutine 71 [select, 25 minutes]:
2018-03-23T08:47:11.670039895Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc420234820)
2018-03-23T08:47:11.670047967Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x192
2018-03-23T08:47:11.670051763Z created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2018-03-23T08:47:11.670056062Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x680
2018-03-23T08:47:11.670059740Z 
2018-03-23T08:47:11.670062965Z goroutine 72 [select, 25 minutes]:
2018-03-23T08:47:11.670066332Z github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).expirer(0xc432f78240)
2018-03-23T08:47:11.670069750Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:232 +0x128
2018-03-23T08:47:11.670073323Z created by github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).ensureExpirer.func1
2018-03-23T08:47:11.670076768Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:223 +0x3e
2018-03-23T08:47:11.670080259Z 
2018-03-23T08:47:11.670083534Z goroutine 73 [select]:
2018-03-23T08:47:11.670086830Z github.com/ethereum/go-ethereum/p2p/discover.(*Table).loop(0xc4201d28c0)
2018-03-23T08:47:11.670090353Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:381 +0x402
2018-03-23T08:47:11.670093835Z created by github.com/ethereum/go-ethereum/p2p/discover.newTable
2018-03-23T08:47:11.670097260Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:151 +0x552
2018-03-23T08:47:11.670100830Z 
2018-03-23T08:47:11.670104140Z goroutine 74 [select]:
2018-03-23T08:47:11.670107509Z github.com/ethereum/go-ethereum/p2p/discover.(*udp).loop(0xc425074ee0)
2018-03-23T08:47:11.670110967Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:386 +0x5e0
2018-03-23T08:47:11.670114486Z created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
2018-03-23T08:47:11.670117969Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:263 +0x308
2018-03-23T08:47:11.670121477Z 
2018-03-23T08:47:11.670124731Z goroutine 75 [runnable]:
2018-03-23T08:47:11.670128136Z internal/poll.runtime_pollWait(0x7f6811acef70, 0x72, 0x0)
2018-03-23T08:47:11.670131672Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.670135125Z internal/poll.(*pollDesc).wait(0xc4201ceb98, 0x72, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.670138670Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.670142294Z internal/poll.(*pollDesc).waitRead(0xc4201ceb98, 0xc42defc000, 0x500, 0x500)
2018-03-23T08:47:11.670145732Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-03-23T08:47:11.670152197Z internal/poll.(*FD).ReadFrom(0xc4201ceb80, 0xc42defc000, 0x500, 0x500, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.670155959Z 	/usr/local/go/src/internal/poll/fd_unix.go:170 +0x19a
2018-03-23T08:47:11.677785952Z net.(*netFD).readFrom(0xc4201ceb80, 0xc42defc000, 0x500, 0x500, 0x2bac2fc0e983f863, 0x83f9b1e58a5439e2, 0xdab65f96294290dd, 0x103a98ad4d0a02ff, 0x8ecdc437b1b38e57)
2018-03-23T08:47:11.677800655Z 	/usr/local/go/src/net/fd_unix.go:208 +0x5e
2018-03-23T08:47:11.677804958Z net.(*UDPConn).readFrom(0xc4201a9090, 0xc42defc000, 0x500, 0x500, 0xc4201b06b0, 0xc422b6fed8, 0x911fa1, 0xe39a20)
2018-03-23T08:47:11.677808928Z 	/usr/local/go/src/net/udpsock_posix.go:47 +0x6a
2018-03-23T08:47:11.677812593Z net.(*UDPConn).ReadFromUDP(0xc4201a9090, 0xc42defc000, 0x500, 0x500, 0x500, 0x1832d20, 0xc4201b06b0, 0x0)
2018-03-23T08:47:11.677816218Z 	/usr/local/go/src/net/udpsock.go:106 +0x72
2018-03-23T08:47:11.677819667Z github.com/ethereum/go-ethereum/p2p/discover.(*udp).readLoop(0xc425074ee0, 0x0)
2018-03-23T08:47:11.677823234Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:521 +0xec
2018-03-23T08:47:11.677826756Z created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
2018-03-23T08:47:11.677830223Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:264 +0x33a
2018-03-23T08:47:11.677833927Z 
2018-03-23T08:47:11.677840125Z goroutine 76 [IO wait, 5 minutes]:
2018-03-23T08:47:11.677843737Z internal/poll.runtime_pollWait(0x7f6811acedf0, 0x72, 0xffffffffffffffff)
2018-03-23T08:47:11.677847171Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.677850650Z internal/poll.(*pollDesc).wait(0xc425150c18, 0x72, 0xc431879d00, 0x0, 0x0)
2018-03-23T08:47:11.677854295Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.677857748Z internal/poll.(*pollDesc).waitRead(0xc425150c18, 0xffffffffffffff00, 0x0, 0x0)
2018-03-23T08:47:11.677861208Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-03-23T08:47:11.677864725Z internal/poll.(*FD).Accept(0xc425150c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.677868201Z 	/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
2018-03-23T08:47:11.677871741Z net.(*netFD).accept(0xc425150c00, 0x145c890, 0xc44f877ea0, 0x1b91d80)
2018-03-23T08:47:11.677875212Z 	/usr/local/go/src/net/fd_unix.go:238 +0x42
2018-03-23T08:47:11.677878671Z net.(*TCPListener).accept(0xc4262379f8, 0xc431879f10, 0x418b3b, 0xc424f97d40)
2018-03-23T08:47:11.677882171Z 	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
2018-03-23T08:47:11.677885704Z net.(*TCPListener).Accept(0xc4262379f8, 0x0, 0xc4201d4280, 0x18454e0, 0xc465df5658)
2018-03-23T08:47:11.677889221Z 	/usr/local/go/src/net/tcpsock.go:247 +0x49
2018-03-23T08:47:11.677892630Z github.com/ethereum/go-ethereum/p2p.(*Server).listenLoop(0xc4201d4280)
2018-03-23T08:47:11.677903493Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:762 +0x24a
2018-03-23T08:47:11.677907443Z created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
2018-03-23T08:47:11.677911502Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:519 +0x159
2018-03-23T08:47:11.677915043Z 
2018-03-23T08:47:11.677918298Z goroutine 77 [select, 11 minutes]:
2018-03-23T08:47:11.677921749Z github.com/ethereum/go-ethereum/p2p/nat.Map(0x1841560, 0xc4201699c0, 0xc42056e780, 0xf78475, 0x3, 0x765f, 0x765f, 0xf81470, 0xc)
2018-03-23T08:47:11.677925306Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4cb
2018-03-23T08:47:11.677928826Z github.com/ethereum/go-ethereum/p2p.(*Server).startListening.func1(0xc4201d4280, 0xc42513a540)
2018-03-23T08:47:11.677932296Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:524 +0x8b
2018-03-23T08:47:11.677935857Z created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
2018-03-23T08:47:11.677939241Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:523 +0x1fb
2018-03-23T08:47:11.677942706Z 
2018-03-23T08:47:11.677946045Z goroutine 78 [select]:
2018-03-23T08:47:11.677988543Z github.com/ethereum/go-ethereum/p2p.(*Server).run(0xc4201d4280, 0x18414a0, 0xc4201d1c20)
2018-03-23T08:47:11.677995449Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:589 +0x61b
2018-03-23T08:47:11.677999190Z created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
2018-03-23T08:47:11.678002754Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:504 +0x7f6
2018-03-23T08:47:11.678006282Z 
2018-03-23T08:47:11.678009554Z goroutine 79 [select, 25 minutes]:
2018-03-23T08:47:11.728372458Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728386838Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728391016Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728394563Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728398103Z 
2018-03-23T08:47:11.728401503Z goroutine 80 [select, 25 minutes]:
2018-03-23T08:47:11.728405025Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728408554Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728412049Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728415552Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728427646Z 
2018-03-23T08:47:11.728431429Z goroutine 81 [select, 25 minutes]:
2018-03-23T08:47:11.728434931Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728438513Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728442052Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728445501Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728449021Z 
2018-03-23T08:47:11.728452504Z goroutine 82 [select, 25 minutes]:
2018-03-23T08:47:11.728455956Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728459466Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728462978Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728466473Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728470010Z 
2018-03-23T08:47:11.728473390Z goroutine 83 [select, 25 minutes]:
2018-03-23T08:47:11.728476754Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728480284Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728483802Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728487308Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728490821Z 
2018-03-23T08:47:11.728494068Z goroutine 84 [select, 25 minutes]:
2018-03-23T08:47:11.728497385Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728501745Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728505309Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728508737Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728512265Z 
2018-03-23T08:47:11.728515542Z goroutine 85 [select, 25 minutes]:
2018-03-23T08:47:11.728518909Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728522322Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728525846Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728529260Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728532763Z 
2018-03-23T08:47:11.728538073Z goroutine 86 [select, 25 minutes]:
2018-03-23T08:47:11.728541511Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728544989Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728548512Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728551933Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728555416Z 
2018-03-23T08:47:11.728558682Z goroutine 87 [select, 25 minutes]:
2018-03-23T08:47:11.728562012Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728565436Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728568944Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728572354Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728575863Z 
2018-03-23T08:47:11.728579117Z goroutine 88 [select, 25 minutes]:
2018-03-23T08:47:11.728582497Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728586032Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728589580Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728593054Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728596583Z 
2018-03-23T08:47:11.728599821Z goroutine 89 [select, 25 minutes]:
2018-03-23T08:47:11.728603196Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728606862Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728610742Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728614214Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728617756Z 
2018-03-23T08:47:11.728621014Z goroutine 90 [select, 25 minutes]:
2018-03-23T08:47:11.728624344Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728627876Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728631378Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728634837Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728638410Z 
2018-03-23T08:47:11.728644540Z goroutine 91 [select, 25 minutes]:
2018-03-23T08:47:11.728647987Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728651513Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728655008Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728658446Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728662024Z 
2018-03-23T08:47:11.728665275Z goroutine 92 [select, 25 minutes]:
2018-03-23T08:47:11.728668670Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728672138Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728675635Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728679059Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728682565Z 
2018-03-23T08:47:11.728685827Z goroutine 93 [select, 25 minutes]:
2018-03-23T08:47:11.728689147Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728692601Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728696093Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728699508Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728703020Z 
2018-03-23T08:47:11.728706290Z goroutine 94 [select, 25 minutes]:
2018-03-23T08:47:11.728709621Z github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc420292700)
2018-03-23T08:47:11.728713039Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:55 +0x3c2
2018-03-23T08:47:11.728716497Z created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
2018-03-23T08:47:11.728720214Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:53 +0x48
2018-03-23T08:47:11.728723779Z 
2018-03-23T08:47:11.728727049Z goroutine 95 [select, 25 minutes]:
2018-03-23T08:47:11.728730391Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txBroadcastLoop(0xc4201b2480)
2018-03-23T08:47:11.728733850Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:741 +0x179
2018-03-23T08:47:11.728737343Z created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
2018-03-23T08:47:11.728740774Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:209 +0xcb
2018-03-23T08:47:11.728744223Z 
2018-03-23T08:47:11.728749974Z goroutine 96 [chan receive, 25 minutes]:
2018-03-23T08:47:11.728753497Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).minedBroadcastLoop(0xc4201b2480)
2018-03-23T08:47:11.728756988Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:730 +0x59
2018-03-23T08:47:11.728760426Z created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
2018-03-23T08:47:11.728763958Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:213 +0x15f
2018-03-23T08:47:11.728767485Z 
2018-03-23T08:47:11.728770766Z goroutine 97 [select]:
2018-03-23T08:47:11.728774120Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer(0xc4201b2480)
2018-03-23T08:47:11.728777579Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:145 +0x1e8
2018-03-23T08:47:11.728781010Z created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
2018-03-23T08:47:11.728784504Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:216 +0x181
2018-03-23T08:47:11.728787984Z 
2018-03-23T08:47:11.728791266Z goroutine 98 [select, 25 minutes]:
2018-03-23T08:47:11.728794652Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txsyncLoop(0xc4201b2480)
2018-03-23T08:47:11.728798131Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:109 +0x2bc
2018-03-23T08:47:11.728801606Z created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
2018-03-23T08:47:11.728805017Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:217 +0x1a3
2018-03-23T08:47:11.728808493Z 
2018-03-23T08:47:11.728811797Z goroutine 99 [select]:
2018-03-23T08:47:11.728815134Z github.com/ethereum/go-ethereum/eth/downloader.(*PublicDownloaderAPI).eventLoop(0xc430927b80)
2018-03-23T08:47:11.728818687Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:63 +0x336
2018-03-23T08:47:11.728822211Z created by github.com/ethereum/go-ethereum/eth/downloader.NewPublicDownloaderAPI
2018-03-23T08:47:11.728825696Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:49 +0xe7
2018-03-23T08:47:11.728829240Z 
2018-03-23T08:47:11.728832499Z goroutine 100 [select, 25 minutes]:
2018-03-23T08:47:11.728835938Z github.com/ethereum/go-ethereum/eth/filters.(*EventSystem).eventLoop(0xc42ea803c0)
2018-03-23T08:47:11.728839732Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/filter_system.go:443 +0x846
2018-03-23T08:47:11.728843299Z created by github.com/ethereum/go-ethereum/eth/filters.NewEventSystem
2018-03-23T08:47:11.728846772Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/filter_system.go:113 +0x104
2018-03-23T08:47:11.728850311Z 
2018-03-23T08:47:11.728856011Z goroutine 101 [chan receive]:
2018-03-23T08:47:11.728859485Z github.com/ethereum/go-ethereum/eth/filters.(*PublicFilterAPI).timeoutLoop(0xc4203645f0)
2018-03-23T08:47:11.728862991Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:83 +0x124
2018-03-23T08:47:11.728866552Z created by github.com/ethereum/go-ethereum/eth/filters.NewPublicFilterAPI
2018-03-23T08:47:11.728869991Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:73 +0x1a8
2018-03-23T08:47:11.728873468Z 
2018-03-23T08:47:11.728876772Z goroutine 102 [IO wait, 25 minutes]:
2018-03-23T08:47:11.728880122Z internal/poll.runtime_pollWait(0x7f6811aced30, 0x72, 0xffffffffffffffff)
2018-03-23T08:47:11.728883568Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.728887074Z internal/poll.(*pollDesc).wait(0xc432b73c98, 0x72, 0xc431c7bd00, 0x0, 0x0)
2018-03-23T08:47:11.728890523Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.728894006Z internal/poll.(*pollDesc).waitRead(0xc432b73c98, 0xffffffffffffff00, 0x0, 0x0)
2018-03-23T08:47:11.728897494Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-03-23T08:47:11.728900948Z internal/poll.(*FD).Accept(0xc432b73c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.728904415Z 	/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
2018-03-23T08:47:11.728907829Z net.(*netFD).accept(0xc432b73c80, 0x2, 0xc424d3a000, 0x14a86e0)
2018-03-23T08:47:11.728923828Z 	/usr/local/go/src/net/fd_unix.go:238 +0x42
2018-03-23T08:47:11.728927330Z net.(*UnixListener).accept(0xc4250c1470, 0xf87851, 0x13, 0x3)
2018-03-23T08:47:11.728930839Z 	/usr/local/go/src/net/unixsock_posix.go:162 +0x32
2018-03-23T08:47:11.728934463Z net.(*UnixListener).Accept(0xc4250c1470, 0xf87851, 0x13, 0xc42fd94d40, 0x2)
2018-03-23T08:47:11.728937992Z 	/usr/local/go/src/net/unixsock.go:241 +0x49
2018-03-23T08:47:11.728941490Z github.com/ethereum/go-ethereum/node.(*Node).startIPC.func1(0xc4201d4000, 0x183f660, 0xc4250c1470, 0xc42def4c00)
2018-03-23T08:47:11.728945050Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:330 +0x107
2018-03-23T08:47:11.728948609Z created by github.com/ethereum/go-ethereum/node.(*Node).startIPC
2018-03-23T08:47:11.728952104Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:326 +0x392
2018-03-23T08:47:11.728955622Z 
2018-03-23T08:47:11.728958871Z goroutine 103 [IO wait, 25 minutes]:
2018-03-23T08:47:11.728962346Z internal/poll.runtime_pollWait(0x7f6811acec70, 0x72, 0xffffffffffffffff)
2018-03-23T08:47:11.728966237Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.728970650Z internal/poll.(*pollDesc).wait(0xc42f9aaa98, 0x72, 0xc426295500, 0x0, 0x0)
2018-03-23T08:47:11.728974130Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.728980845Z internal/poll.(*pollDesc).waitRead(0xc42f9aaa98, 0xffffffffffffff00, 0x0, 0x0)
2018-03-23T08:47:11.728984465Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-03-23T08:47:11.728987983Z internal/poll.(*FD).Accept(0xc42f9aaa80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.728991438Z 	/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
2018-03-23T08:47:11.728994859Z net.(*netFD).accept(0xc42f9aaa80, 0x4258f8, 0x30, 0xee68c0)
2018-03-23T08:47:11.728998353Z 	/usr/local/go/src/net/fd_unix.go:238 +0x42
2018-03-23T08:47:11.729001794Z net.(*TCPListener).accept(0xc4250d8fa0, 0x525ada, 0xee68c0, 0xc4329715f0)
2018-03-23T08:47:11.729005229Z 	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
2018-03-23T08:47:11.729008670Z net.(*TCPListener).Accept(0xc4250d8fa0, 0xc42001c018, 0xe30700, 0x1802b20, 0xf5a0e0)
2018-03-23T08:47:11.729012232Z 	/usr/local/go/src/net/tcpsock.go:247 +0x49
2018-03-23T08:47:11.741833838Z net/http.(*Server).Serve(0xc4250a7ee0, 0x183f620, 0xc4250d8fa0, 0x0, 0x0)
2018-03-23T08:47:11.741845036Z 	/usr/local/go/src/net/http/server.go:2695 +0x1b2
2018-03-23T08:47:11.741849281Z created by github.com/ethereum/go-ethereum/node.(*Node).startWS
2018-03-23T08:47:11.741852810Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:449 +0x570
2018-03-23T08:47:11.741856534Z 
2018-03-23T08:47:11.741859915Z goroutine 104 [chan receive, 25 minutes]:
2018-03-23T08:47:11.754164915Z github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1(0xc4201d4000)
2018-03-23T08:47:11.754177844Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:70 +0x105
2018-03-23T08:47:11.754182456Z created by github.com/ethereum/go-ethereum/cmd/utils.StartNode
2018-03-23T08:47:11.754186266Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:66 +0xb7
2018-03-23T08:47:11.754190265Z 
2018-03-23T08:47:11.754193715Z goroutine 105 [chan receive, 25 minutes]:
2018-03-23T08:47:11.754197281Z main.startNode.func1(0xc4201d4000, 0xc4250a2f60)
2018-03-23T08:47:11.754200777Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:260 +0x32d
2018-03-23T08:47:11.754204416Z created by main.startNode
2018-03-23T08:47:11.754207862Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:245 +0x28b
2018-03-23T08:47:11.754211416Z 
2018-03-23T08:47:11.754214783Z goroutine 106 [semacquire, 25 minutes]:
2018-03-23T08:47:11.768352719Z sync.runtime_notifyListWait(0xc42fe00340, 0xc400000000)
2018-03-23T08:47:11.768364831Z 	/usr/local/go/src/runtime/sema.go:507 +0x110
2018-03-23T08:47:11.768368954Z sync.(*Cond).Wait(0xc42fe00330)
2018-03-23T08:47:11.768372483Z 	/usr/local/go/src/sync/cond.go:56 +0x80
2018-03-23T08:47:11.768382743Z io.(*pipe).read(0xc42fe00300, 0xc4260bbe00, 0x200, 0x200, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.768386753Z 	/usr/local/go/src/io/pipe.go:47 +0xc6
2018-03-23T08:47:11.780343010Z io.(*PipeReader).Read(0xc4250d8fa8, 0xc4260bbe00, 0x200, 0x200, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.780372685Z 	/usr/local/go/src/io/pipe.go:130 +0x4c
2018-03-23T08:47:11.791266020Z encoding/json.(*Decoder).refill(0xc4304e7180, 0xe335a0, 0x6d6fd7c8bd901d9d)
2018-03-23T08:47:11.791320447Z 	/usr/local/go/src/encoding/json/stream.go:152 +0x114
2018-03-23T08:47:11.791328107Z encoding/json.(*Decoder).readValue(0xc4304e7180, 0x0, 0x0, 0x4258f8)
2018-03-23T08:47:11.791332010Z 	/usr/local/go/src/encoding/json/stream.go:128 +0x275
2018-03-23T08:47:11.791335628Z encoding/json.(*Decoder).Decode(0xc4304e7180, 0xe49140, 0xc42fd94d80, 0x0, 0x0)
2018-03-23T08:47:11.791339312Z 	/usr/local/go/src/encoding/json/stream.go:57 +0x78
2018-03-23T08:47:11.806709674Z github.com/ethereum/go-ethereum/rpc.(*jsonCodec).ReadRequestHeaders(0xc420364c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.806800614Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/json.go:126 +0xeb
2018-03-23T08:47:11.806807213Z github.com/ethereum/go-ethereum/rpc.(*Server).readRequest(0xc430927bc0, 0x1845980, 0xc420364c80, 0x1, 0xe98a80, 0xc4212a0980, 0x1840d20, 0xc432971740, 0x11)
2018-03-23T08:47:11.806811210Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:378 +0x5f
2018-03-23T08:47:11.806815133Z github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest(0xc430927bc0, 0x1845980, 0xc420364c80, 0x4000000000000, 0x3, 0x0, 0x0)
2018-03-23T08:47:11.806819014Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:162 +0x2e2
2018-03-23T08:47:11.806822846Z github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0xc430927bc0, 0x1845980, 0xc420364c80, 0x3)
2018-03-23T08:47:11.806826530Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:218 +0x7c
2018-03-23T08:47:11.806830094Z created by github.com/ethereum/go-ethereum/rpc.DialInProc.func1
2018-03-23T08:47:11.806833677Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/inproc.go:29 +0x2c1
2018-03-23T08:47:11.806837370Z 
2018-03-23T08:47:11.806863942Z goroutine 107 [select, 25 minutes]:
2018-03-23T08:47:11.806869466Z github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0xc425153d50, 0x1845600, 0xc42511b720)
2018-03-23T08:47:11.806873099Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:500 +0x440
2018-03-23T08:47:11.806876743Z created by github.com/ethereum/go-ethereum/rpc.newClient
2018-03-23T08:47:11.806880209Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:203 +0x358
2018-03-23T08:47:11.806884719Z 
2018-03-23T08:47:11.806888287Z goroutine 108 [semacquire, 25 minutes]:
2018-03-23T08:47:11.807019890Z sync.runtime_notifyListWait(0xc42fe00400, 0xc400000000)
2018-03-23T08:47:11.807026178Z 	/usr/local/go/src/runtime/sema.go:507 +0x110
2018-03-23T08:47:11.807029794Z sync.(*Cond).Wait(0xc42fe003f0)
2018-03-23T08:47:11.807033354Z 	/usr/local/go/src/sync/cond.go:56 +0x80
2018-03-23T08:47:11.807036786Z io.(*pipe).read(0xc42fe003c0, 0xc4260bbc00, 0x200, 0x200, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.807040268Z 	/usr/local/go/src/io/pipe.go:47 +0xc6
2018-03-23T08:47:11.807044095Z io.(*PipeReader).Read(0xc4250d8fb8, 0xc4260bbc00, 0x200, 0x200, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.807047730Z 	/usr/local/go/src/io/pipe.go:130 +0x4c
2018-03-23T08:47:11.807051112Z encoding/json.(*Decoder).refill(0xc4304e7340, 0xdd3d80, 0x40000040)
2018-03-23T08:47:11.807054727Z 	/usr/local/go/src/encoding/json/stream.go:152 +0x114
2018-03-23T08:47:11.807058152Z encoding/json.(*Decoder).readValue(0xc4304e7340, 0x0, 0x0, 0x4258f8)
2018-03-23T08:47:11.807064085Z 	/usr/local/go/src/encoding/json/stream.go:128 +0x275
2018-03-23T08:47:11.807067768Z encoding/json.(*Decoder).Decode(0xc4304e7340, 0xe49140, 0xc42e4bc520, 0x4793d440fd42a1f0, 0x32c53a2ad808c861)
2018-03-23T08:47:11.807071425Z 	/usr/local/go/src/encoding/json/stream.go:57 +0x78
2018-03-23T08:47:11.807074925Z github.com/ethereum/go-ethereum/rpc.(*Client).read.func1(0xea0000, 0xc42e4bc540, 0xc42511b720, 0x7f68112eba60, 0xc42511b720)
2018-03-23T08:47:11.807078540Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:641 +0x79
2018-03-23T08:47:11.807112683Z github.com/ethereum/go-ethereum/rpc.(*Client).read(0xc425153d50, 0x1845600, 0xc42511b720, 0x0, 0x0)
2018-03-23T08:47:11.807119029Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:654 +0x152
2018-03-23T08:47:11.807122750Z created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
2018-03-23T08:47:11.807126215Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:476 +0x71
2018-03-23T08:47:11.807129850Z 
2018-03-23T08:47:11.807133143Z goroutine 53296 [copystack]:
2018-03-23T08:47:11.818714191Z github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.encodeBlock(0xc442c52c02, 0x12c7, 0x12c7, 0xc459378000, 0xfff, 0x1e00, 0x0)
2018-03-23T08:47:11.818727948Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode_amd64.s:265 +0x38a
2018-03-23T08:47:11.818732490Z github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy.Encode(0xc442c52c00, 0x12c9, 0x12c9, 0x0, 0x0, 0x0, 0x0, 0x7f6811ace7f8, 0xc43068bc00)
2018-03-23T08:47:11.818736372Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/golang/snappy/encode.go:37 +0x1b2
2018-03-23T08:47:11.818740187Z github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).WriteMsg(0xc420012540, 0x1f, 0xfff, 0x1833820, 0xc461dc82d0, 0x0, 0x0, 0x0, 0xffc, 0xc42001e070)
2018-03-23T08:47:11.818743904Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:609 +0x10b
2018-03-23T08:47:11.818768866Z github.com/ethereum/go-ethereum/p2p.(*rlpx).WriteMsg(0xc46407aea0, 0x1f, 0xfff, 0x1833820, 0xc461dc82d0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.818773867Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:102 +0x102
2018-03-23T08:47:11.830225080Z github.com/ethereum/go-ethereum/p2p.(*conn).WriteMsg(0xc42dfd5400, 0x1f, 0xfff, 0x1833820, 0xc461dc82d0, 0x0, 0x0, 0x0, 0x7c, 0x1b931a0)
2018-03-23T08:47:11.830238502Z 	<autogenerated>:1 +0x64
2018-03-23T08:47:11.830326513Z github.com/ethereum/go-ethereum/p2p.(*protoRW).WriteMsg(0xc4200125b0, 0x1f, 0xfff, 0x1833820, 0xc461dc82d0, 0x0, 0x0, 0x0, 0xc427bb7710, 0x60f176)
2018-03-23T08:47:11.830333539Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:390 +0x1d4
2018-03-23T08:47:11.830461992Z github.com/ethereum/go-ethereum/p2p.Send(0x7f681115f490, 0xc4200125b0, 0xf, 0xdd7fc0, 0xc45930a000, 0x2, 0x0)
2018-03-23T08:47:11.830470823Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/message.go:97 +0x10f
2018-03-23T08:47:11.830545831Z github.com/ethereum/go-ethereum/eth.(*peer).RequestReceipts(0xc45d2af320, 0xc4668fd000, 0x7c, 0x7c, 0x6, 0x6)
2018-03-23T08:47:11.830553919Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:228 +0x1d2
2018-03-23T08:47:11.841833005Z created by github.com/ethereum/go-ethereum/eth/downloader.(*peerConnection).FetchReceipts
2018-03-23T08:47:11.841846054Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/peer.go:203 +0x24b
2018-03-23T08:47:11.841850543Z 
2018-03-23T08:47:11.841854113Z goroutine 51228 [chan receive]:
2018-03-23T08:47:11.841966765Z github.com/ethereum/go-ethereum/core.(*HeaderChain).ValidateHeaderChain(0xc4203103f0, 0xc442ef4600, 0xc0, 0xc0, 0x64, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.841973806Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/headerchain.go:245 +0x9db
2018-03-23T08:47:11.841977686Z github.com/ethereum/go-ethereum/core.(*BlockChain).InsertHeaderChain(0xc4205a0000, 0xc442ef4600, 0xc0, 0xc0, 0x64, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.842019166Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1439 +0xbe
2018-03-23T08:47:11.842061823Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc4200edb00, 0x2dbbad, 0x5053f1, 0xc425540580, 0x0, 0x0)
2018-03-23T08:47:11.842100404Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1265 +0x6c9
2018-03-23T08:47:11.842144319Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0x8a35cb0d00000008, 0x10086a0)
2018-03-23T08:47:11.842150149Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:466 +0x49
2018-03-23T08:47:11.842193338Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc425a5ca60, 0xc44c67e600, 0xc45b7278f0)
2018-03-23T08:47:11.842267413Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x51
2018-03-23T08:47:11.842275929Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
2018-03-23T08:47:11.842335373Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xc7
2018-03-23T08:47:11.842341600Z 
2018-03-23T08:47:11.842345130Z goroutine 50636 [IO wait]:
2018-03-23T08:47:11.842348665Z internal/poll.runtime_pollWait(0x7f6811ace7f0, 0x72, 0x0)
2018-03-23T08:47:11.842389449Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.842431649Z internal/poll.(*pollDesc).wait(0xc45c5a3a98, 0x72, 0xffffffffffffff00, 0x1837d60, 0x1830b10)
2018-03-23T08:47:11.842469207Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.842519905Z internal/poll.(*pollDesc).waitRead(0xc45c5a3a98, 0xc457151000, 0x20, 0x20)
2018-03-23T08:47:11.842561288Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-03-23T08:47:11.842602349Z internal/poll.(*FD).Read(0xc45c5a3a80, 0xc457151020, 0x20, 0x20, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.842660309Z 	/usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
2018-03-23T08:47:11.842666121Z net.(*netFD).Read(0xc45c5a3a80, 0xc457151020, 0x20, 0x20, 0x100190100507633, 0x0, 0x37c44ac0f11)
2018-03-23T08:47:11.842705878Z 	/usr/local/go/src/net/fd_unix.go:202 +0x52
2018-03-23T08:47:11.842781504Z net.(*conn).Read(0xc4615f7198, 0xc457151020, 0x20, 0x20, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.842787301Z 	/usr/local/go/src/net/net.go:176 +0x6d
2018-03-23T08:47:11.842917187Z io.ReadAtLeast(0x7f681115f418, 0xc4615f7198, 0xc457151020, 0x20, 0x20, 0x20, 0xf50b20, 0x168be3cb400, 0x7f681115f418)
2018-03-23T08:47:11.842959666Z 	/usr/local/go/src/io/io.go:309 +0x86
2018-03-23T08:47:11.843021872Z io.ReadFull(0x7f681115f418, 0xc4615f7198, 0xc457151020, 0x20, 0x20, 0x20, 0x0, 0x6fc2392f0)
2018-03-23T08:47:11.843027515Z 	/usr/local/go/src/io/io.go:327 +0x58
2018-03-23T08:47:11.843069613Z github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).ReadMsg(0xc420012540, 0xbea54dab29d38da1, 0x168be3cb46a, 0x1b91d80, 0x0, 0x0, 0x1b91d80, 0x161c218e936, 0x1b91d80, 0x20002)
2018-03-23T08:47:11.843111377Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:656 +0x100
2018-03-23T08:47:11.843204185Z github.com/ethereum/go-ethereum/p2p.(*rlpx).ReadMsg(0xc46407aea0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.843210441Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:95 +0x148
2018-03-23T08:47:11.843214072Z github.com/ethereum/go-ethereum/p2p.(*Peer).readLoop(0xc4207f1320, 0xc4207f1500)
2018-03-23T08:47:11.843217409Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:256 +0xad
2018-03-23T08:47:11.843265046Z created by github.com/ethereum/go-ethereum/p2p.(*Peer).run
2018-03-23T08:47:11.843276648Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:194 +0xf2
2018-03-23T08:47:11.843319447Z 
2018-03-23T08:47:11.843325154Z goroutine 52524 [select]:
2018-03-23T08:47:11.843391457Z net.(*netFD).connect.func2(0x1840ce0, 0xc4299c5020, 0xc460bfd880, 0xc4415057a0, 0xc441505740)
2018-03-23T08:47:11.843467002Z 	/usr/local/go/src/net/fd_unix.go:129 +0xf2
2018-03-23T08:47:11.843473633Z created by net.(*netFD).connect
2018-03-23T08:47:11.843556807Z 	/usr/local/go/src/net/fd_unix.go:128 +0x2a3
2018-03-23T08:47:11.843564259Z 
2018-03-23T08:47:11.843569481Z goroutine 51327 [chan receive]:
2018-03-23T08:47:11.857548779Z github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).Wait(...)
2018-03-23T08:47:11.857561577Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:261
2018-03-23T08:47:11.857566032Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processFastSyncContent.func1(0xc47058b8f0, 0xc4200edb00)
2018-03-23T08:47:11.857569760Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1361 +0x44
2018-03-23T08:47:11.857633259Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processFastSyncContent
2018-03-23T08:47:11.857640086Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1360 +0x11b
2018-03-23T08:47:11.857643804Z 
2018-03-23T08:47:11.857647101Z goroutine 51220 [chan receive]:
2018-03-23T08:47:11.857706805Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync(0xc4200edb00, 0xc451b11a40, 0x5, 0x8, 0x5, 0xc451b11a40)
2018-03-23T08:47:11.857714938Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:495 +0x14c
2018-03-23T08:47:11.857757122Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc4200edb00, 0xc46c4de870, 0x3db76b1cd021dffd, 0x509272f376475aa5, 0x3e97b2fe6214b12, 0x2209be1bde1c5ab1, 0xc425540580, 0x0, 0x0)
2018-03-23T08:47:11.857800634Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:473 +0x6a3
2018-03-23T08:47:11.857911767Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc4200edb00, 0xc45fb89890, 0x10, 0x3db76b1cd021dffd, 0x509272f376475aa5, 0x3e97b2fe6214b12, 0x2209be1bde1c5ab1, 0xc425540580, 0x1, 0x0, ...)
2018-03-23T08:47:11.858132402Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:398 +0x3e2
2018-03-23T08:47:11.858141194Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc4200edb00, 0xc45fb89890, 0x10, 0x3db76b1cd021dffd, 0x509272f376475aa5, 0x3e97b2fe6214b12, 0x2209be1bde1c5ab1, 0xc425540580, 0x1, 0x4cd806f3bd76ea57, ...)
2018-03-23T08:47:11.858145471Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:316 +0x97
2018-03-23T08:47:11.858149332Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc4201b2480, 0xc45d2af320)
2018-03-23T08:47:11.858184686Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:192 +0x263
2018-03-23T08:47:11.858190805Z created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
2018-03-23T08:47:11.858194315Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x23c
2018-03-23T08:47:11.858198010Z 
2018-03-23T08:47:11.858201544Z goroutine 50637 [select]:
2018-03-23T08:47:11.858205086Z github.com/ethereum/go-ethereum/p2p.(*Peer).pingLoop(0xc4207f1320)
2018-03-23T08:47:11.858208752Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:240 +0x182
2018-03-23T08:47:11.858212322Z created by github.com/ethereum/go-ethereum/p2p.(*Peer).run
2018-03-23T08:47:11.858215903Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:195 +0x117
2018-03-23T08:47:11.858219430Z 
2018-03-23T08:47:11.858222921Z goroutine 50638 [select]:
2018-03-23T08:47:11.858226353Z github.com/ethereum/go-ethereum/p2p.(*protoRW).ReadMsg(0xc4200125b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.858229882Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:403 +0x148
2018-03-23T08:47:11.858233546Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handleMsg(0xc4201b2480, 0xc45d2af320, 0x0, 0x0)
2018-03-23T08:47:11.858237109Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:320 +0xbd
2018-03-23T08:47:11.858247837Z github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc4201b2480, 0xc45d2af320, 0x0, 0x0)
2018-03-23T08:47:11.858372034Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:309 +0x8ae
2018-03-23T08:47:11.858379379Z github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4207f1320, 0x18394a0, 0xc4200125b0, 0x0, 0x0)
2018-03-23T08:47:11.858383070Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1f2
2018-03-23T08:47:11.858386705Z github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc4200125b0, 0xc4207f1320, 0x18394a0, 0xc4200125b0)
2018-03-23T08:47:11.858390300Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:349 +0x66
2018-03-23T08:47:11.858394005Z created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
2018-03-23T08:47:11.858397475Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:348 +0x228
2018-03-23T08:47:11.858401065Z 
2018-03-23T08:47:11.858404431Z goroutine 150 [select]:
2018-03-23T08:47:11.858418833Z github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).loop(0xc42045f6c0)
2018-03-23T08:47:11.858564537Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:316 +0x804
2018-03-23T08:47:11.858571726Z created by github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).Start
2018-03-23T08:47:11.858580834Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:178 +0x3f
2018-03-23T08:47:11.858584661Z 
2018-03-23T08:47:11.871049349Z goroutine 151 [select, 25 minutes, locked to thread]:
2018-03-23T08:47:11.873029167Z runtime.gopark(0x1008558, 0x0, 0xf7b290, 0x6, 0x18, 0x1)
2018-03-23T08:47:11.873038316Z 	/usr/local/go/src/runtime/proc.go:287 +0x12c
2018-03-23T08:47:11.873042118Z runtime.selectgo(0xc421d58f50, 0xc424d3c120)
2018-03-23T08:47:11.873045582Z 	/usr/local/go/src/runtime/select.go:395 +0x1149
2018-03-23T08:47:11.873049197Z runtime.ensureSigM.func1()
2018-03-23T08:47:11.873052672Z 	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
2018-03-23T08:47:11.873056210Z runtime.goexit()
2018-03-23T08:47:11.873059734Z 	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1
2018-03-23T08:47:11.873063193Z 
2018-03-23T08:47:11.873066709Z goroutine 52758 [IO wait]:
2018-03-23T08:47:11.873070148Z internal/poll.runtime_pollWait(0x7f6811acea30, 0x77, 0xc42001e000)
2018-03-23T08:47:11.873073756Z 	/usr/local/go/src/runtime/netpoll.go:173 +0x57
2018-03-23T08:47:11.873077351Z internal/poll.(*pollDesc).wait(0xc460bfd898, 0x77, 0xc460375600, 0x4497fb, 0xc4603756a0)
2018-03-23T08:47:11.873080858Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
2018-03-23T08:47:11.873084324Z internal/poll.(*pollDesc).waitWrite(0xc460bfd898, 0xc460375600, 0xc4603756d8, 0x5d0d33)
2018-03-23T08:47:11.873087914Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:94 +0x3d
2018-03-23T08:47:11.873091632Z internal/poll.(*FD).WaitWrite(0xc460bfd880, 0x1007d20, 0x1840ce0)
2018-03-23T08:47:11.874877244Z 	/usr/local/go/src/internal/poll/fd_unix.go:403 +0x37
2018-03-23T08:47:11.874882178Z net.(*netFD).connect(0xc460bfd880, 0x1840ce0, 0xc4299c5020, 0x0, 0x0, 0x1835c20, 0xc4466d57a0, 0x0, 0x0, 0x0, ...)
2018-03-23T08:47:11.874885801Z 	/usr/local/go/src/net/fd_unix.go:152 +0x2b6
2018-03-23T08:47:11.874889280Z net.(*netFD).dial(0xc460bfd880, 0x1840ce0, 0xc4299c5020, 0x1844400, 0x0, 0x1844400, 0xc44595bdd0, 0xc4603758f8, 0x5ee9be)
2018-03-23T08:47:11.874892976Z 	/usr/local/go/src/net/sock_posix.go:142 +0xe9
2018-03-23T08:47:11.874896466Z net.socket(0x1840ce0, 0xc4299c5020, 0xf78475, 0x3, 0x2, 0x1, 0x0, 0x0, 0x1844400, 0x0, ...)
2018-03-23T08:47:11.874899949Z 	/usr/local/go/src/net/sock_posix.go:93 +0x1a5
2018-03-23T08:47:11.874903346Z net.internetSocket(0x1840ce0, 0xc4299c5020, 0xf78475, 0x3, 0x1844400, 0x0, 0x1844400, 0xc44595bdd0, 0x1, 0x0, ...)
2018-03-23T08:47:11.874906916Z 	/usr/local/go/src/net/ipsock_posix.go:141 +0x129
2018-03-23T08:47:11.874910351Z net.doDialTCP(0x1840ce0, 0xc4299c5020, 0xf78475, 0x3, 0x0, 0xc44595bdd0, 0x1bbc348, 0x0, 0x0)
2018-03-23T08:47:11.874913838Z 	/usr/local/go/src/net/tcpsock_posix.go:62 +0xb9
2018-03-23T08:47:11.874923497Z net.dialTCP(0x1840ce0, 0xc4299c5020, 0xf78475, 0x3, 0x0, 0xc44595bdd0, 0xbea54da533bf0936, 0x16331a54035, 0x1b91d80)
2018-03-23T08:47:11.874927429Z 	/usr/local/go/src/net/tcpsock_posix.go:58 +0xe4
2018-03-23T08:47:11.874930899Z net.dialSingle(0x1840ce0, 0xc4299c5020, 0xc460bfd800, 0x183a320, 0xc44595bdd0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.874934509Z 	/usr/local/go/src/net/dial.go:547 +0x3e2
2018-03-23T08:47:11.874939408Z net.dialSerial(0x1840ce0, 0xc4299c5020, 0xc460bfd800, 0xc4262d1d80, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.874943058Z 	/usr/local/go/src/net/dial.go:515 +0x247
2018-03-23T08:47:11.874946556Z net.(*Dialer).DialContext(0xc4200e9aa0, 0x1840ca0, 0xc42001c018, 0xf78475, 0x3, 0xc46098e040, 0x14, 0x0, 0x0, 0x0, ...)
2018-03-23T08:47:11.874950638Z 	/usr/local/go/src/net/dial.go:397 +0x6ee
2018-03-23T08:47:11.874954055Z net.(*Dialer).Dial(0xc4200e9aa0, 0xf78475, 0x3, 0xc46098e040, 0x14, 0x30, 0xc427bbb468, 0x7f6810f09018, 0x0)
2018-03-23T08:47:11.874957734Z 	/usr/local/go/src/net/dial.go:320 +0x75
2018-03-23T08:47:11.874961111Z github.com/ethereum/go-ethereum/p2p.TCPDialer.Dial(0xc4200e9aa0, 0xc45f8a0d20, 0x30, 0xeb4dc0, 0x92bd5695270e9801, 0xc444fbc4b0)
2018-03-23T08:47:11.874964797Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:65 +0xb6
2018-03-23T08:47:11.874968351Z github.com/ethereum/go-ethereum/p2p.(*dialTask).dial(0xc444fbd290, 0xc4201d4280, 0xc45f8a0d20, 0x6c238827c1821aea, 0xe8b547d32612822c)
2018-03-23T08:47:11.874972019Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:349 +0x48
2018-03-23T08:47:11.874975675Z github.com/ethereum/go-ethereum/p2p.(*dialTask).Do(0xc444fbd290, 0xc4201d4280)
2018-03-23T08:47:11.874979237Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:297 +0x58
2018-03-23T08:47:11.874982727Z github.com/ethereum/go-ethereum/p2p.(*Server).run.func2.1(0x1833620, 0xc444fbd290, 0xc4201d4280, 0xc425002ae0)
2018-03-23T08:47:11.874986230Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570 +0x3b
2018-03-23T08:47:11.874989705Z created by github.com/ethereum/go-ethereum/p2p.(*Server).run.func2
2018-03-23T08:47:11.874993145Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570 +0x1b3
2018-03-23T08:47:11.874996651Z 
2018-03-23T08:47:11.874999888Z goroutine 51226 [select]:
2018-03-23T08:47:11.875003301Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchParts(0xc4200edb00, 0x1832d20, 0xc4201b1460, 0xc4200e95c0, 0xc4583c1360, 0xc420236070, 0xc47f183eb0, 0xc47f183f00, 0xc47f183ef0, 0xc47f183ee0, ...)
2018-03-23T08:47:11.875007129Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1004 +0x23f
2018-03-23T08:47:11.875010683Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchBodies(0xc4200edb00, 0x2dacad, 0xc427bb9f98, 0xa283d2101012a6f1)
2018-03-23T08:47:11.875014303Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:933 +0x33e
2018-03-23T08:47:11.875020887Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func4(0x9b15080b00000008, 0x10086a0)
2018-03-23T08:47:11.875024895Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:464 +0x37
2018-03-23T08:47:11.875028550Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc425a5ca60, 0xc44c67e600, 0xc473704640)
2018-03-23T08:47:11.875032142Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x51
2018-03-23T08:47:11.875035690Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
2018-03-23T08:47:11.875039121Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xc7
2018-03-23T08:47:11.875042681Z 
2018-03-23T08:47:11.875045965Z goroutine 50316 [runnable]:
2018-03-23T08:47:11.875049323Z github.com/ethereum/go-ethereum/consensus/ethash.makeHasher.func1(0x7f680b69f908, 0xda24c0, 0xda24c0, 0xc446059080, 0x40, 0x40)
2018-03-23T08:47:11.875052828Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:101 +0xb2
2018-03-23T08:47:11.875056342Z github.com/ethereum/go-ethereum/consensus/ethash.generateCache(0x7f680a7a2008, 0x727f70, 0x727f70, 0x65, 0xc45b158b60, 0x20, 0x20)
2018-03-23T08:47:11.875059905Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:188 +0x570
2018-03-23T08:47:11.875063424Z github.com/ethereum/go-ethereum/consensus/ethash.(*cache).generate.func1.1(0x7f680a7a2008, 0x727f70, 0x727f70)
2018-03-23T08:47:11.875066921Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:250 +0x66
2018-03-23T08:47:11.875070858Z github.com/ethereum/go-ethereum/consensus/ethash.memoryMapAndGenerate(0xc4278d03c0, 0x36, 0x1c9fdc0, 0xc422b6ee98, 0x2, 0x2, 0x0, 0x0, 0x0, 0x18359a0, ...)
2018-03-23T08:47:11.875074548Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:133 +0x249
2018-03-23T08:47:11.875078107Z github.com/ethereum/go-ethereum/consensus/ethash.(*cache).generate.func1()
2018-03-23T08:47:11.875081608Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:250 +0x664
2018-03-23T08:47:11.875085107Z sync.(*Once).Do(0xc45cdc6810, 0xc427bb5f78)
2018-03-23T08:47:11.875088494Z 	/usr/local/go/src/sync/once.go:44 +0xbe
2018-03-23T08:47:11.875091869Z github.com/ethereum/go-ethereum/consensus/ethash.(*cache).generate(0xc45cdc67d0, 0xc4200162c0, 0x1b, 0x3, 0xc463a7d500)
2018-03-23T08:47:11.875095468Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:216 +0x96
2018-03-23T08:47:11.875099004Z created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).cache
2018-03-23T08:47:11.875102896Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:506 +0x123
2018-03-23T08:47:11.875108984Z 
2018-03-23T08:47:11.875112416Z goroutine 53294 [semacquire]:
2018-03-23T08:47:11.875115798Z sync.runtime_Semacquire(0xc4202346b4)
2018-03-23T08:47:11.875119158Z 	/usr/local/go/src/runtime/sema.go:56 +0x39
2018-03-23T08:47:11.875122544Z sync.(*RWMutex).RLock(0xc4202346a8)
2018-03-23T08:47:11.875126237Z 	/usr/local/go/src/sync/rwmutex.go:50 +0x49
2018-03-23T08:47:11.875129727Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).getMems(0xc420234680, 0x0, 0x0)
2018-03-23T08:47:11.875133266Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:159 +0x4c
2018-03-23T08:47:11.875136937Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc420234680, 0x0, 0x0, 0x0, 0x0, 0xc4621ab140, 0x29, 0x30, 0x2707f95, 0x0, ...)
2018-03-23T08:47:11.875140687Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:765 +0x113
2018-03-23T08:47:11.875144251Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc420234680, 0xc4621ab140, 0x29, 0x30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.875150665Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:844 +0x163
2018-03-23T08:47:11.875154195Z github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Get(0xc42045e0d0, 0xc4621ab140, 0x29, 0x30, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.875157737Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:121 +0xb2
2018-03-23T08:47:11.875161285Z github.com/ethereum/go-ethereum/core.GetHeaderRLP(0x7f68116f1840, 0xc42045e0d0, 0xaa97f6ab6ce1c746, 0x842b91e8d04ab703, 0xbcb28a56ac69e179, 0xa2eaecd83b581fdb, 0x2dbbaf, 0xc45937c629, 0x180fb90, 0xc46de7c820)
2018-03-23T08:47:11.875165011Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/database_util.go:163 +0x189
2018-03-23T08:47:11.875168534Z github.com/ethereum/go-ethereum/core.GetHeader(0x7f68116f1840, 0xc42045e0d0, 0xaa97f6ab6ce1c746, 0x842b91e8d04ab703, 0xbcb28a56ac69e179, 0xa2eaecd83b581fdb, 0x2dbbaf, 0x20)
2018-03-23T08:47:11.875172166Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/database_util.go:170 +0x7a
2018-03-23T08:47:11.875176081Z github.com/ethereum/go-ethereum/core.(*HeaderChain).GetHeader(0xc4203103f0, 0xaa97f6ab6ce1c746, 0x842b91e8d04ab703, 0xbcb28a56ac69e179, 0xa2eaecd83b581fdb, 0x2dbbaf, 0xc425f5d408)
2018-03-23T08:47:11.875179811Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/headerchain.go:351 +0x141
2018-03-23T08:47:11.875183420Z github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).verifyHeaderWorker(0xc420312d20, 0x1843fe0, 0xc4203103f0, 0xc442ef4600, 0xc0, 0xc0, 0xc471e869c0, 0xc0, 0xc0, 0x2, ...)
2018-03-23T08:47:11.875187100Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:162 +0x18a
2018-03-23T08:47:11.875190722Z github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders.func1(0xc471e88a20, 0xc471e81800, 0xc0, 0xc0, 0xc420312d20, 0x1843fe0, 0xc4203103f0, 0xc442ef4600, 0xc0, 0xc0, ...)
2018-03-23T08:47:11.875197117Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:116 +0xff
2018-03-23T08:47:11.875200849Z created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders
2018-03-23T08:47:11.875204292Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:114 +0x2d7
2018-03-23T08:47:11.875207848Z 
2018-03-23T08:47:11.875211096Z goroutine 51227 [select]:
2018-03-23T08:47:11.875214565Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchParts(0xc4200edb00, 0x1832d20, 0xc4201b1470, 0xc4200e9620, 0xc4583c1370, 0xc4202360e0, 0xc454159eb0, 0xc454159f00, 0xc454159ef0, 0xc454159ee0, ...)
2018-03-23T08:47:11.875218292Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1004 +0x23f
2018-03-23T08:47:11.875221830Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchReceipts(0xc4200edb00, 0x2dacad, 0xc42028e798, 0x66576e8054f7f01c)
2018-03-23T08:47:11.875225401Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:957 +0x33e
2018-03-23T08:47:11.875228959Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func5(0x5fd3016700000008, 0x10086a0)
2018-03-23T08:47:11.875232472Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:465 +0x37
2018-03-23T08:47:11.875236011Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc425a5ca60, 0xc44c67e600, 0xc473704660)
2018-03-23T08:47:11.875239589Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x51
2018-03-23T08:47:11.875243177Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
2018-03-23T08:47:11.875246906Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xc7
2018-03-23T08:47:11.875250490Z 
2018-03-23T08:47:11.875253797Z goroutine 50352 [select]:
2018-03-23T08:47:11.875257293Z github.com/ethereum/go-ethereum/consensus/ethash.generateCache.func2(0xc464dc4420, 0x1845900, 0xc45d450f00, 0xc4566468f8, 0x727f7, 0xbea54d7a4a2c3825, 0x13b37ad81b0, 0x1b91d80)
2018-03-23T08:47:11.875260992Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:160 +0x12e
2018-03-23T08:47:11.875264558Z created by github.com/ethereum/go-ethereum/consensus/ethash.generateCache
2018-03-23T08:47:11.875267971Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/algorithm.go:158 +0x2f6
2018-03-23T08:47:11.875271511Z 
2018-03-23T08:47:11.875274783Z goroutine 50635 [select]:
2018-03-23T08:47:11.875278119Z github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4207f1320, 0xdb6c60, 0xc45812c680, 0x0)
2018-03-23T08:47:11.875281584Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:204 +0x2fe
2018-03-23T08:47:11.875287386Z github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc4201d4280, 0xc4207f1320)
2018-03-23T08:47:11.875290956Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:894 +0x122
2018-03-23T08:47:11.875294434Z created by github.com/ethereum/go-ethereum/p2p.(*Server).run
2018-03-23T08:47:11.875297919Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:646 +0x1443
2018-03-23T08:47:11.875301430Z 
2018-03-23T08:47:11.875304689Z goroutine 51328 [select]:
2018-03-23T08:47:11.875308037Z github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).loop(0xc469921ab0, 0x0, 0x0)
2018-03-23T08:47:11.875311551Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:290 +0x257
2018-03-23T08:47:11.875315086Z github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc469921ab0)
2018-03-23T08:47:11.875318511Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:255 +0x2b
2018-03-23T08:47:11.875321988Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
2018-03-23T08:47:11.875325430Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:106 +0x134
2018-03-23T08:47:11.875328942Z 
2018-03-23T08:47:11.875332197Z goroutine 51225 [select]:
2018-03-23T08:47:11.875335641Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchParts(0xc4200edb00, 0x1832d20, 0xc4201b1450, 0xc4200e9560, 0xc44f282c20, 0xc426400d90, 0xc420fdb950, 0xc420fdb990, 0xc420fdb980, 0x1006408, ...)
2018-03-23T08:47:11.875339703Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1004 +0x23f
2018-03-23T08:47:11.875343206Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fillHeaderSkeleton(0xc4200edb00, 0x2dacad, 0xc4201e4900, 0x80, 0x8d, 0x0, 0x0, 0x0, 0xc45c5a3a80, 0xc4717cf520, ...)
2018-03-23T08:47:11.875346831Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:907 +0x37d
2018-03-23T08:47:11.875350353Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).fetchHeaders(0xc4200edb00, 0xc46c4de870, 0x2dacad, 0x5053f1, 0x0, 0x0)
2018-03-23T08:47:11.875353910Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:832 +0x130e
2018-03-23T08:47:11.875357419Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func3(0xc400000008, 0x10086a0)
2018-03-23T08:47:11.875360887Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:463 +0x49
2018-03-23T08:47:11.875364377Z github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc425a5ca60, 0xc44c67e600, 0xc45b7278c0)
2018-03-23T08:47:11.875367887Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x51
2018-03-23T08:47:11.875371459Z created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
2018-03-23T08:47:11.875379876Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xc7
2018-03-23T08:47:11.875383601Z 
2018-03-23T08:47:11.875386860Z goroutine 53053 [runnable]:
2018-03-23T08:47:11.875390257Z github.com/ethereum/go-ethereum/p2p/discover.(*Table).lookup(0xc4201d28c0, 0xac6f82561ac22d51, 0x7cb137a835db104b, 0xfd776dc5c81f6865, 0xf0613ecde9b77f8e, 0xc6d71626af2251e4, 0x566f9affe88e17d2, 0x9f8f01dd2a3cfffc, 0x8956809bc66c24cf, 0x1, ...)
2018-03-23T08:47:11.875394059Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:286 +0x1ff
2018-03-23T08:47:11.875397604Z github.com/ethereum/go-ethereum/p2p/discover.(*Table).Lookup(0xc4201d28c0, 0xac6f82561ac22d51, 0x7cb137a835db104b, 0xfd776dc5c81f6865, 0xf0613ecde9b77f8e, 0xc6d71626af2251e4, 0x566f9affe88e17d2, 0x9f8f01dd2a3cfffc, 0x8956809bc66c24cf, 0x0, ...)
2018-03-23T08:47:11.875401414Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:279 +0x50
2018-03-23T08:47:11.875404936Z github.com/ethereum/go-ethereum/p2p.(*discoverTask).Do(0xc4635cfac0, 0xc4201d4280)
2018-03-23T08:47:11.875408836Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/dial.go:372 +0x1af
2018-03-23T08:47:11.875412401Z github.com/ethereum/go-ethereum/p2p.(*Server).run.func2.1(0x1833660, 0xc4635cfac0, 0xc4201d4280, 0xc425002ae0)
2018-03-23T08:47:11.875415945Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570 +0x3b
2018-03-23T08:47:11.875419440Z created by github.com/ethereum/go-ethereum/p2p.(*Server).run.func2
2018-03-23T08:47:11.875423014Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:570 +0x1b3
2018-03-23T08:47:11.875426474Z 
2018-03-23T08:47:11.875429748Z goroutine 53293 [semacquire]:
2018-03-23T08:47:11.875433115Z sync.runtime_Semacquire(0xc4202346b4)
2018-03-23T08:47:11.875436576Z 	/usr/local/go/src/runtime/sema.go:56 +0x39
2018-03-23T08:47:11.875439986Z sync.(*RWMutex).RLock(0xc4202346a8)
2018-03-23T08:47:11.875443498Z 	/usr/local/go/src/sync/rwmutex.go:50 +0x49
2018-03-23T08:47:11.875446905Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).getMems(0xc420234680, 0x0, 0x0)
2018-03-23T08:47:11.875450476Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:159 +0x4c
2018-03-23T08:47:11.875454148Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc420234680, 0x0, 0x0, 0x0, 0x0, 0xc4621ab110, 0x29, 0x30, 0x2707f95, 0x0, ...)
2018-03-23T08:47:11.875457759Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:765 +0x113
2018-03-23T08:47:11.875461349Z github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc420234680, 0xc4621ab110, 0x29, 0x30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.875464959Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:844 +0x163
2018-03-23T08:47:11.875471637Z github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Get(0xc42045e0d0, 0xc4621ab110, 0x29, 0x30, 0x0, 0x0, 0x0, 0x0, 0x0)
2018-03-23T08:47:11.875475314Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:121 +0xb2
2018-03-23T08:47:11.875478837Z github.com/ethereum/go-ethereum/core.GetHeaderRLP(0x7f68116f1840, 0xc42045e0d0, 0x9d360ca2e4dabf69, 0xb22a9570af5365fc, 0x2eb829ed560f71c8, 0xa916c1f8e2839feb, 0x2dbbb0, 0xc445909ea9, 0x180fb90, 0xc426244000)
2018-03-23T08:47:11.875482552Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/database_util.go:163 +0x189
2018-03-23T08:47:11.875486546Z github.com/ethereum/go-ethereum/core.GetHeader(0x7f68116f1840, 0xc42045e0d0, 0x9d360ca2e4dabf69, 0xb22a9570af5365fc, 0x2eb829ed560f71c8, 0xa916c1f8e2839feb, 0x2dbbb0, 0x20)
2018-03-23T08:47:11.875490222Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/database_util.go:170 +0x7a
2018-03-23T08:47:11.875493700Z github.com/ethereum/go-ethereum/core.(*HeaderChain).GetHeader(0xc4203103f0, 0x9d360ca2e4dabf69, 0xb22a9570af5365fc, 0x2eb829ed560f71c8, 0xa916c1f8e2839feb, 0x2dbbb0, 0xc46c63d408)
2018-03-23T08:47:11.875497345Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/headerchain.go:351 +0x141
2018-03-23T08:47:11.875500833Z github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).verifyHeaderWorker(0xc420312d20, 0x1843fe0, 0xc4203103f0, 0xc442ef4600, 0xc0, 0xc0, 0xc471e869c0, 0xc0, 0xc0, 0x3, ...)
2018-03-23T08:47:11.875504472Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:162 +0x18a
2018-03-23T08:47:11.875508024Z github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders.func1(0xc471e88a20, 0xc471e81800, 0xc0, 0xc0, 0xc420312d20, 0x1843fe0, 0xc4203103f0, 0xc442ef4600, 0xc0, 0xc0, ...)
2018-03-23T08:47:11.875511671Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:116 +0xff
2018-03-23T08:47:11.875515195Z created by github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).VerifyHeaders
2018-03-23T08:47:11.875518660Z 	/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/consensus/ethash/consensus.go:114 +0x2d7
2018-03-23T08:47:15.810019011Z INFO [03-23|08:47:15] Maximum peer count                       ETH=25 LES=0 total=25
2018-03-23T08:47:15.813359244Z INFO [03-23|08:47:15] Starting peer-to-peer node               instance=Geth/v1.8.2-stable/linux-amd64/go1.9.4
2018-03-23T08:47:15.813403765Z INFO [03-23|08:47:15] Allocated cache and file handles         database=/root/.ethereum/geth/chaindata cache=768 handles=1024
2018-03-23T08:47:21.032834659Z INFO [03-23|08:47:21] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: <nil> Engine: ethash}"
2018-03-23T08:47:21.032889942Z INFO [03-23|08:47:21] Disk storage enabled for ethash caches   dir=/root/.ethereum/geth/ethash count=3
2018-03-23T08:47:21.032896839Z INFO [03-23|08:47:21] Disk storage enabled for ethash DAGs     dir=/root/.ethash               count=2
2018-03-23T08:47:21.032934585Z INFO [03-23|08:47:21] Initialising Ethereum protocol           versions="[63 62]" network=1
2018-03-23T08:47:21.041455061Z INFO [03-23|08:47:21] Loaded most recent local header          number=2999882 hash=c34248…adf48a td=119963881908411413293
2018-03-23T08:47:21.041494336Z INFO [03-23|08:47:21] Loaded most recent local full block      number=0       hash=d4e567…cb8fa3 td=17179869184
2018-03-23T08:47:21.041500814Z INFO [03-23|08:47:21] Loaded most recent local fast block      number=2995361 hash=f9fc57…1277a2 td=119496996859318936739
2018-03-23T08:47:21.047201488Z INFO [03-23|08:47:21] Loaded local transaction journal         transactions=0 dropped=0
2018-03-23T08:47:21.047316609Z INFO [03-23|08:47:21] Regenerated local transaction journal    transactions=0 accounts=0
2018-03-23T08:47:21.047466879Z INFO [03-23|08:47:21] Upgrading chain index                    type=bloombits percentage=99
2018-03-23T08:47:21.055182918Z INFO [03-23|08:47:21] Starting P2P networking 
2018-03-23T08:47:21.448363979Z INFO [03-23|08:47:21] Finished upgrading chain index           type=bloombits
2018-03-23T08:47:23.251858038Z INFO [03-23|08:47:23] UDP listener up                          self=enode://467f98b4d65dd2c87e91047df920cf42f6842ec5026cdca93c6b4dfa80338bcb93432c8b911e804dc324f90ec8a58c7b95e44d748fbd1ddfc1380e9a4c229111@[::]:30303
2018-03-23T08:47:23.254043831Z INFO [03-23|08:47:23] WebSocket endpoint opened                url=ws://[::]:8546
2018-03-23T08:47:23.254215401Z INFO [03-23|08:47:23] RLPx listener up                         self=enode://467f98b4d65dd2c87e91047df920cf42f6842ec5026cdca93c6b4dfa80338bcb93432c8b911e804dc324f90ec8a58c7b95e44d748fbd1ddfc1380e9a4c229111@[::]:30303
2018-03-23T08:47:23.255253716Z INFO [03-23|08:47:23] IPC endpoint opened                      url=/root/.ethereum/geth.ipc
2018-03-23T08:47:53.256363219Z INFO [03-23|08:47:53] Block synchronisation started 
2018-03-23T08:47:54.917600269Z INFO [03-23|08:47:54] Imported new block headers               count=0 elapsed=5.644ms number=2995553 hash=aabecf…87c2fc ignored=192
2018-03-23T08:47:55.089727363Z INFO [03-23|08:47:55] Imported new block headers               count=0 elapsed=7.148ms number=2995745 hash=83269e…897c01 ignored=192
2018-03-23T08:47:55.097301524Z INFO [03-23|08:47:55] Imported new block receipts              count=0 elapsed=133.973µs number=2995363 hash=d5eec8…14420c size=0.00B ignored=2
2018-03-23T08:47:55.227679861Z INFO [03-23|08:47:55] Imported new block headers               count=0 elapsed=5.100ms   number=2995937 hash=b13139…3d74f3 ignored=192
2018-03-23T08:47:55.277502425Z INFO [03-23|08:47:55] Imported new block receipts              count=0 elapsed=684.068µs number=2995398 hash=3151bf…11f1ee size=0.00B ignored=35
2018-03-23T08:47:55.555592356Z INFO [03-23|08:47:55] Imported new state entries               count=268 elapsed=10.936µs  processed=3136356 pending=4278 retry=0 duplicate=0 unexpected=0
2018-03-23T08:47:55.596098924Z INFO [03-23|08:47:55] Imported new block headers               count=0   elapsed=7.858ms   number=2996129 hash=317c60…4b1f8e ignored=192
2018-03-23T08:47:59.319014376Z INFO [03-23|08:47:59] Imported new block receipts              count=89  elapsed=25.650ms  number=2995607 hash=d42769…acb92f size=293.47kB ignored=120
2018-03-23T08:48:03.624534226Z INFO [03-23|08:48:03] Imported new block headers               count=0   elapsed=7.431ms   number=2996321 hash=569d64…e736c5 ignored=192
2018-03-23T08:48:03.767250140Z INFO [03-23|08:48:03] Imported new state entries               count=384 elapsed=10.007µs  processed=3136740 pending=10213 retry=0 duplicate=0 unexpected=0
2018-03-23T08:48:03.792492295Z INFO [03-23|08:48:03] Imported new block receipts              count=219 elapsed=55.043ms  number=2995826 hash=11797c…c9e98c size=830.64kB ignored=0
2018-03-23T08:48:05.888332032Z INFO [03-23|08:48:05] Imported new block headers               count=0   elapsed=7.513ms   number=2996513 hash=cd2d05…8069e1 ignored=192
2018-03-23T08:48:06.002482455Z INFO [03-23|08:48:06] Imported new block receipts              count=201 elapsed=15.159ms  number=2996027 hash=d5a977…23cbee size=570.54kB ignored=0
2018-03-23T08:48:10.861325641Z WARN [03-23|08:48:10] Node data write error                    err="state node 23be2e…1f310b failed with all peers (0 tries, 0 peers)"
2018-03-23T08:48:10.861366810Z WARN [03-23|08:48:10] Synchronisation failed, retrying         err="block body download canceled (requested)"
2018-03-23T08:48:29.842778409Z WARN [03-23|08:48:29] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (1 tries, 1 peers)"
2018-03-23T08:48:29.842892736Z WARN [03-23|08:48:29] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (1 tries, 1 peers)"
2018-03-23T08:48:33.725051226Z WARN [03-23|08:48:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (1 tries, 1 peers)"
2018-03-23T08:48:33.725337612Z WARN [03-23|08:48:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (1 tries, 1 peers)"
2018-03-23T08:48:43.718695420Z WARN [03-23|08:48:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:48:43.718791927Z WARN [03-23|08:48:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:48:53.719332594Z WARN [03-23|08:48:53] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:48:53.722176451Z WARN [03-23|08:48:53] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:03.727181913Z WARN [03-23|08:49:03] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:03.727251778Z WARN [03-23|08:49:03] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:13.721323699Z WARN [03-23|08:49:13] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:13.721661357Z WARN [03-23|08:49:13] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:23.731557760Z WARN [03-23|08:49:23] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:23.731894941Z WARN [03-23|08:49:23] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:33.715523637Z WARN [03-23|08:49:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:33.715850346Z WARN [03-23|08:49:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:43.718693875Z WARN [03-23|08:49:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:43.721464645Z WARN [03-23|08:49:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:53.727569628Z WARN [03-23|08:49:53] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:49:53.727849213Z WARN [03-23|08:49:53] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:03.716360285Z WARN [03-23|08:50:03] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:03.718998474Z WARN [03-23|08:50:03] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:13.713366602Z WARN [03-23|08:50:13] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:13.713707024Z WARN [03-23|08:50:13] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:23.714296781Z WARN [03-23|08:50:23] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:23.714608377Z WARN [03-23|08:50:23] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:33.721508355Z WARN [03-23|08:50:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:33.721613076Z WARN [03-23|08:50:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:43.741436746Z WARN [03-23|08:50:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:43.741840112Z WARN [03-23|08:50:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:50:53.959713220Z INFO [03-23|08:50:53] Imported new block headers               count=0   elapsed=3.835ms   number=2996219 hash=9f3dc4…18c567 ignored=192
2018-03-23T08:50:53.975147190Z INFO [03-23|08:50:53] Imported new block headers               count=0   elapsed=4.158ms   number=2996411 hash=8868e8…5808ec ignored=192
2018-03-23T08:50:59.638657445Z INFO [03-23|08:50:59] Imported new block receipts              count=384 elapsed=61.561ms  number=2996411 hash=8868e8…5808ec size=900.52kB ignored=0
2018-03-23T08:50:59.771595945Z INFO [03-23|08:50:59] Imported new block headers               count=0   elapsed=125.345ms number=2998331 hash=227080…9f4109 ignored=1920
2018-03-23T08:51:01.014657784Z INFO [03-23|08:51:01] Imported new block receipts              count=226 elapsed=15.628ms  number=2996637 hash=afd439…72fc88 size=595.13kB ignored=0
2018-03-23T08:51:04.902036171Z WARN [03-23|08:51:04] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:51:04.902067355Z WARN [03-23|08:51:04] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:51:15.107726559Z INFO [03-23|08:51:15] Imported new block headers               count=0   elapsed=9.992ms   number=2997021 hash=7e206c…91e712 ignored=384
2018-03-23T08:51:15.352044730Z INFO [03-23|08:51:15] Imported new block receipts              count=2   elapsed=208.157µs number=2996639 hash=46f061…ca11b1 size=2.07kB   ignored=0
2018-03-23T08:51:15.366367606Z INFO [03-23|08:51:15] Imported new block receipts              count=2   elapsed=361.655µs number=2996641 hash=b5f732…354a72 size=7.05kB   ignored=0
2018-03-23T08:51:21.324660492Z WARN [03-23|08:51:21] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:21.324779946Z WARN [03-23|08:51:21] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:23.778866577Z WARN [03-23|08:51:23] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:23.782020313Z WARN [03-23|08:51:23] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:33.717278414Z WARN [03-23|08:51:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:33.720130036Z WARN [03-23|08:51:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:43.720695799Z WARN [03-23|08:51:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:43.723754771Z WARN [03-23|08:51:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:53.716305961Z WARN [03-23|08:51:53] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:51:53.716372170Z WARN [03-23|08:51:53] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:03.723022292Z WARN [03-23|08:52:03] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:03.725984674Z WARN [03-23|08:52:03] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:13.724480866Z WARN [03-23|08:52:13] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:13.724856884Z WARN [03-23|08:52:13] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:23.804217319Z WARN [03-23|08:52:23] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:23.804304216Z WARN [03-23|08:52:23] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:33.714853885Z WARN [03-23|08:52:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:33.715253735Z WARN [03-23|08:52:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:43.730226736Z WARN [03-23|08:52:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:43.730329076Z WARN [03-23|08:52:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:53.719388892Z WARN [03-23|08:52:53] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:52:53.719468270Z WARN [03-23|08:52:53] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:53:03.729958630Z WARN [03-23|08:53:03] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:53:03.730272310Z WARN [03-23|08:53:03] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (2 tries, 2 peers)"
2018-03-23T08:53:14.003276967Z INFO [03-23|08:53:14] Imported new block headers               count=0   elapsed=3.665ms   number=2996833 hash=a12f64…cca8dd ignored=192
2018-03-23T08:53:14.614593737Z WARN [03-23|08:53:14] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:14.614879685Z WARN [03-23|08:53:14] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:25.531073973Z WARN [03-23|08:53:25] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:25.533901963Z WARN [03-23|08:53:25] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:36.764897216Z WARN [03-23|08:53:36] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:36.767888388Z WARN [03-23|08:53:36] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:43.842160725Z WARN [03-23|08:53:43] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:43.845312073Z WARN [03-23|08:53:43] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:53.999748086Z INFO [03-23|08:53:53] Imported new block headers               count=0   elapsed=4.564ms   number=2996833 hash=a12f64…cca8dd ignored=192
2018-03-23T08:53:54.909502073Z WARN [03-23|08:53:54] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:53:54.909801869Z WARN [03-23|08:53:54] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:03.913748906Z WARN [03-23|08:54:03] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:03.916672451Z WARN [03-23|08:54:03] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:13.878404571Z WARN [03-23|08:54:13] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:13.878579838Z WARN [03-23|08:54:13] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:24.172353452Z INFO [03-23|08:54:24] Imported new block headers               count=0   elapsed=13.175ms  number=2997025 hash=b8e456…1a5949 ignored=384
2018-03-23T08:54:24.364964019Z INFO [03-23|08:54:24] Imported new block receipts              count=7   elapsed=464.012µs number=2996648 hash=54fea3…822f7a size=11.08kB  ignored=0
2018-03-23T08:54:26.926650307Z WARN [03-23|08:54:26] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:26.926767201Z WARN [03-23|08:54:26] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:34.163410708Z WARN [03-23|08:54:34] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:34.163791135Z WARN [03-23|08:54:34] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:44.060080082Z INFO [03-23|08:54:44] Imported new block headers               count=0   elapsed=7.256ms   number=2997032 hash=c72426…22308e ignored=384
2018-03-23T08:54:44.307511983Z INFO [03-23|08:54:44] Imported new block receipts              count=6   elapsed=683.165µs number=2996654 hash=6caf04…4f0840 size=21.30kB  ignored=0
2018-03-23T08:54:45.132228568Z INFO [03-23|08:54:45] Imported new block receipts              count=117 elapsed=37.842ms  number=2996771 hash=39edf9…726575 size=548.56kB ignored=0
2018-03-23T08:54:45.200075693Z INFO [03-23|08:54:45] Imported new block headers               count=0   elapsed=70.057ms  number=2998952 hash=e1eaf0…dc1c72 ignored=1920
2018-03-23T08:54:45.220593704Z INFO [03-23|08:54:45] Imported new block headers               count=0   elapsed=5.319ms   number=2999144 hash=097e5e…68695b ignored=192
2018-03-23T08:54:45.238769235Z WARN [03-23|08:54:45] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:45.238918285Z INFO [03-23|08:54:45] Imported new block headers               count=0   elapsed=11.034ms  number=2999528 hash=eda371…5b8349 ignored=384
2018-03-23T08:54:45.246025442Z WARN [03-23|08:54:45] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:54.476595856Z INFO [03-23|08:54:54] Imported new block headers               count=0   elapsed=7.022ms   number=2996963 hash=f8fa4d…3f16a6 ignored=192
2018-03-23T08:54:54.557181898Z INFO [03-23|08:54:54] Imported new block headers               count=0   elapsed=5.201ms   number=2997155 hash=b8a722…15b1b4 ignored=192
2018-03-23T08:54:54.577392080Z INFO [03-23|08:54:54] Imported new block receipts              count=2   elapsed=6.871ms   number=2996773 hash=b7c163…4ad976 size=49.56kB  ignored=0
2018-03-23T08:54:54.656506991Z INFO [03-23|08:54:54] Imported new block receipts              count=3   elapsed=857.377µs number=2996776 hash=22878c…cce00d size=34.28kB  ignored=0
2018-03-23T08:54:55.640979272Z WARN [03-23|08:54:55] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:54:55.641179331Z WARN [03-23|08:54:55] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:04.413771736Z WARN [03-23|08:55:04] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:04.414120001Z WARN [03-23|08:55:04] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:13.926848830Z INFO [03-23|08:55:13] Imported new block headers               count=0   elapsed=3.622ms   number=2996968 hash=3081c9…0cec90 ignored=192
2018-03-23T08:55:14.012304630Z INFO [03-23|08:55:14] Imported new block headers               count=0   elapsed=3.698ms   number=2997160 hash=8f29ca…a28982 ignored=192
2018-03-23T08:55:14.146705223Z INFO [03-23|08:55:14] Imported new block headers               count=0   elapsed=6.362ms   number=2997352 hash=6b9e1d…0a35dc ignored=192
2018-03-23T08:55:19.845650900Z WARN [03-23|08:55:19] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:19.845897234Z WARN [03-23|08:55:19] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:24.027542346Z INFO [03-23|08:55:24] Imported new block headers               count=0   elapsed=10.059ms  number=2997160 hash=8f29ca…a28982 ignored=384
2018-03-23T08:55:24.362700551Z INFO [03-23|08:55:24] Imported new block receipts              count=5   elapsed=1.851ms   number=2996781 hash=18a3ea…4498db size=83.23kB  ignored=0
2018-03-23T08:55:25.333311757Z WARN [03-23|08:55:25] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:25.333516099Z WARN [03-23|08:55:25] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:34.627601680Z WARN [03-23|08:55:34] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:34.627674070Z WARN [03-23|08:55:34] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:43.887613197Z INFO [03-23|08:55:43] Imported new block headers               count=0   elapsed=3.782ms   number=2996973 hash=8e3b31…8b676e ignored=192
2018-03-23T08:55:43.964163832Z INFO [03-23|08:55:43] Imported new block receipts              count=3   elapsed=588.057µs number=2996784 hash=093fed…91edc2 size=18.32kB  ignored=0
2018-03-23T08:55:44.005924406Z INFO [03-23|08:55:44] Imported new block headers               count=0   elapsed=4.710ms   number=2997165 hash=dd15d0…a94781 ignored=192
2018-03-23T08:55:44.033013111Z INFO [03-23|08:55:44] Imported new block receipts              count=2   elapsed=484.425µs number=2996786 hash=5370b9…b0d947 size=13.54kB  ignored=0
2018-03-23T08:55:45.046573280Z WARN [03-23|08:55:45] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:45.046749367Z WARN [03-23|08:55:45] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:54.067375081Z INFO [03-23|08:55:54] Imported new block headers               count=0   elapsed=4.115ms   number=2996978 hash=8618dd…3537b7 ignored=192
2018-03-23T08:55:54.100558769Z INFO [03-23|08:55:54] Imported new block headers               count=0   elapsed=18.261ms  number=2997170 hash=82d581…be02f2 ignored=192
2018-03-23T08:55:54.224497038Z INFO [03-23|08:55:54] Imported new block receipts              count=5   elapsed=809.618µs number=2996791 hash=b71473…7f17c2 size=32.31kB  ignored=0
2018-03-23T08:55:57.838211367Z WARN [03-23|08:55:57] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:55:57.838380589Z WARN [03-23|08:55:57] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:04.521672156Z WARN [03-23|08:56:04] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:04.521797537Z WARN [03-23|08:56:04] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:13.983064566Z INFO [03-23|08:56:13] Imported new block headers               count=0   elapsed=3.966ms   number=2996983 hash=4b2abd…2af1ed ignored=192
2018-03-23T08:56:14.035485596Z INFO [03-23|08:56:14] Imported new block headers               count=0   elapsed=4.877ms   number=2997175 hash=03ae98…13ab8b ignored=192
2018-03-23T08:56:14.132057678Z INFO [03-23|08:56:14] Imported new block receipts              count=2   elapsed=9.459ms   number=2996793 hash=117b1a…574c05 size=26.71kB  ignored=0
2018-03-23T08:56:14.132220061Z INFO [03-23|08:56:14] Imported new block receipts              count=1   elapsed=121.328µs number=2996794 hash=25d464…0a7d5e size=1.56kB   ignored=0
2018-03-23T08:56:14.172991313Z WARN [03-23|08:56:14] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:14.173091601Z WARN [03-23|08:56:14] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:24.424197098Z INFO [03-23|08:56:24] Imported new block headers               count=0   elapsed=9.660ms   number=2997178 hash=0e927e…8696d4 ignored=384
2018-03-23T08:56:24.679126325Z INFO [03-23|08:56:24] Imported new block receipts              count=4   elapsed=628.582µs number=2996798 hash=526d31…c84a1f size=21.93kB  ignored=0
2018-03-23T08:56:24.965919545Z WARN [03-23|08:56:24] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:24.966022358Z WARN [03-23|08:56:24] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (3 tries, 3 peers)"
2018-03-23T08:56:33.984746844Z WARN [03-23|08:56:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:56:33.984934150Z WARN [03-23|08:56:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:56:44.109381003Z INFO [03-23|08:56:44] Imported new block headers               count=0   elapsed=7.472ms   number=2997182 hash=06cf5a…9226dd ignored=384
2018-03-23T08:56:44.250830650Z INFO [03-23|08:56:44] Imported new block receipts              count=2   elapsed=889.358µs number=2996800 hash=989ccf…e35d61 size=26.57kB  ignored=0
2018-03-23T08:56:44.252742678Z INFO [03-23|08:56:44] Imported new block receipts              count=2   elapsed=287.342µs number=2996802 hash=a6d108…c2801b size=7.57kB   ignored=0
2018-03-23T08:56:44.323613624Z INFO [03-23|08:56:44] Imported new block headers               count=0   elapsed=39.799ms  number=2998334 hash=22c555…5fae7a ignored=1152
2018-03-23T08:56:44.402136633Z INFO [03-23|08:56:44] Imported new block headers               count=0   elapsed=17.928ms  number=2998910 hash=5ee3c0…eedee7 ignored=576
2018-03-23T08:56:45.936672653Z INFO [03-23|08:56:45] Imported new block receipts              count=449 elapsed=41.607ms  number=2997251 hash=906e10…23b1bc size=1.74mB   ignored=0
2018-03-23T08:56:46.087419953Z INFO [03-23|08:56:46] Imported new block receipts              count=71  elapsed=8.207ms   number=2997322 hash=9cfbd9…02ce98 size=361.54kB ignored=0
2018-03-23T08:56:46.128642792Z INFO [03-23|08:56:46] Imported new block receipts              count=222 elapsed=22.829ms  number=2997544 hash=a90fae…9a0609 size=909.61kB ignored=0
2018-03-23T08:56:46.228589880Z WARN [03-23|08:56:46] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:56:46.231419471Z WARN [03-23|08:56:46] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:56:54.590091866Z WARN [03-23|08:56:54] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:56:54.601176617Z INFO [03-23|08:56:54] Imported new block headers               count=0   elapsed=6.405ms   number=2997736 hash=1dfd67…12b1ee ignored=192
2018-03-23T08:56:54.602387102Z WARN [03-23|08:56:54] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:04.043559276Z INFO [03-23|08:57:04] Imported new block headers               count=0   elapsed=10.066ms  number=2997928 hash=9220ba…ed6704 ignored=384
2018-03-23T08:57:04.144717477Z INFO [03-23|08:57:04] Imported new block receipts              count=5   elapsed=414.411µs number=2997549 hash=7e24be…c8c903 size=4.86kB   ignored=0
2018-03-23T08:57:04.186436425Z INFO [03-23|08:57:04] Imported new block receipts              count=2   elapsed=288.445µs number=2997551 hash=4472dc…fc60c8 size=5.83kB   ignored=0
2018-03-23T08:57:04.326318637Z WARN [03-23|08:57:04] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:04.328462505Z WARN [03-23|08:57:04] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:14.855004722Z INFO [03-23|08:57:14] Imported new block headers               count=0   elapsed=3.724ms   number=2997743 hash=c91935…d1a5f4 ignored=192
2018-03-23T08:57:14.975804845Z INFO [03-23|08:57:14] Imported new block headers               count=0   elapsed=5.012ms   number=2997935 hash=7fac1c…21f920 ignored=192
2018-03-23T08:57:15.216660060Z INFO [03-23|08:57:15] Imported new block headers               count=0   elapsed=15.425ms  number=2998319 hash=450522…9b7cbd ignored=384
2018-03-23T08:57:15.378393319Z WARN [03-23|08:57:15] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:15.473274759Z INFO [03-23|08:57:15] Imported new block headers               count=0   elapsed=67.742ms  number=3000047 hash=ecd395…9d8a37 ignored=1728
2018-03-23T08:57:15.482860829Z WARN [03-23|08:57:15] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:19.602089615Z INFO [03-23|08:57:19] Imported new block headers               count=0   elapsed=5.402ms   number=2997743 hash=c91935…d1a5f4 ignored=192
2018-03-23T08:57:19.916238599Z INFO [03-23|08:57:19] Imported new block receipts              count=7   elapsed=1.266ms   number=2997558 hash=41ea40…0fa70d size=14.99kB  ignored=0
2018-03-23T08:57:19.939042348Z INFO [03-23|08:57:19] Imported new block headers               count=0   elapsed=15.762ms  number=2998319 hash=450522…9b7cbd ignored=576
2018-03-23T08:57:24.934173995Z WARN [03-23|08:57:24] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:25.081654135Z INFO [03-23|08:57:25] Imported new block headers               count=0   elapsed=110.828ms number=3000367 hash=2874d8…2c30ed ignored=2048
2018-03-23T08:57:25.353860826Z INFO [03-23|08:57:25] Imported new block receipts              count=761 elapsed=426.244ms number=2998319 hash=450522…9b7cbd size=4.95mB   ignored=0
2018-03-23T08:57:25.520569276Z INFO [03-23|08:57:25] Imported new block headers               count=485 elapsed=371.505ms number=3002415 hash=52a122…6bc8df ignored=1563
2018-03-23T08:57:25.551729477Z WARN [03-23|08:57:25] Rolled back headers                      count=485 header=3002415->3001930 fast=2998319->2998319 block=0->0
2018-03-23T08:57:25.552382840Z WARN [03-23|08:57:25] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:28.030224723Z INFO [03-23|08:57:28] Imported new block headers               count=0   elapsed=6.301ms   number=2998511 hash=10ee4e…493d00 ignored=192
2018-03-23T08:57:28.141148367Z INFO [03-23|08:57:28] Imported new block headers               count=0   elapsed=6.745ms   number=2998703 hash=d40b01…b2785e ignored=192
2018-03-23T08:57:28.276364231Z INFO [03-23|08:57:28] Imported new block receipts              count=7   elapsed=457.036µs number=2998326 hash=248bd0…9edc75 size=8.69kB   ignored=0
2018-03-23T08:57:30.394619689Z WARN [03-23|08:57:30] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:30.395907297Z WARN [03-23|08:57:30] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:32.352788234Z INFO [03-23|08:57:32] Imported new block headers               count=0   elapsed=6.519ms   number=2998518 hash=e4c9c4…5fd568 ignored=192
2018-03-23T08:57:33.002415337Z INFO [03-23|08:57:33] Imported new block receipts              count=3   elapsed=754.58µs  number=2998329 hash=5650df…e535c9 size=29.10kB  ignored=0
2018-03-23T08:57:33.530601422Z INFO [03-23|08:57:33] Imported new block receipts              count=189 elapsed=26.823ms  number=2998518 hash=e4c9c4…5fd568 size=585.40kB ignored=0
2018-03-23T08:57:33.593773543Z WARN [03-23|08:57:33] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:33.594711013Z INFO [03-23|08:57:33] Imported new block headers               count=0   elapsed=70.194ms  number=3000566 hash=177f06…3a5a10 ignored=2048
2018-03-23T08:57:33.606224755Z WARN [03-23|08:57:33] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:36.098072262Z INFO [03-23|08:57:36] Imported new block headers               count=0   elapsed=5.216ms   number=2998710 hash=316d8a…d79f80 ignored=192
2018-03-23T08:57:36.150104789Z INFO [03-23|08:57:36] Imported new block headers               count=0   elapsed=5.383ms   number=2998902 hash=569821…349e7d ignored=192
2018-03-23T08:57:36.156180744Z WARN [03-23|08:57:36] Node data write error                    err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:36.156268512Z WARN [03-23|08:57:36] Synchronisation failed, retrying         err="state node 4ef0e6…525bc4 failed with all peers (5 tries, 5 peers)"
2018-03-23T08:57:36.681773726Z WARN [03-23|08:57:36] Synchronisation failed, dropping peer    peer=a6bef54e4cb56bbb err="action from bad peer ignored"
2018-03-23T08:57:43.644424463Z INFO [03-23|08:57:43] Imported new block headers               count=0   elapsed=4.954ms   number=2998710 hash=316d8a…d79f80 ignored=192
2018-03-23T08:57:43.725809402Z INFO [03-23|08:57:43] Imported new block receipts              count=2   elapsed=243.137µs number=2998520 hash=8cf526…a6ebfa size=2.56kB   ignored=0
2018-03-23T08:57:44.336399843Z INFO [03-23|08:57:44] Imported new block receipts              count=142 elapsed=35.509ms  number=2998662 hash=c9f4f6…b0223b size=423.80kB ignored=0
2018-03-23T08:57:44.420128592Z INFO [03-23|08:57:44] Imported new block headers               count=0   elapsed=79.288ms  number=3000758 hash=8fb6bb…d9c72d ignored=2048
2018-03-23T08:57:44.459251522Z INFO [03-23|08:57:44] Imported new block headers               count=0   elapsed=10.810ms  number=3001014 hash=7af6d5…bdf2cd ignored=256
2018-03-23T08:57:44.474478262Z INFO [03-23|08:57:44] Imported new block headers               count=0   elapsed=10.908ms  number=3001206 hash=04f763…57ecfc ignored=192
2018-03-23T08:57:44.576312942Z INFO [03-23|08:57:44] Imported new block headers               count=0   elapsed=10.753ms  number=3001590 hash=91bbc2…57dd07 ignored=384
2018-03-23T08:57:44.734951868Z WARN [03-23|08:57:44] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:44.736137051Z WARN [03-23|08:57:44] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:53.725476000Z INFO [03-23|08:57:53] Imported new block headers               count=0   elapsed=7.601ms   number=2998854 hash=4dc9f3…d1cb1b ignored=192
2018-03-23T08:57:55.089747304Z WARN [03-23|08:57:55] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:57:55.089932628Z WARN [03-23|08:57:55] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:04.578736061Z WARN [03-23|08:58:04] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:04.578799950Z WARN [03-23|08:58:04] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:13.932759151Z INFO [03-23|08:58:13] Imported new block headers               count=0   elapsed=4.845ms   number=2998854 hash=4dc9f3…d1cb1b ignored=192
2018-03-23T08:58:15.015819561Z INFO [03-23|08:58:15] Imported new block receipts              count=3   elapsed=263.627µs number=2998665 hash=2f85e3…359404 size=2.27kB   ignored=0
2018-03-23T08:58:15.056406851Z INFO [03-23|08:58:15] Imported new block receipts              count=189 elapsed=35.623ms  number=2998854 hash=4dc9f3…d1cb1b size=656.87kB ignored=0
2018-03-23T08:58:15.124570060Z INFO [03-23|08:58:15] Imported new block headers               count=0   elapsed=68.954ms  number=3000902 hash=9ef14b…533915 ignored=2048
2018-03-23T08:58:15.183305720Z WARN [03-23|08:58:15] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:15.301196762Z INFO [03-23|08:58:15] Imported new block headers               count=535 elapsed=140.698ms number=3002950 hash=6c3468…0492d5 ignored=1513
2018-03-23T08:58:15.330845671Z WARN [03-23|08:58:15] Rolled back headers                      count=535 header=3002950->3002415 fast=2998854->2998854 block=0->0
2018-03-23T08:58:15.330865434Z WARN [03-23|08:58:15] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:23.646073636Z INFO [03-23|08:58:23] Imported new block headers               count=0   elapsed=5.038ms   number=2999046 hash=32f930…fac81f ignored=192
2018-03-23T08:58:23.871591655Z INFO [03-23|08:58:23] Imported new block receipts              count=5   elapsed=794.373µs number=2998859 hash=6a1e1b…e34e59 size=20.92kB  ignored=0
2018-03-23T08:58:29.939885097Z INFO [03-23|08:58:29] Imported new block receipts              count=187 elapsed=24.339ms  number=2999046 hash=32f930…fac81f size=700.39kB ignored=0
2018-03-23T08:58:30.025026843Z INFO [03-23|08:58:30] Imported new block headers               count=0   elapsed=66.217ms  number=3001094 hash=4c25e8…bc461e ignored=2048
2018-03-23T08:58:30.142040026Z INFO [03-23|08:58:30] Imported new block headers               count=192 elapsed=86.451ms  number=3003142 hash=02c9a2…eb8499 ignored=1856
2018-03-23T08:58:30.504591303Z INFO [03-23|08:58:30] Imported new block receipts              count=106 elapsed=16.307ms  number=2999152 hash=37a412…40127e size=325.40kB ignored=0
2018-03-23T08:58:30.523237671Z INFO [03-23|08:58:30] Imported new block headers               count=2048 elapsed=341.262ms number=3005190 hash=2f6a8a…3523a9 ignored=0
2018-03-23T08:58:30.815576156Z INFO [03-23|08:58:30] Imported new block receipts              count=76   elapsed=10.522ms  number=2999228 hash=a3693b…1195bf size=183.41kB ignored=0
2018-03-23T08:58:30.924303047Z INFO [03-23|08:58:30] Imported new block headers               count=2048 elapsed=363.932ms number=3007238 hash=622f78…981c5c ignored=0
2018-03-23T08:58:31.115262607Z INFO [03-23|08:58:31] Imported new block receipts              count=319  elapsed=54.379ms  number=2999547 hash=77abd9…3af2ff size=871.96kB ignored=0
2018-03-23T08:58:31.321028106Z INFO [03-23|08:58:31] Imported new block headers               count=2048 elapsed=354.759ms number=3009286 hash=6839df…25db11 ignored=0
2018-03-23T08:58:31.599147157Z INFO [03-23|08:58:31] Imported new block receipts              count=287  elapsed=41.327ms  number=2999834 hash=716ffb…a6f310 size=1.08mB   ignored=0
2018-03-23T08:58:31.720922479Z INFO [03-23|08:58:31] Imported new block headers               count=2048 elapsed=360.562ms number=3011334 hash=44b471…b14916 ignored=0
2018-03-23T08:58:31.825917935Z WARN [03-23|08:58:31] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:32.094233124Z INFO [03-23|08:58:32] Imported new block headers               count=2048 elapsed=333.462ms number=3013382 hash=625ffa…c511e5 ignored=0
2018-03-23T08:58:32.185667811Z WARN [03-23|08:58:32] Rolled back headers                      count=2048 header=3013382->3011334 fast=2999834->2999834 block=0->0
2018-03-23T08:58:32.185750569Z WARN [03-23|08:58:32] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:33.824233312Z INFO [03-23|08:58:33] Imported new block headers               count=0    elapsed=5.210ms   number=3000026 hash=181d53…c6fd3e ignored=192
2018-03-23T08:58:33.873305043Z INFO [03-23|08:58:33] Imported new block headers               count=0    elapsed=10.671ms  number=3000410 hash=950115…d3da84 ignored=384
2018-03-23T08:58:34.318227341Z WARN [03-23|08:58:34] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:34.319287351Z WARN [03-23|08:58:34] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:45.691980648Z WARN [03-23|08:58:45] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:45.692116757Z WARN [03-23|08:58:45] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:53.681344191Z INFO [03-23|08:58:53] Imported new block headers               count=0    elapsed=5.163ms   number=3000026 hash=181d53…c6fd3e ignored=192
2018-03-23T08:58:53.991278148Z WARN [03-23|08:58:53] Node data write error                    err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
2018-03-23T08:58:53.996931870Z WARN [03-23|08:58:53] Synchronisation failed, retrying         err="state node 80e90b…69c5ae failed with all peers (4 tries, 4 peers)"
@TediPapajorgji
Copy link

TediPapajorgji commented Mar 29, 2018

Also receiving this issue. Geth slowly eats up all available ram on the system until it crashes.

Edit:
Ended up increasing the server ram to 8gb, things are running okay.

@sapph1re
Copy link

sapph1re commented Mar 30, 2018

Having the same issue, built from the latest 1.8.3 source, Linux, RAM 2 GB. Sync mode fast, Ropsten network.
Is it possible to limit geth's RAM usage instead of increasing the server RAM?

UPD: adding a 8 GB swap file didn't help. Same behaviour: eats up all available memory and dies.

@sapph1re
Copy link

Also, has this problem appeared only in 1.8.2? Will reverting to 1.8.1 or 1.8.0 help?

@TediPapajorgji
Copy link

TediPapajorgji commented Mar 30, 2018

Hey Sapph1re, I dont think there's anyway to limit geth's ram usage at the current moment.
Also not too sure if reverting to an older version will help. Although this being said, after upgrading to 8gb of ram my geth has not crashed and it's ram usage seems to have stabilized around 4gb.

I've included the systemd service file that i'm using to launch geth, it may be of use to you:

[Unit]
Description=The Ethereum Blockchain Command Line Interface
Documentation=man:geth(1)
After=network.target

[Service]
User=ethereum
ExecStart=/usr/bin/geth --syncmode "fast" --cache 1024 --rpcaddr "0.0.0.0" --rpcport 9551 --rpc --rpcapi personal,eth,ssh,web3 --rpccorsdomain "52.224.54.206" 2>&1 >> /home/ethereum/geth.l$
Restart=always

[Install]
WantedBy=default.target

Server specifications:
Ubuntu 16.04.4 x64
8 GB Memory
160 GB SSD Disk
4 vCPU's
digitalocean.com droplet

@ekryski
Copy link

ekryski commented Apr 13, 2018

Setting the --cache flag is supposed to do it but there is either a mem-leak or it's not respecting it. We've also noticed this both in fast-sync mode and proper full-sync mode on mainnet and ropsten testnet.

In general we've noticed that when syncing Geth will eat up as much memory as there is available on the machine. I don't think this an issue with just v1.8.2. We've seen it all the back to v1.7.x.

Once in sync the memory and CPU profile of Geth drastically decreases but it spikes every time a block is verified (for obvious reasons). So you need to give it quite a bit more than your --cache value. In, general we've been seeing things stabilizing at whatever your cache value is + 3GB.

@dbblackdiamond
Copy link

I am having an "out of memory" issue with Geth 1.8.9 that is not triggered by fast sync, but just by running Geth. Whenever I start Geth, if I let it run, after a while, it will crash with an "out of memory" error message. My Ubutun Xenial server is configured with 8GB of RAM and 32GB of swap. The command line to start Geth is the following: go-ethereum/build/bin/geth --nodiscover --cache=2048 --rpc --rpcaddr 0.0.0.0 --rpccorsdomain "*" --rpcapi "db,eth,net,web3,personal,debug,txpool" js ./ethereum/mine_pending.js

i used to run Geth 1.6.x with the same command and never had an issue with it running out of memory. I have actually tripled the swap space on this server to accommodate the "out of memory" issue and it is still there.

@ekryski
Copy link

ekryski commented Jun 15, 2018

Yeah. Running more nodes, this is becoming a real problem. Both in ropsten testnet and mainnet. There is a memory leak somewhere. Geth continually eats up memory until it runs out of machine memory.

Seeing this in v1.8.2 and later versions. But we haven't been able to upgrade passed v1.8.2 due to #16846

We're running on Kubernetes so this is particularly painful because our Geth nodes continually get restarted.

@sulliwane
Copy link
Author

I should probably have mentioned that I'm running 3 geth dockers on one single machine (Mainnet, Ropsten, Rinkeby).

@gregbkr
Copy link

gregbkr commented Sep 28, 2018

I notice that it crash after few hours only once I activate --rpcapi "admin,db,eth,miner,net,web3,personal,txpool"
Without that argument, geth is running fine on ropsten.

The failing setup:

cat docker-compose.yml
version: "3"

services:

  geth_testnet:
    container_name: geth_testnet
    image: ethereum/client-go:v1.8.16
    command: '--testnet --syncmode "fast" --rpc --rpcaddr 0.0.0.0 --rpccorsdomain "*" --rpcvhosts "*" --rpcapi "admin,db,eth,miner,net,web3,personal,txpool"'
    ports:
      - "8545:8545"
      - "30303:30303"
      - "30303:30303/udp"
    volumes:
      - testnet_data:/root/.ethereum

volumes:
  testnet_data:

Hope that's help!

@TediPapajorgji
Copy link

I notice that it crash after few hours only once I activate --rpcapi "admin,db,eth,miner,net,web3,personal,txpool"
Without that argument, geth is running fine on ropsten.

The failing setup:

cat docker-compose.yml
version: "3"

services:

  geth_testnet:
    container_name: geth_testnet
    image: ethereum/client-go:v1.8.16
    command: '--testnet --syncmode "fast" --rpc --rpcaddr 0.0.0.0 --rpccorsdomain "*" --rpcvhosts "*" --rpcapi "admin,db,eth,miner,net,web3,personal,txpool"'
    ports:
      - "8545:8545"
      - "30303:30303"
      - "30303:30303/udp"
    volumes:
      - testnet_data:/root/.ethereum

volumes:
  testnet_data:

Hope that's help!

Ah thats interesting! I upgraded my node to 16gb of memory and geth has yet to crash on me anymore. The memory limit flag doesn't actually limit geth's memory usage it seems.

@eliceaas
Copy link

same issue.
when i change from centos7.4 to ubuntu16.04 and change the rpcport from 8581 to 8681. it seems to be stoped! i think the rpcport may be the main case.

here is my specifications:

Server
Ubuntu 16.04.4 x64
4 GB Memory
40 GB HDD Disk
2 vCPU's

Geth
Version: 1.8.16-stable
Architecture: amd64
Go Version: go1.11.1

@matingwei
Copy link

matingwei commented Oct 17, 2018 via email

@hashfury42
Copy link

hi, guys, I also encountered this problem. But when I removed the "--rpc" config ,geth works well. I think the RPC service is the main problem, and hope someone can fix this.

@holiman
Copy link
Contributor

holiman commented Oct 17, 2018

The main problem is most likely attackers on the internet discovering the rpc service and doing bruteforce password guessing against personal.unlock. Luckily, decrypting keystore is very memory intensive.

Note: this doesn't seem to be the case for the original reporter, but very likely for those of you that experience problems only when rpc is enabled.

@hashfury42
Copy link

hashfury42 commented Oct 18, 2018

The main problem is most likely attackers on the internet discovering the rpc service and doing bruteforce password guessing against personal.unlock. Luckily, decrypting keystore is very memory intensive.

Note: this doesn't seem to be the case for the original reporter, but very likely for those of you that experience problems only when rpc is enabled.

hi, holiman
I set up a firewall to proctect the RPC port, and this problem has disappeared.
Thank you very much, and I hope this will help someone else who encountered this problem. 👍

@NIVJAIN
Copy link

NIVJAIN commented Oct 22, 2018

@hashfury42 could you please share how did you setup your firewall on port, did you allow only particular ip to have access to that port?

1 similar comment
@NIVJAIN
Copy link

NIVJAIN commented Oct 22, 2018

@hashfury42 could you please share how did you setup your firewall on port, did you allow only particular ip to have access to that port?

@hashfury42
Copy link

@hashfury42 could you please share how did you setup your firewall on port, did you allow only particular ip to have access to that port?

Yes, only particular ip have access to that port, if you use Ubuntu you can read this https://help.ubuntu.com/community/UFW

@Ninjamannn
Copy link

@holiman
thanks
in my case this method helped me.
I created a solution on: https://stackoverflow.com/questions/53206228/ethereum-geth-out-of-memory/
I hope it will be useful

@NIVJAIN
Copy link

NIVJAIN commented Nov 13, 2018 via email

@Ninjamannn
Copy link

@NIVJAIN
Hi Jain!

repeat the iptable commands twice with different ip

if I understand correctly, yes. However, I recommend reading about linux iptables. Your main task is to have strictly defined addresses access to your node on the port.

Provided access to node 2 ip addresses do not forget to specify this parameter: -- rpccorsdomain

--rpccorsdomain value Comma separated list of domains from which to accept cross origin requests (browser enforced)

please write about your results...

@hapsody
Copy link

hapsody commented Jan 8, 2019

I'm not sure if it's same problem or not.
I used to be troubled in same issue geth's "out of memory".
so I expanded AWS Server spec and the problem was gone.
but new issue came, CPU and MEM high usage.

The issue is it slowly increases to cpu and mem limit.
it cause AWS Server paralysis

Server
AWS t3.medium(64bit - ubuntu 16.04, 2core, 4GB Ram, 8GB Swap memory)
version : 1.8.1-stable, 1.8.16-stable
fast and light syncmode both.

geth --rinkeby --rpc --rpcaddr "0.0.0.0" --rpcvhosts=* --rpcport "8545" --rpccorsdomain "neojuneELB-1439772252.ap-northeast-2.elb.amazonaws.com" --rpcapi "eth,net,web3,personal,admin" --syncmode "light" --cache "64"

@princesinha19
Copy link
Contributor

princesinha19 commented Feb 21, 2020

It happened with me right now on my private ethereum network. Is there any progress on this?

Geth command:-

geth --rpc --rpcport "8545" --rpcaddr "127.0.0.1" --rpccorsdomain "*" --ws --wsorigins "*" --wsaddr "127.0.0.1" --wsport "8546" --rpcapi "web3,personal,eth,net" --wsapi personal,web3,eth,net,db --cache 2048 console

Log:-

fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x11ecc01, 0x16)
        /build/ethereum-CazSBy/.go/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc194000000, 0x10000000, 0x20db3f8)
        /build/ethereum-CazSBy/.go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x20a9840, 0x10000000, 0x0, 0x882b79)
        /build/ethereum-CazSBy/.go/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0x20a9840, 0x8000, 0x1ffffffff)
        /build/ethereum-CazSBy/.go/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0x20a9840, 0x8000, 0x20db408, 0x181b730)
        /build/ethereum-CazSBy/.go/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0x20a9840, 0x8000, 0x7f0084680101, 0x7f00846879a0)
        /build/ethereum-CazSBy/.go/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
        /build/ethereum-CazSBy/.go/src/runtime/mheap.go:1093 +0x4c
runtime.(*mheap).alloc(0x20a9840, 0x8000, 0x7f0084010101, 0x7f007d7b4eb0)
        /build/ethereum-CazSBy/.go/src/runtime/mheap.go:1092 +0x8a
runtime.largeAlloc(0x10000000, 0x7f007d7b0101, 0x46fabe)

@holiman
Copy link
Contributor

holiman commented Sep 10, 2020

This is a very old ticket, and we've worked on, and fixed many memory issues over time. I'm closing this, it's better if new tickets are opened relating to the more recent versions

@holiman holiman closed this as completed Sep 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests