add start-es-inserts-after-block option to elasticsearch plugin

[oxarbitrage]

Due to the blockchain halt we recently had all ES nodes stopped working and replay is required.

This takes a huge amount of time. Most of the data is already on ES so i am thinking on an option start-es-inserts-after-block or similar that will start the heavy work after a block number passed, resulting in a faster synchronization.

As some important business rely on the ES plugin for their business, i propose that feature or something to speed up the process to be added.

Looking for your opinions.

Tasks:

• start-es-after-block
• [ ] dump-es-at-block

[crypto-1]

This feature would be very useful for CryptoBridge.

[clockworkgr]

yaaaaaaaaaaaaaaaaaaay!

I pretty much suggested the same thing earlier today in TG Node Admins...

[cogutvalera]

Very nice idea !

[sschiessl-bcp]

Could an ES dump be provided as well that matches es-inserts-after-block=32600000?

That would be asweome!

[oxarbitrage]

Could an ES dump be provided as well that matches es-inserts-after-block=32600000?

This is also a good idea. I added 2 tasks to this issue to do start-es-at-block and dump-es-at-block

[sschiessl-bcp]

I suppose if the backend creates the dump, it could also read it, i.e. 'import-es-on-block'

Just to say it, intent was to use the native dump and import es functional

[sschiessl-bcp]

Maybe replace 'at' with 'before' or 'after', so the timeline is clear with respect to the block

[clockworkgr]

i see no need to add this to the plugin...just documentation of how to use native ES dump/import

And maybe we can provide weekly ES snapshots (those of us running a node)

[sschiessl-bcp]

No need maybe, but it's certainly nice. Install ES in default, rest is done by plugin.

[clockworkgr]

where would the plugin get the dump from?

[sschiessl-bcp]

That command would have a file, or github reference to e.g. BitShares/bitshares-core-dumps as parameter

[oxarbitrage]

• i removed the dump option from the task list, lets keep it simple. i agree with @clockworkgr; it is probably better to use ES native tools or third parties(https://github.com/taskrabbit/elasticsearch-dump) to do this because also if we export we will want to import and that can start to became more complicated without need.
• changed the option to start-es-after-block

[Agorise]

As we are bringing entire cities onto point-of-sale now, any down time could cost merchants dearly, so if this will help get nodes up faster, I'm all for it.

[oxarbitrage]

Closed by #1455

Now ES nodes can recover from crashes in minutes by start saving data only after a specific block pass.

[clockworkgr]

Closed by #1455

Now ES nodes can recover from crashes in minutes by start saving data only after a specific block pass.

closed by #1458... 1455 is this issue

[oxarbitrage]

oops, sorry and thanks.