generate_index inserts ideal initial data

[jeffwashington]

Problem

Validator startup time is commonly experienced. Generating the index is a large part of that time.

Summary of Changes

When inserting new items, calculate the actual btreemap 'value' for initial insertion instead of blank values. Then, skip calling 'update' if the pubkey did not exist and we inserted the correct initial data.

Fixes #

[codecov]

Codecov Report

Merging #17247 (3b73a7c) into master (b5302e7) will increase coverage by 0.0%.
The diff coverage is 99.0%.

@@           Coverage Diff           @@
##           master   #17247   +/-   ##
=======================================
  Coverage    82.6%    82.7%           
=======================================
  Files         423      423           
  Lines      118053   118121   +68     
=======================================
+ Hits        97626    97708   +82     
+ Misses      20427    20413   -14     

[jeffwashington]

generate_index total_us=39113701i scan_stores_us=22296476i (this pr)
generate_index total_us=42341212i scan_stores_us=22534402i (master)

lemond colo machine, snapshot-77779424-AP7QKgrw31e5tBhtPqxejbJbYU8rs6LA97L1Qq7gqpdX.tar.zst

[sakridge]

Is there test coverage that would catch a change to update() would require a corresponding change here?

[sakridge]

I think it's questionable that the performance improvement justifies the extra code complexity/risk. Does this build on a further change? The performance increase seems to be ~8%, is that right?

[jeffwashington]

I think it's questionable that the performance improvement justifies the extra code complexity/risk. Does this build on a further change? The performance increase seems to be ~8%, is that right?

this pr builds on the initial data change:
and is now approaching 2x faster overall from master right now

[jeffwashington]

Is there test coverage that would catch a change to update() would require a corresponding change here?

I can rework this to make sure that is tested well.

[jeffwashington]

discussed with @carllin in context of other changes going on.

[jeffwashington]

updated results:

this change:
generate_index total_us=40146488i scan_stores_us=11513893i

master:
generate_index total_us=45073105i scan_stores_us=11413174i

[jeffwashington]

@carllin fyi. reworked this extensively with stephen.