[dbnode] AggregateTiles RPC - minimal E2E flow

[linasm]

What this PR does / why we need it:
A minimal end to end flow: RPC to copy the data from one namespace to another, without any processing, single block only.

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:
NONE

Does this PR require updating code package or user-facing documentation?:
NONE

[notbdu]

nit: returning multi error here and below is a little awkward since these are single errors.

[linasm]

Hmm, looks like this method (coldFlushSingleShard) takes a multiErr argument to accumulate errors across calls. Is that correct, @gediminasgu ?

[gediminasgu]

@notbdu as I see there could be shard ColdFlush, namespace onColdFlush/shard ColdFlush.Done and flushPersist.DoneFlush error. As I see we can get here 3 errors, that's why I used here multiErr in place of a single err. Please let me know if we see it in the same way.

[notbdu]

Hmm, so I haven't seen any prior art where we return a MultiError from a fn call. I think the usage is prob fine. But usually we'd return single errors here and then below we'd accumulate errors in the MultiError and return multiErr.FinalError() at the end which is consistent with the go way of returning an error type from fns.

[gediminasgu]

Makes sense. I'll change it.

[notbdu]

Is there some way to batch these writes?

Also, when doing a large aggregation, wouldn't we starve the CPU and block regular writes? How are we going to manage resource allocation?

[linasm]

@gediminasgu

[notbdu]

So I don't understand the need for cold flushing here but I'm missing a lot of context on this PR. I also think that this cold flushing running simultaneously w/ the regular cold flushing process will cause issues.

[linasm]

@gediminasgu

[notbdu]

Won't the time range of the request sometimes span multiple index blocks?

I think some metrics here on the amount of work its doing would be helpful as well.

[linasm]

Already added a check to ensure that the writes are limited to a single target block. For source blocks (which will be smaller), we will support reading from multiple blocks, but that is being added in a different PR.

[linasm]

Regarding the second comment, @gediminasgu I see you added some write error metrics, perhaps there should be write success metrics as well?

[codecov]

Codecov Report

Merging #2466 into large-tiles-aggregation will not change coverage.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##           large-tiles-aggregation   #2466   +/-   ##
=======================================================
  Coverage                     33.3%   33.3%           
=======================================================
  Files                          799     799           
  Lines                        78559   78559           
=======================================================
  Hits                         26238   26238           
  Misses                       49607   49607           
  Partials                      2714    2714           

Flag	Coverage Δ
#aggregator	57.9% <0.0%> (ø)
#cluster	28.5% <0.0%> (ø)
#collector	41.8% <0.0%> (ø)
#dbnode	43.1% <0.0%> (ø)
#m3em	44.8% <0.0%> (ø)
#m3ninx	46.6% <0.0%> (ø)
#query	1.3% <0.0%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81ce0ed...aecba62. Read the comment docs.