Stop checkpoint validation when encountering a valid checkpoint

[the-mikedavis]

@mkuratczyk noticed that with many QQs on the qq-v4 branch and each QQ having many checkpoints, we spend a fair amount of effort reading the checkpoints during recovery. This is because ra_snapshot:find_checkpoints/1 uses the ra_snapshot:validate/1 callback to ensure that each snapshot is valid. validate/1 is somewhat expensive in ra_log_snapshot since it fully reads and decodes the checkpoint, discarding the result.

Not all of this validation is necessary: we can stop validating checkpoints when we find the latest checkpoint which is valid. This is likely to be good enough. I've also updated find_checkpoints/1 to stop its search when it finds a checkpoint with a lower index than the current snapshot as any checkpoints lower than the snapshot index won't be used for promotion and should be removed. For many QQs with many checkpoints each this should save some I/O usage and memory.

[the-mikedavis]

I took some rough measurements with tprof from OTP 27. The gist is that time and memory savings look pretty good: 1.62s down to 0.28s and 178 million words of memory down to ~20 million for ra_snapshot:init/6 on a QQ's checkpoint directory (from the qq-v4 branch) with 5 million messages.

Results...

Queue created with perf-test -qq -u qq -x 1 -y 0 -C 5000000 -c 3000

Measured with:

tprof:profile(fun() -> ra_snapshot:init(<<"uuid">>, ra_log_snapshot, "./snapshots", "./checkpoints", undefined, 3) end, #{type => call_time}).

and #{type => call_memory} for the memory breakdowns.

This branch:

FUNCTION                                        CALLS  TIME (μs)  PER CALL  [    %]
...
erlang:universaltime_to_localtime/1                 6         69     11.50  [ 0.02]
prim_file:close_nif/1                              17         91      5.35  [ 0.03]
prim_file:list_dir_nif/1                            2         92     46.00  [ 0.03]
prim_file:read_nif/2                               34        155      4.56  [ 0.05]
file:file_name_1/2                               1037        192      0.19  [ 0.07]
filename:join1/4                                 1914        203      0.11  [ 0.07]
prim_file:open_nif/2                               17        291     17.12  [ 0.10]
erlang:crc32/1                                      1      13109  13109.00  [ 4.54]
prim_file:read_file_nif/1                           1      52475  52475.00  [18.18]
ra_log_snapshot:parse_snapshot/1                    1      93138  93138.00  [32.26]
erlang:binary_to_term/1                            19     128266   6750.84  [44.43]
                                                          288697            [100.0]

0.28s

FUNCTION                                CALLS     WORDS    PER CALL  [    %]
...
prim_file:internal_native2name/1           17      1122       66.00  [ 0.01]
file:file_name_1/2                       1037      2040        1.97  [ 0.01]
lists:reverse/2                            42      3726       88.71  [ 0.02]
filename:join1/4                         1914      3758        1.96  [ 0.02]
erlang:crc32/1                              1      7450     7450.00  [ 0.04]
erlang:binary_to_term/1                    19  19871694  1045878.63  [99.89]
                                               19892656              [100.0]

---

main:

FUNCTION                                        CALLS  TIME (μs)  PER CALL  [    %]
...
erlang:universaltime_to_localtime/1                 6         71     11.83  [ 0.00]
prim_file:list_dir_nif/1                            2         93     46.50  [ 0.01]
prim_file:close_nif/1                              17        165      9.71  [ 0.01]
file:file_name_1/2                               1037        168      0.16  [ 0.01]
prim_file:read_nif/2                               34        190      5.59  [ 0.01]
filename:join1/4                                 2890        249      0.09  [ 0.02]
prim_file:open_nif/2                               17        704     41.41  [ 0.04]
erlang:crc32/1                                     17     141498   8323.41  [ 8.72]
prim_file:read_file_nif/1                          17     154438   9084.59  [ 9.52]
ra_log_snapshot:parse_snapshot/1                   17     520271  30604.18  [32.08]
erlang:binary_to_term/1                            51     803040  15745.88  [49.51]
                                                         1621975            [100.0]

FUNCTION                                    CALLS      WORDS    PER CALL  [    %]
...
prim_file:internal_native2name/1               17       1122       66.00  [ 0.00]
file:file_name_1/2                           1037       2040        1.97  [ 0.00]
lists:reverse/2                                57       5552       97.40  [ 0.00]
filename:join1/4                             2890       5678        1.96  [ 0.00]
erlang:crc32/1                                 17      66600     3917.65  [ 0.04]
erlang:binary_to_term/1                        51  177721626  3484737.76  [99.95]
                                                   177806402              [100.0]

[kjnilsson]

Other checkpoints we can validate during promotion and discard ones that fai

For a quorum queue were consumers keep up with ingress checkpoints are promoted very often. It would be nice not to to have to do the validation work every time just because we optimised recovery. My thought was that once we'd found a valid checkpoint during recovery we'd assume all prior checkpoints are also valid. That should be roughly as good as promoting any other checkpoint.

The most likely way a checkpoint would become corrupted is if the server hard stopped during a write or fsync. Sure there are other ways checkpoints could become corrupted but at least we guard against the most likely one.

[kjnilsson]

a few minor things