Resilient ledger recovery

[jumaffre]

Resolves #3459

The recovery process is now resilient to ledger files having been partially written, as opposed to throwing early if an entry wasn't completely written to a ledger file (1.0) or marking a file as .corrupted and ignoring it entirely (main currently). Instead, we recover as many entries as possible when this is the case and truncate the (uncommitted) ledger file if it wasn't recovered entirely.

This is tested by a new end-to-end test in recovery.py which tests that the full recovery procedure can be done on a corrupted ledger (corrupted at the first tx, last tx and first signature) as well as a more precise unit test (see ledger.cpp).

TODO:

• Cleanup ledger unit test
• Add ledger unit test for recovery
• Remove mentions of .corrupted files in docs

[ccf-bot]

truncated_ledger_recovery@42438 aka 20220223.41 vs main ewma over 20 builds from 42121 to 42430

Click to see table

main

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
42121	20220218.24	5867.23	9.00506e+07	19773.3	1.66503e+07	5581.76	1.56017e+07	2542.89	1.03588e+07	1629.23	1.645e+08	2164.03	1.00967e+07	1479.84	9.84392e+07	1968.76	9.04813e+06	17943.9	893929	1.39395e+06	9.39445e+06	3.55556e+07
42144	20220218.32	5924.88	8.97885e+07	19749.4	1.69124e+07	5693.79	1.58639e+07	2523.42	1.0621e+07	1619.45	1.63975e+08	2165.48	1.14074e+07	1460.44	9.89635e+07	1938.67	9.57242e+06	19984.8	907952	1.38377e+06	9.21265e+06	3.58663e+07
42175	20220221.2	6075.56	9.13613e+07	19781	1.63882e+07	5494.48	1.56017e+07	2529.54	1.0621e+07	1624.12	1.61616e+08	2119.01	9.83456e+06	1475.48	9.84392e+07	1971.31	9.31027e+06	18140.9	902755	1.37902e+06	9.4334e+06	3.57417e+07
42191	20220221.7	6051.75	9.03128e+07	20063.8	1.69124e+07	5638.91	1.56017e+07	2540.21	1.08831e+07	1661.68	1.61354e+08	2182.64	9.83456e+06	1491.37	9.84392e+07	1943.5	9.57242e+06	17974.1	894982	1.38284e+06	9.27108e+06	3.56174e+07
42211	20220221.14	5930.94	9.13613e+07	19594.8	1.63882e+07	5652.28	1.56017e+07	2425.32	1.03588e+07	1612.61	1.63975e+08	2158.03	1.03588e+07	1473.15	9.84392e+07	2015	8.78598e+06	19769.4	879795	1.38154e+06	9.26269e+06	3.54939e+07
42229	20220221.20	5790.78	9.10992e+07	19795.7	1.6126e+07	5669.65	1.56017e+07	2402.08	1.0621e+07	1574.69	1.63713e+08	2159.76	1.32424e+07	1464.57	9.87014e+07	1926.65	9.57242e+06	17878.3	880893	1.34976e+06	9.19615e+06	3.5249e+07
42256	20220222.2	6017.97	9.18856e+07	19773.7	1.69124e+07	5594.56	1.56017e+07	2533.2	1.0621e+07	1650.07	1.64237e+08	2167.91	1.00967e+07	1474.53	9.87014e+07	1971.99	9.57242e+06	20985.9	898634	1.37634e+06	9.33876e+06	3.63766e+07
42274	20220222.10	5909.34	9.03128e+07	19948.7	1.69124e+07	5482.21	1.58639e+07	2540.55	1.0621e+07	1624.27	1.61616e+08	2168.01	9.83456e+06	1458.42	9.73907e+07	1812.84	1.0621e+07	20640.9	878213	1.37256e+06	9.29215e+06	3.54933e+07
42288	20220222.16	5744.88	9.05749e+07	19615.6	1.66503e+07	5621.54	1.58639e+07	2535.8	1.00967e+07	1610.04	1.64237e+08	2155.18	9.83456e+06	1393.29	9.87014e+07	1921.31	9.31027e+06	17777.5	913585	1.38218e+06	9.25432e+06	3.56794e+07
42293	20220222.18	5906.28	9.03128e+07	19469.5	1.69124e+07	5618.43	1.56017e+07	2552.77	1.00967e+07	1622.7	1.63975e+08	2200.94	9.31027e+06	1460.63	9.87014e+07	1971.63	9.31027e+06	19807.6	903670	1.39547e+06	9.39876e+06	3.58036e+07
42304	20220222.23	5948.35	9.05749e+07	19954.4	1.58639e+07	5648.99	1.56017e+07	2525.89	1.03588e+07	1637.15	1.63451e+08	2169.94	1.00967e+07	1463.39	9.84392e+07	1926.63	9.31027e+06	20133.1	904307	1.41523e+06	9.32176e+06	3.58042e+07
42313	20220222.27	5982.15	9.13613e+07	19778.6	1.69124e+07	5612.99	1.58639e+07	2527.02	1.03588e+07	1625.02	1.63451e+08	2159.41	1.03588e+07	1468.53	9.84392e+07	1913.15	1.16696e+07	18797	927151	1.37136e+06	9.3601e+06	3.56794e+07
42320	20220222.30	5920.73	9.13613e+07	19699.4	1.76989e+07	5608.83	1.56017e+07	2533.24	1.03588e+07	1624.96	1.63975e+08	2238.46	9.57242e+06	1428.2	9.84392e+07	1930.82	9.31027e+06	19850.5	911995	1.3819e+06	9.30901e+06	3.57411e+07
42335	20220222.37	6022.19	8.97885e+07	19906.9	1.58639e+07	5665.61	1.56017e+07	2591.34	1.03588e+07	1611.78	1.61354e+08	2110.27	1.00967e+07	1475.4	9.79149e+07	1970.6	9.57242e+06	17461.7	892447	1.38555e+06	9.48139e+06	3.64413e+07
42340	20220223.1	5927.85	8.97885e+07	20024.9	1.6126e+07	5653.94	1.56017e+07	2541.52	1.0621e+07	1662.2	1.61092e+08	2174.67	1.03588e+07	1494.42	9.87014e+07	2012.99	9.04813e+06	18159	894558	1.37459e+06	9.30909e+06	3.56174e+07
42353	20220223.7	5912.65	9.00506e+07	20216.4	1.66503e+07	5539.46	1.53396e+07	2536.54	1.0621e+07	1621.06	1.64237e+08	2155.5	1.03588e+07	1459.33	9.89635e+07	1966.69	9.57242e+06	22016.1	875171	1.37726e+06	9.28794e+06	3.59298e+07
42374	20220223.16	5895.23	9.00506e+07	20002.7	1.6126e+07	5581.94	1.56017e+07	2526.5	1.03588e+07	1623.18	1.63713e+08	2212.85	9.04813e+06	1474.99	9.87014e+07	1890.55	9.83456e+06	20607.7	881879	1.38237e+06	9.17555e+06	3.55549e+07
42394	20220223.25	5909.78	9.03128e+07	19555.8	1.69124e+07	5629.48	1.6126e+07	2515.33	1.03588e+07	1619.53	1.63975e+08	2156.91	9.83456e+06	1465.29	9.87014e+07	1971.37	9.57242e+06	19961.8	904629	1.3721e+06	9.21273e+06	3.57411e+07
42411	20220223.32	5902.39	9.05749e+07	19851.4	1.58639e+07	5477.34	1.56017e+07	2478.09	1.27181e+07	1627.1	1.62402e+08	2154.39	9.57242e+06	1469.07	9.84392e+07	1960.46	9.57242e+06	20206.2	885617	1.37283e+06	9.21684e+06	3.56174e+07
42430	20220223.39	5837.66	9.03128e+07	18642.1	1.69124e+07	5588.65	1.58639e+07	2546.43	1.08831e+07	1620.83	1.63713e+08	2198.02	1.00967e+07	1482.28	9.84392e+07	2017.15	8.78598e+06	18115.8	901759	1.36733e+06	9.18382e+06	3.59923e+07

truncated_ledger_recovery

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
42398	20220223.26	5629.75	9.16235e+07	20200.3	1.66503e+07	5643.08	1.56017e+07	2544.44	1.0621e+07	1652.06	1.63975e+08	2195.17	1.08831e+07	1473.36	9.84392e+07	1975.54	9.04813e+06	22044.6	901521	1.38004e+06	9.19206e+06	3.56168e+07
42420	20220223.35	6041.66	9.08371e+07	19452.1	1.66503e+07	4809.5	1.56017e+07	2178.29	9.83456e+06	1438.54	1.61092e+08	1938.15	9.31027e+06	1361.23	9.79149e+07	1957.61	9.57242e+06	22405.4	904788	1.38415e+06	9.28373e+06	3.58669e+07
42438	20220223.41	6040.54	9.08371e+07	19679.4	1.76989e+07	5546.96	1.56017e+07	2529.67	1.08831e+07	1624.79	1.63713e+08	2108.6	1.00967e+07	1440.77	9.87014e+07	1963.69	9.31027e+06	19962	904229	1.36296e+06	9.37725e+06	3.66369e+07