Fix issues with snapshot generation for CHAMP

[jumaffre]

Fix for the issue that recently cropped up in the CI: https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=57709&view=logs&j=8f3dc89c-3708-5926-47e7-27120a268dab&t=bb1a7e6d-8f5b-56e4-638c-b498b20b4b62&l=19514

The underlying KV state data structure (CHAMP) keeps track of its total serialised size when inserting/removing keys so that a snapshot can be efficiently allocated when necessary. However, the size calculation was wrong in some niche cases depending on the put()/remove() patterns in a map.

This is now fixed, with a much deeper coverage of this in the map_test unit test.

[ccf-bot]

champ_snapshot_fix@58173 aka 20221214.34 vs main ewma over 20 builds from 57514 to 58130

Click to see table

main

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	ls_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_virtual_cft^	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	pi_ls_jwt_virtual_cft^	ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_jwt_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	hist_sgx_cft^	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
57514	20221208.23	0.786024	17166.7	43803.7	6335.16	8.27106e+07	15440.4	1.50774e+07	44691.6	15528.3	8.78598e+06	14107.5	12566.4	4566.39	3677.88	3429.24	5823	7.47526e+06	5886.13	1.53396e+07	23022	1567.88	7.99955e+06	1288.49	7.47526e+06	1305.99	6.95098e+06	834150	1.17979e+06	8.04134e+06	3.07882e+07
57575	20221208.40	0.757691	17522.3	48265	6345.34	8.29727e+07	15478.3	1.50774e+07	45168.4	15508.1	8.78598e+06	14208.6	12485.6	4552.48	3556.59	3458.1	5808.8	7.73741e+06	5874.94	1.53396e+07	22964.9	1566.48	7.99955e+06	1290.16	7.99955e+06	1303.72	7.21312e+06	831599	1.17802e+06	8.15628e+06	3.07651e+07
57584	20221209.2	0.78448	17530.6	43951.5	6282.45	8.21863e+07	15363	1.6126e+07	47452.4	15481	8.78598e+06	13259.8	14092.9	4366.84	3525.13	3496.66	5795.2	7.73741e+06	5807.67	1.53396e+07	27357.2	1556.1	7.99955e+06	1286.1	7.47526e+06	1300.71	7.21312e+06	831868	1.18326e+06	8.15079e+06	3.08063e+07
57612	20221209.10	0.783278	17485.6	45764.4	6328.1	8.27106e+07	15464	1.53396e+07	47107	15528.6	8.78598e+06	14228.5	12401.9	4502.52	3521.64	3440.94	5808.9	7.21312e+06	5836.55	1.53396e+07	28844.9	1560.12	7.99955e+06	1284.84	7.47526e+06	1286.07	7.21312e+06	839803	1.17423e+06	8.13923e+06	3.0829e+07
57631	20221209.16	0.769375	17341.6	43665.4	6252.83	8.19242e+07	15421.6	1.58639e+07	46388.6	15520.9	8.52384e+06	13046.9	12589	4461.23	3513.64	3350.53	5822.4	7.73741e+06	5791.83	1.53396e+07	28237.3	1567.46	7.99955e+06	1286.85	7.47526e+06	1305.43	7.21312e+06	836299	1.17598e+06	8.15089e+06	3.07873e+07
57661	20221209.24	0.786548	17305.2	44036	6346.77	8.24484e+07	15416.6	1.53396e+07	46948.5	15471.5	8.78598e+06	14083.5	12406.5	4592.87	3525.53	3309.58	5769.6	6.95098e+06	5818.07	1.50774e+07	25035.2	1560.78	7.99955e+06	1279.87	7.21312e+06	1283.14	7.21312e+06	827958	1.17771e+06	8.17402e+06	3.06069e+07
57709	20221212.3	0.768848	17469.6	45964.8	6316.67	8.27106e+07	15541.1	1.50774e+07	47218.5	15487.8	8.78598e+06	13863.6	12932.9	4409.75	3545.97	3283.75	5876.5	6.68883e+06	5853.62	1.53396e+07	22507.1	1564.18	7.99955e+06	1287.05	7.47526e+06	1293.65	7.21312e+06	830076	1.1791e+06	8.13428e+06	3.07531e+07
57731	20221212.11	0.760781	17519.4	43913.6	6302.7	8.27106e+07	15496.3	1.53396e+07	47662.1	15587.2	8.52384e+06	14091.2	12532.5	4521.8	3539.52	3433.59	5813.3	6.95098e+06	5822.37	1.48153e+07	23930.6	1565.51	7.99955e+06	1284.63	7.47526e+06	1295.33	7.21312e+06	835119	1.17422e+06	8.154e+06	3.07314e+07
57750	20221212.16	0.800014	17395.2	43784.2	6327.65	8.24484e+07	15382.8	1.53396e+07	47121.4	15450.6	8.52384e+06	13945.5	12466.7	4554.56	3688.29	3426.81	5814.6	7.21312e+06	5780.04	1.53396e+07	22168.7	1569.46	7.99955e+06	1293.77	7.47526e+06	1307.33	7.21312e+06	832288	1.18152e+06	8.14078e+06	3.12e+07
57770	20221212.23	0.769973	17468	48044.1	6267.59	8.21863e+07	15420.8	1.50774e+07	44756.4	15479.2	8.78598e+06	13182.6	12366	4504.27	3524.18	3332.77	5831.2	7.21312e+06	5831.11	1.53396e+07	27061.6	1565.58	7.99955e+06	1285.99	7.73741e+06	1289.86	6.95098e+06	808729	1.16155e+06	8.15141e+06	3.14458e+07
57780	20221212.26	0.768107	17259	48188.8	6253.49	8.24484e+07	15422.1	1.50774e+07	47111.4	15468.6	8.78598e+06	13767.6	12397.4	4469.91	3684.63	3445.95	5803.9	7.21312e+06	5809.98	1.50774e+07	23546.6	1566.41	7.99955e+06	1280.77	7.21312e+06	1289.3	7.21312e+06	828365	1.17822e+06	8.13541e+06	3.1529e+07
57788	20221213.1	0.770024	17717.2	44036.9	6291.26	8.21863e+07	15568	1.53396e+07	46316.2	15611.1	8.52384e+06	13186.7	12490.6	4519.51	3586.5	3421.02	5828.5	7.99955e+06	5621	1.4291e+07	24374.4	1568.15	7.99955e+06	1288.65	7.47526e+06	1308.88	7.21312e+06	833925	1.17887e+06	8.1565e+06	3.08489e+07
57807	20221213.7	0.776065	17301.2	45803.8	6318.74	8.13999e+07	15554.2	1.50774e+07	46496.2	15556.4	8.78598e+06	13129.6	12428	4514.25	3556.77	3519.18	5851.3	7.47526e+06	5635.24	1.45532e+07	23406.9	1568.9	7.99955e+06	1288.28	7.73741e+06	1297.5	7.21312e+06	835897	1.18519e+06	8.17284e+06	3.07272e+07
57842	20221213.16	0.766579	17643.6	45917.4	6312.62	8.27106e+07	15471.7	1.53396e+07	44838.3	15585.7	8.78598e+06	14036.9	12602.6	4470.82	3682.72	3513.42	5858	6.95098e+06	5888.6	1.48153e+07	23974.3	1567.55	7.99955e+06	1289.59	7.47526e+06	1309.74	7.21312e+06	830352	1.18086e+06	8.17059e+06	3.0778e+07
57905	20221213.32	0.813671	17173.9	45769.1	6296.64	8.21863e+07	15499.3	1.53396e+07	45710.7	15467	8.78598e+06	14169.6	14092.6	4550.78	3533.51	3461.89	5831.7	6.68883e+06	5818.2	1.50774e+07	21622.8	1572.18	7.73741e+06	1281.57	7.21312e+06	1279.52	6.95098e+06	831250	1.17613e+06	8.17252e+06	3.0791e+07
57931	20221213.41	0.785537	17123.4	45924.4	6267.43	8.27106e+07	15709.5	1.53396e+07	45854.9	15660.4	8.52384e+06	13286.9	14051	4625.12	3692.99	3454.7	5837.6	6.68883e+06	5838.08	1.50774e+07	23163.7	1570.7	7.73741e+06	1294.99	7.73741e+06	1293.53	6.95098e+06	835261	1.18015e+06	8.13641e+06	3.08137e+07
57967	20221213.52	0.761508	17161.7	43775.2	6339.49	8.21863e+07	15643.8	1.50774e+07	44567.7	15597	8.78598e+06	13236.2	12490.4	4645.9	3663.13	3432.87	5856.8	6.68883e+06	5817.83	1.50774e+07	24572.1	1575.7	7.73741e+06	1285.81	7.21312e+06	1300.81	6.95098e+06	834640	1.17501e+06	8.16564e+06	3.07877e+07
58016	20221213.65	0.785305	17163	43759.4	6335.69	8.24484e+07	15486.9	1.56017e+07	47159.6	15619.8	8.52384e+06	13374.2	12541	4664.3	3692.57	3442.9	5852	6.68883e+06	5836.24	1.50774e+07	27891.4	1570.55	7.99955e+06	1285.03	7.21312e+06	1306.57	6.95098e+06	834585	1.17872e+06	8.15462e+06	3.14936e+07
58072	20221214.2	0.793672	17173.7	45293.5	6319.6	8.24484e+07	15476.1	1.50774e+07	47904.6	15592.7	8.78598e+06	14153.7	12612	4453.13	3664.96	3473.84	5838.9	6.68883e+06	5833.02	1.50774e+07	24782.7	1575.52	7.73741e+06	1288.52	7.21312e+06	1301.54	6.95098e+06	828456	1.17998e+06	8.13593e+06	3.01442e+07
58130	20221214.21	0.789091	17231.1	45555	5658.97	8.27106e+07	15505.6	1.50774e+07	45172.5	15506.4	8.52384e+06	13279.6	12509.8	4662.74	3657.87	3473.39	5851.6	6.68883e+06	5830.67	1.50774e+07	23788.3	1571.88	7.73741e+06	1280.75	7.21312e+06	1301.01	6.95098e+06	837723	1.17236e+06	8.169e+06	3.07933e+07

champ_snapshot_fix

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_virtual_cft^	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_virtual_cft^	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	pi_ls_jwt_virtual_cft^	ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_js_jwt_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	hist_sgx_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
58023	20221213.66	0.799932	17134	6364.1	8.21863e+07	45800.4	15552.6	1.53396e+07	47074.7	15669.1	8.52384e+06	13981.8	12598.1	4603.36	3668.49	5883.3	6.68883e+06	3435.96	5827.46	1.50774e+07	1575.55	7.73741e+06	23895.7	1297.95	7.21312e+06	1297.47	6.95098e+06	832346	1.17806e+06	8.15569e+06	3.05982e+07
58064	20221213.76	0.81212	17305.7	6315.87	8.24484e+07	43663.9	15564.1	1.53396e+07	47059.9	15647.5	8.78598e+06	13901.2	12850.5	4547.72	3538.85	5851.7	6.68883e+06	3463.04	5819.24	1.50774e+07	1573.93	7.73741e+06	23184.1	1295.62	7.99955e+06	1301.81	6.95098e+06	830308	1.17574e+06	8.15355e+06	3.07526e+07
58173	20221214.34	0.789536	17329.5	6238.02	8.19242e+07	45757.8	15321.3	1.50774e+07	44563.2	15622.8	8.78598e+06	13950.5	12276.2	4399.52	3557.78	5821.2	6.68883e+06	3430.65	5827.07	1.50774e+07	1561.55	7.99955e+06	28333.9	1287.88	7.73741e+06	1302.4	6.95098e+06	825002	1.17182e+06	8.15543e+06	3.10797e+07

[jumaffre]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[github-actions]

💔 All backports failed

Status	Branch	Result
❌	release/2.x	Backport failed because of merge conflicts You might need to backport the following PRs to release/2.x: - Session consistency: Parse forwarded messages with consistency info (#4595)
❌	release/3.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 4730

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details