Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Keep track of time of backup latest ACK on primary #3769

Merged
merged 18 commits into from
Apr 22, 2022
Merged

Keep track of time of backup latest ACK on primary #3769

merged 18 commits into from
Apr 22, 2022

Conversation

jumaffre
Copy link
Contributor

@jumaffre jumaffre commented Apr 14, 2022 •
edited
Loading

Resolves #3688

The primary node keeps track of the time since last ACK from each backup and report it in the GET /node/consensus endpoint. This is useful to detect one-way partitions between the primary and backups, and a useful building block towards CheckQuorum.

This is reflected in the health_watcher.py sample script and tested by the existing partitions_test.py end-to-end test.

TODO:

  • Changelog
  • Implement in health_watcher.py sample
  • Add tests in partitions_test.py

@ccf-bot
Copy link
Collaborator

ccf-bot commented Apr 14, 2022 •
edited
Loading

backup_last_ack_time_primary@44638 aka 20220422.12 vs main ewma over 20 builds from 44329 to 44633

Click to see table

main

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
44329 20220413.1 6318.87 8.37592e+07 19571.8 1.63882e+07 5326.63 1.56017e+07 2474.35 1.2456e+07 1622.18 1.62665e+08 2073.79 9.31027e+06 1450.62 9.84392e+07 2009.1 1.14074e+07 17480.9 884850 1.37255e+06 9.27948e+06 3.59298e+07
44338 20220413.5 5984.76 8.42835e+07 18616.5 1.6126e+07 5415.11 1.53396e+07 2482.9 9.83456e+06 1589.58 1.6083e+08 1959.44 9.04813e+06 1395.21 9.76528e+07 1909.61 1.11453e+07 20134.4 907917 1.36578e+06 9.20859e+06 3.58669e+07
44348 20220413.9 6115.29 8.37592e+07 19130.5 1.63882e+07 5367.76 1.56017e+07 2460.55 9.83456e+06 1554.79 1.62927e+08 2068.35 9.04813e+06 1403.35 9.84392e+07 1915.44 8.78598e+06 17429.5 893034 1.38528e+06 9.25855e+06 3.58669e+07
44368 20220413.17 6113.31 8.3497e+07 18888.1 1.56017e+07 5374.56 1.53396e+07 2453.32 9.83456e+06 1566.39 1.6083e+08 1931.04 1.21939e+07 1410.63 9.87014e+07 1905.67 8.78598e+06 19002.7 888069 1.37144e+06 9.27944e+06 3.5128e+07
44380 20220413.22 6205.84 8.42835e+07 19126.4 1.66503e+07 5525.93 1.56017e+07 2466.19 9.83456e+06 1612.49 1.6083e+08 2067.28 9.04813e+06 1392.29 9.81771e+07 1872.3 1.14074e+07 18780.3 895454 1.382e+06 9.25436e+06 3.51286e+07
44390 20220414.2 6097.35 8.45456e+07 19068.6 1.63882e+07 5518.22 1.53396e+07 2530.35 1.00967e+07 1612.86 1.62927e+08 2138.24 9.04813e+06 1428.52 9.81771e+07 1915.92 8.78598e+06 21057 889853 1.36524e+06 9.20449e+06 3.58042e+07
44438 20220414.20 5443.52 8.32349e+07 16250 1.58639e+07 4662.56 1.50774e+07 2074.9 8.78598e+06 1380.57 1.62402e+08 1685.38 8.52384e+06 1196.64 9.71285e+07 1588.47 1.00967e+07 18369.1 739852 1.19983e+06 7.7195e+06 3.13625e+07
44442 20220415.2 5591.36 8.40213e+07 16034.4 1.6126e+07 4817.04 1.53396e+07 2133.26 8.78598e+06 1365.81 1.6214e+08 1733.19 9.04813e+06 1207.12 9.76528e+07 1614.55 7.99955e+06 17234 767352 1.17021e+06 7.66461e+06 3.13145e+07
44450 20220418.1 6236.8 8.3497e+07 19495.3 1.63882e+07 5617.15 1.56017e+07 2518.31 1.00967e+07 1650.22 1.61092e+08 2107.02 9.04813e+06 1456.29 9.84392e+07 1966.16 8.78598e+06 19534.6 898004 1.37698e+06 9.20441e+06 3.5189e+07
44456 20220419.1 6385.12 8.42835e+07 19687.5 1.66503e+07 5565.56 1.58639e+07 2530.55 1.03588e+07 1677.46 1.63189e+08 2112.94 9.04813e+06 1457.06 9.84392e+07 1969.5 8.78598e+06 20420 901046 1.38359e+06 9.21688e+06 3.63121e+07
44478 20220419.11 6255.71 8.29727e+07 19728.3 1.56017e+07 5571.48 1.56017e+07 2537.57 1.00967e+07 1634.82 1.61092e+08 2119.03 9.04813e+06 1453.09 9.84392e+07 1975.38 8.78598e+06 17514.4 892408 1.39909e+06 9.3901e+06 3.58042e+07
44495 20220419.17 6234.68 8.37592e+07 19338.6 1.66503e+07 5577.5 1.58639e+07 2520.06 1.0621e+07 1605.26 1.62927e+08 2111.81 9.31027e+06 1429.12 9.81771e+07 1964.87 8.78598e+06 19672.2 886916 1.37301e+06 9.20441e+06 3.58663e+07
44522 20220420.1 5471.13 8.48077e+07 19521.8 1.6126e+07 5907.1 1.50774e+07 2548.13 1.00967e+07 1636.76 1.63189e+08 2110.2 9.04813e+06 1460.94 9.81771e+07 1969.05 8.78598e+06 17267.2 909448 1.37624e+06 9.26274e+06 3.58042e+07
44537 20220420.7 6263.2 8.48077e+07 19447 1.58639e+07 5596.09 1.53396e+07 2518.95 1.00967e+07 1622.95 1.61092e+08 2112.56 9.31027e+06 1435.69 9.84392e+07 1923.77 9.04813e+06 18504.3 902746 1.37985e+06 9.24592e+06 3.5128e+07
44549 20220420.12 6287.23 8.45456e+07 19403.7 1.66503e+07 5549.62 1.53396e+07 2522.07 1.00967e+07 1623.35 1.6083e+08 2113.35 9.04813e+06 1429.35 9.63421e+07 1924.57 8.52384e+06 19222.7 896584 1.39252e+06 9.3472e+06 3.64406e+07
44565 20220421.2 6153.74 8.29727e+07 19394.2 1.63882e+07 5502.4 1.56017e+07 2542.42 1.00967e+07 1646.62 1.63451e+08 2111.36 9.31027e+06 1426 9.81771e+07 1919.71 1.08831e+07 21446.3 891147 1.37899e+06 9.27532e+06 3.57405e+07
44588 20220421.9 6254.36 8.42835e+07 19270.9 1.66503e+07 5259.9 1.53396e+07 2416.19 1.2456e+07 1580.81 1.62927e+08 2063.71 1.16696e+07 1444.25 9.81771e+07 1967.81 8.78598e+06 18563.6 885808 1.36788e+06 9.25436e+06 3.24559e+07
44604 20220422.2 6360.83 8.37592e+07 19996.8 1.6126e+07 5607.58 1.53396e+07 2591.48 9.83456e+06 1645.05 1.61092e+08 2115.67 1.29803e+07 1450.48 9.81771e+07 1970.71 8.78598e+06 19598.8 888228 1.35925e+06 9.20441e+06 3.58042e+07
44624 20220422.8 6255.19 8.40213e+07 19269.7 1.69124e+07 5956.06 1.50774e+07 2545.64 9.83456e+06 1623.32 1.63189e+08 2044.18 9.04813e+06 1425.08 9.81771e+07 1927.69 8.52384e+06 19441.9 923015 1.34665e+06 9.3858e+06 3.58669e+07
44633 20220422.11 6145.55 8.40213e+07 19313.1 1.66503e+07 5571.89 1.58639e+07 2524.69 1.00967e+07 1565.49 1.63451e+08 2109.3 9.04813e+06 1427.41 9.84392e+07 2016.24 8.78598e+06 20462.1 905067 1.37884e+06 9.27532e+06 3.58042e+07

backup_last_ack_time_primary

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
44531 20220420.4 6388.56 8.40213e+07 19356.1 1.63882e+07 5613.37 1.53396e+07 2545.05 1.03588e+07 1643.5 1.63189e+08 2116.82 9.04813e+06 1435.11 9.87014e+07 1976.71 8.78598e+06 19754.6 904987 1.3732e+06 9.09406e+06 3.51884e+07
44545 20220420.10 6288.67 8.29727e+07 19215.7 1.6126e+07 5581.5 1.56017e+07 2520.98 1.00967e+07 1613.28 1.63451e+08 2108.76 9.04813e+06 1440.74 9.81771e+07 1967.29 8.78598e+06 17618.3 906784 1.33601e+06 9.05782e+06 3.56168e+07
44556 20220420.14 6307.6 8.45456e+07 19161.9 1.6126e+07 5372.5 1.56017e+07 2472.84 1.00967e+07 1616.7 1.62927e+08 2080.02 9.04813e+06 1430.61 9.68664e+07 1940.97 1.11453e+07 17898.2 866972 1.36558e+06 9.33438e+06 3.59292e+07
44593 20220421.10 6257.32 8.45456e+07 18626.4 1.66503e+07 5400.37 1.50774e+07 2551.04 1.00967e+07 1579.63 1.62402e+08 2122.1 1.19317e+07 1456.36 9.84392e+07 1935.14 1.0621e+07 17504.9 871523 1.37634e+06 9.30905e+06 3.58042e+07
44638 20220422.12 6339.97 8.32349e+07 19527.9 1.69124e+07 5584.84 1.53396e+07 2534.19 1.00967e+07 1664.22 1.63451e+08 2124.22 1.16696e+07 1444.6 9.84392e+07 1880.82 8.52384e+06 21557.8 903112 1.39442e+06 9.20032e+06 3.45946e+07

images

@jumaffre jumaffre marked this pull request as ready for review April 19, 2022 13:34
@jumaffre jumaffre requested a review from a team as a code owner April 19, 2022 13:34
@jumaffre jumaffre changed the title Keep track of backup last ACK on primary Keep track of time of backup latest ACK on primary Apr 19, 2022
@jumaffre
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@jumaffre
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

achamayou
achamayou reviewed Apr 20, 2022
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
@jumaffre jumaffre merged commit 314c408 into microsoft:main Apr 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@achamayou achamayou achamayou approved these changes

@raghavthind2005 raghavthind2005 raghavthind2005 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Report time of last backup ACK in GET /node/consensus endpoint
4 participants
@jumaffre @ccf-bot @achamayou @raghavthind2005