KeyValueStoreRocksDB histograms to track latencies #6149

neethuhaneesha · 2021-12-15T00:14:20Z

KeyValueStoreRocksDB histograms to track latencies

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2021-12-15T01:14:32Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: b940f21
Result: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T01:30:14Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: eb1d115
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T02:35:29Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: deabed2
Result: FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T02:56:47Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: deabed2
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T18:46:06Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: eec774b
Result: FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T18:49:20Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: e85896e
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T19:27:41Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: e85896e
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T20:22:53Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: c30bd4e
Result: FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T20:44:35Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: c30bd4e
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-15T22:18:15Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: c30bd4e
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-17T23:31:06Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 6328caa
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-18T00:02:49Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 6328caa
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

xumengpanda · 2021-12-18T00:18:14Z

fdbserver/KeyValueStoreRocksDB.actor.cpp

 			rocksdb::PinnableSlice value;
 			auto options = getReadOptions();
-			uint64_t deadlineMircos =
-			    db->GetEnv()->NowMicros() + (readValueTimeout - (timer_monotonic() - a.startTime)) * 1000000;
+			uint64_t deadlineMircos = db->GetEnv()->NowMicros() + (readValueTimeout - (now() - a.startTime)) * 1000000;


why changing to now()?

Finally I have decided to use timer_monotonic for measuring the histograms.
Simulation: timer_monotonic is calling timer() which is 0.1 seconds ahead of now(). Might not show exact latencies, but it should be fine since simulation test are not used for perf measurements.
Non-simulation: timer_monotonic() returns a high precision monotonic clock. Will show exact latencies and expensive, so currently sampling with a knob at the rate of 1 out of 1000 read ops.
Sampling is not perfect but I could not find a better way.

fdbserver/KeyValueStoreRocksDB.actor.cpp

liquid-helium · 2021-12-21T05:11:31Z

fdbserver/KeyValueStoreRocksDB.actor.cpp

 			rocksdb::PinnableSlice value;
 			auto options = getReadOptions();
-			uint64_t deadlineMircos =
-			    db->GetEnv()->NowMicros() + (readValueTimeout - (timer_monotonic() - a.startTime)) * 1000000;
+			uint64_t deadlineMircos = db->GetEnv()->NowMicros() + (readValueTimeout - (now() - a.startTime)) * 1000000;


liquid-helium · 2021-12-21T05:16:22Z

fdbserver/KeyValueStoreRocksDB.actor.cpp

 			auto s = db->Get(options, db->DefaultColumnFamily(), toSlice(a.key), &value);
+			readValueGetHistogram->sampleSeconds(now() - dbGetBeginTime);


Do we need to move this to the below if(s.ok()) scope, to calculate only the successful reads?

foundationdb-ci · 2021-12-21T21:44:23Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 995bf25
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-21T22:42:19Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 995bf25
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-21T23:21:28Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: dc8ba37
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-22T00:21:07Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: dc8ba37
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

fdbclient/ServerKnobs.h

xumengpanda · 2021-12-22T01:55:48Z

fdbclient/ServerKnobs.cpp

@@ -354,6 +354,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
 	init( ROCKSDB_READ_QUEUE_SOFT_MAX,                           500 );
 	init( ROCKSDB_FETCH_QUEUE_HARD_MAX,                          100 );
 	init( ROCKSDB_FETCH_QUEUE_SOFT_MAX,                           50 );
+	init( ROCKSDB_HISTOGRAMS_SAMPLE_RATE,                      0.001 );


BUGGIFY the knob to -1 to cover the code path when this histogram is disabled.

xumengpanda · 2021-12-22T01:56:29Z

What's the sim test result after resolving the comments?

foundationdb-ci · 2021-12-22T07:23:49Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 1f30368
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-22T08:22:58Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 1f30368
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

neethuhaneesha · 2021-12-22T08:27:28Z

What's the sim test result after resolving the comments?

20211222-071327-neethuhaneeshabingi-e8449ab7ef9f700e ended=100000 fail=4

KeyValueStoreRocksDB histograms to track latencies

neethuhaneesha marked this pull request as draft December 15, 2021 00:14

neethuhaneesha force-pushed the rocksdbHistograms branch from b940f21 to eb1d115 Compare December 15, 2021 00:18

neethuhaneesha force-pushed the rocksdbHistograms branch from eb1d115 to deabed2 Compare December 15, 2021 02:23

neethuhaneesha force-pushed the rocksdbHistograms branch 2 times, most recently from e85896e to eec774b Compare December 15, 2021 18:34

neethuhaneesha force-pushed the rocksdbHistograms branch from eec774b to c30bd4e Compare December 15, 2021 20:11

neethuhaneesha force-pushed the rocksdbHistograms branch from c30bd4e to 6328caa Compare December 17, 2021 22:57

neethuhaneesha requested review from xumengpanda and liquid-helium December 17, 2021 23:59

neethuhaneesha marked this pull request as ready for review December 18, 2021 00:02

xumengpanda reviewed Dec 18, 2021

View reviewed changes

neethuhaneesha marked this pull request as draft December 20, 2021 19:14

liquid-helium reviewed Dec 21, 2021

View reviewed changes

neethuhaneesha force-pushed the rocksdbHistograms branch from 6328caa to 995bf25 Compare December 21, 2021 21:37

neethuhaneesha marked this pull request as ready for review December 21, 2021 21:37

neethuhaneesha marked this pull request as draft December 21, 2021 22:48

neethuhaneesha force-pushed the rocksdbHistograms branch from 995bf25 to dc8ba37 Compare December 21, 2021 23:14

neethuhaneesha marked this pull request as ready for review December 21, 2021 23:15

neethuhaneesha requested review from xumengpanda and liquid-helium December 21, 2021 23:31

xumengpanda reviewed Dec 22, 2021

View reviewed changes

KeyValueStoreRocksDB histograms to track latencies

1f30368

neethuhaneesha force-pushed the rocksdbHistograms branch from dc8ba37 to 1f30368 Compare December 22, 2021 07:17

neethuhaneesha requested a review from xumengpanda December 22, 2021 08:28

xumengpanda approved these changes Dec 22, 2021

View reviewed changes

liquid-helium approved these changes Dec 22, 2021

View reviewed changes

neethuhaneesha merged commit 3086941 into apple:master Dec 22, 2021

neethuhaneesha deleted the rocksdbHistograms branch December 22, 2021 19:33

neethuhaneesha added a commit to neethuhaneesha/foundationdb that referenced this pull request Dec 22, 2021

cherry-pick apple#6149

0a4a8a0

KeyValueStoreRocksDB histograms to track latencies

neethuhaneesha mentioned this pull request Dec 22, 2021

cherry-pick #6149 to 7.0. KeyValueStoreRocksDB histograms to track latencies #6183

Merged

5 tasks

xumengpanda pushed a commit that referenced this pull request Dec 22, 2021

cherry-pick #6149

0bbf4d5

KeyValueStoreRocksDB histograms to track latencies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KeyValueStoreRocksDB histograms to track latencies #6149

KeyValueStoreRocksDB histograms to track latencies #6149

neethuhaneesha commented Dec 15, 2021 •

edited

Loading

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 15, 2021

foundationdb-ci commented Dec 17, 2021

foundationdb-ci commented Dec 18, 2021

xumengpanda Dec 18, 2021

liquid-helium Dec 21, 2021

neethuhaneesha Dec 21, 2021 •

edited

Loading

liquid-helium Dec 21, 2021

liquid-helium Dec 21, 2021

foundationdb-ci commented Dec 21, 2021

foundationdb-ci commented Dec 21, 2021

foundationdb-ci commented Dec 21, 2021

foundationdb-ci commented Dec 22, 2021

xumengpanda Dec 22, 2021

neethuhaneesha Dec 22, 2021

xumengpanda commented Dec 22, 2021

foundationdb-ci commented Dec 22, 2021

foundationdb-ci commented Dec 22, 2021

neethuhaneesha commented Dec 22, 2021

		auto s = db->Get(options, db->DefaultColumnFamily(), toSlice(a.key), &value);
		readValueGetHistogram->sampleSeconds(now() - dbGetBeginTime);

KeyValueStoreRocksDB histograms to track latencies #6149

KeyValueStoreRocksDB histograms to track latencies #6149

Conversation

neethuhaneesha commented Dec 15, 2021 • edited Loading

Code-Reviewer Section

For Release-Branches

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 15, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 17, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 18, 2021

AWS CodeBuild CI Report for Linux CentOS 7

xumengpanda Dec 18, 2021

Choose a reason for hiding this comment

liquid-helium Dec 21, 2021

Choose a reason for hiding this comment

neethuhaneesha Dec 21, 2021 • edited Loading

Choose a reason for hiding this comment

liquid-helium Dec 21, 2021

Choose a reason for hiding this comment

liquid-helium Dec 21, 2021

Choose a reason for hiding this comment

foundationdb-ci commented Dec 21, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 21, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Dec 21, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 22, 2021

AWS CodeBuild CI Report for Linux CentOS 7

xumengpanda Dec 22, 2021

Choose a reason for hiding this comment

neethuhaneesha Dec 22, 2021

Choose a reason for hiding this comment

xumengpanda commented Dec 22, 2021

foundationdb-ci commented Dec 22, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 22, 2021

AWS CodeBuild CI Report for Linux CentOS 7

neethuhaneesha commented Dec 22, 2021

neethuhaneesha commented Dec 15, 2021 •

edited

Loading

neethuhaneesha Dec 21, 2021 •

edited

Loading