Return error when getRangeAndFlatMap has more & Improve simulation tests #6029

nblintao · 2021-11-19T01:02:29Z

Return error when getRangeAndFlatMap has more, as a mitigation before we are able to support continuation.
Improve simulation test to be more robust and verify more things.
Renamed the simulation test from IndexPrefetchDemo to GetRangeAndMap.
Remove throttling the individual lookups underlying a getRangeAndFlatMap request.
Fix error handlings in quickGetKeyValues and quickGetValue.
Turn QUICK_GET_VALUE_FALLBACK and QUICK_GET_KEY_VALUES_FALLBACK off by default.
Propagate contexts to underlying lookups.

Ran 100k for GetRangeAndMap alone.
20211130-173956-taolin-92b2ee2b1c127e25 compressed=True data_size=22400913 duration=1298112 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:25:07 sanity=False started=100117 stopped=20211130-180503 submitted=20211130-173956 timeout=5400 username=taolin

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2021-11-19T01:45:00Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 9873658
Result: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-30T01:46:36Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 1b57cc5
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-30T02:21:40Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 1b57cc5
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-30T18:13:21Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 9482bea
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2021-11-30T18:55:52Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 9482bea
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

halfprice · 2021-12-02T17:32:14Z

bindings/c/test/unit/unit_tests.cpp

@@ -895,64 +895,78 @@ static std::string recordValue(const int i) {
 	return Tuple().append(dataOfRecord(i)).pack().toString();
 }

-TEST_CASE("fdb_transaction_get_range_and_flat_map") {
+std::map<std::string, std::string> fillInRecords(int n) {


Is the return value used anywhere?

No, but we may want to use it in the future.

halfprice · 2021-12-02T17:34:20Z

bindings/c/test/unit/unit_tests.cpp

+	return data;
+}
+
+GetRangeResult getIndexEntriesAndMap(int beginId, int endId, fdb::Transaction& tr) {


Nit: change tr to a pointer to indicate that it is mutable. Mutable reference is hard to read.

tr here is immutable. I'm using reference to keep get_range_and_flat_map consistent with existing functions like GetRangeResult get_range(fdb::Transaction& tr

halfprice · 2021-12-02T17:37:37Z

bindings/c/test/unit/unit_tests.cpp

-			// result[1]: key=fdbRECORDprimary-key-of-record-00000001, value=data-of-record-00000001
-			CHECK(recordKey(i).compare(key) == 0);
-			CHECK(recordValue(i).compare(value) == 0);
+			//			std::cout << "result[" << i << "]: key=" << key << ", value=" << value << std::endl;


Remove this debugging output?

halfprice · 2021-12-02T17:40:02Z

fdbserver/storageserver.actor.cpp

+ACTOR Future<Optional<Value>> quickGetValue(StorageServer* data,
+                                            StringRef key,
+                                            Version version,
+                                            // To provide some contexts.


Can you extend this comment a bit more? Currently, it doesn't provide any information to understand why this is needed.

Good point. Changed to To provide span context, tags, debug ID to underlying lookups.

halfprice · 2021-12-02T17:42:54Z

fdbserver/storageserver.actor.cpp


-		data->actors.add(data->readGuard(req, getKeyValuesQ));


What's the issue with this being guarded?

// Note that it does not use readGuard to avoid server being overloaded here. Throttling is enforced at the // original request level, rather than individual underlying lookups. The reason is that throttle any // individual underlying lookup will fail the original request, which is not productive.

Guard is used to throttle requests. We already use guard at the original request. If we guard every underlying lookups, it is very likely one of them are throttled and then the entire request is failed, which wastes all successful underlying lookups.

halfprice · 2021-12-02T17:45:48Z

fdbserver/storageserver.actor.cpp

+			if (!reply.error.present()) {
+				++data->counters.quickGetValueHit;
+				return reply.value;
+			}


Is the error here interesting or normal? Do you want to count how many of them are happened?

Error here is normal. For example, it will hit error if the requested key is not in the shard. All errors are counted at ++data->counters.quickGetValueMiss; below.

halfprice · 2021-12-02T19:56:05Z

fdbserver/workloads/GetRangeAndMap.actor.cpp

+			std::cout << "result.size()=" << result.size() << std::endl;
+			std::cout << "result.more=" << result.more << std::endl;
+			ASSERT(result.size() == endId - beginId);
+			const KeyValueRef* it = result.begin();


why not put this in the for loop?

I thought FLOW didn't like it. But it seems working now. Fixed.

before we are able to support continuation

- Improve simulation test to be more robust and verify more things. - Renamed the simulation test from `IndexPrefetchDemo` to `GetRangeAndMap`. - Remove throttling the individual lookups underlying a getRangeAndFlatMap request. - Fix error handlings in `quickGetKeyValues` and `quickGetValue`. - Turn `QUICK_GET_VALUE_FALLBACK` and `QUICK_GET_KEY_VALUES_FALLBACK` off by default. - Propagate contexts to underlying lookups.

foundationdb-ci · 2021-12-03T01:55:25Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: bbb0942
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

halfprice

LGTM!

jzhou77 requested review from halfprice and xumengpanda November 29, 2021 21:17

jzhou77 assigned halfprice Nov 29, 2021

nblintao force-pushed the error-have-more branch from 9873658 to 1b57cc5 Compare November 30, 2021 01:14

nblintao changed the title ~~Return error when getRangeAndFlatMap has more~~ Return error when getRangeAndFlatMap has more & Improve simulation tests Nov 30, 2021

nblintao marked this pull request as draft November 30, 2021 01:15

nblintao marked this pull request as ready for review November 30, 2021 17:23

nblintao force-pushed the error-have-more branch from 1b57cc5 to 9482bea Compare November 30, 2021 17:41

halfprice reviewed Dec 2, 2021

View reviewed changes

nblintao added 4 commits December 2, 2021 14:51

Return error when getRangeAndFlatMap has more

0a319e1

before we are able to support continuation

Allow error when fallback is disabled

36e11e3

Respond to comments

bbb0942

nblintao force-pushed the error-have-more branch from 9482bea to bbb0942 Compare December 3, 2021 00:44

nblintao requested a review from halfprice December 3, 2021 19:28

halfprice approved these changes Dec 3, 2021

View reviewed changes

nblintao merged commit 9b0a9c4 into apple:master Dec 3, 2021

nblintao deleted the error-have-more branch December 3, 2021 20:50

This was referenced Feb 10, 2022

Add overwrite operation to mako sfc-gh-ezhang/foundationdb#1

Closed

Add overwrite operation to mako sfc-gh-sgwydir/foundationdb#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return error when getRangeAndFlatMap has more & Improve simulation tests #6029

Return error when getRangeAndFlatMap has more & Improve simulation tests #6029

nblintao commented Nov 19, 2021 •

edited

Loading

foundationdb-ci commented Nov 19, 2021

foundationdb-ci commented Nov 30, 2021

foundationdb-ci commented Nov 30, 2021

foundationdb-ci commented Nov 30, 2021

foundationdb-ci commented Nov 30, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

halfprice Dec 2, 2021

nblintao Dec 2, 2021

foundationdb-ci commented Dec 3, 2021

halfprice left a comment

Return error when getRangeAndFlatMap has more & Improve simulation tests #6029

Return error when getRangeAndFlatMap has more & Improve simulation tests #6029

Conversation

nblintao commented Nov 19, 2021 • edited Loading

Code-Reviewer Section

For Release-Branches

foundationdb-ci commented Nov 19, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Nov 30, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Nov 30, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Nov 30, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Nov 30, 2021

AWS CodeBuild CI Report for Linux CentOS 7

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foundationdb-ci commented Dec 3, 2021

AWS CodeBuild CI Report for Linux CentOS 7

halfprice left a comment

Choose a reason for hiding this comment

nblintao commented Nov 19, 2021 •

edited

Loading