GetMappedRange support serializable & check RYW & continuation #6181

nblintao · 2021-12-22T01:23:52Z

This PR tries to fix 3 major caveats of the original "getRangeAndMap" implementation:

Support serializable
Support read-your-write (The transaction will fail if it actually reads what it wrote. This is not perfect but at least the semantic will be correct for the transactions that succeed.)
Support limits and continuation

To support this, the most important change is that the responses (at all layers) are no longer a flatten list of key-value pairs that contains the result of all underlying(or saying, secondary) queries,

but a list of objects, where each object contains

the key-value pair in the original range query
the begin and end key selector of the corresponding underlying query.
the list of key-value pairs that contains the the result of the underlying query.

In original design, the underlying query can be either a getRange or getValue, but this PR focuses on the case where they are getRange.

Also the naming is changed from "…AndMap" or "…AndFlatMap" to "Mapped…", in order to make is consistent and precise.

There are simulation tests, C binding unit tests, Java binding integration tests. All tests are updated to reflect the new features and new APIs.

20220203-235726-taolin-cb298d9cef410749 compressed=True data_size=21195353 duration=970376 ended=100000 fail_fast=1 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:24:22 sanity=False started=100053 stopped=20220204-002148 submitted=20220203-235726 timeout=5400 username=taolin

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2021-12-22T01:30:15Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: d493067
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2021-12-22T01:31:01Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: d493067
Result: FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-11T22:50:50Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: cc4ec67
Result: FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-11T22:53:24Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: cc4ec67
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-19T04:34:55Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 526132e
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-19T04:40:35Z

AWS CodeBuild CI Report for macOS Catalina 10.15

CodeBuild project: foundationdb-pr-macos
Commit ID: 526132e
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-20T06:39:38Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: ae3cbe0
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-01-26T00:03:17Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 8d99e1a
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

fdb-windows-ci · 2022-01-26T00:39:18Z

Doxense CI Report for Windows 10

Commit ID: 8d99e1a
Result: SUCCEEDED
Build Logs (available dor 30 days)

foundationdb-ci · 2022-02-04T00:38:31Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: acbe336
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

nblintao · 2022-02-04T00:39:05Z

I separated the part that I have finished as a separate commit 14787a658341d93eece4bdeba3f664d8477bcee4. It includes all the changes needed in NativeAPI layer and below (SS, for example). It also updates the GetRangeAndMap simulation test cover the changes.

20220203-235726-taolin-cb298d9cef410749 compressed=True data_size=21195353 duration=970376 ended=100000 fail_fast=1 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:24:22 sanity=False started=100053 stopped=20220204-002148 submitted=20220203-235726 timeout=5400 username=taolin

To reviewers: You can start reviewing that commit, so that there won't be surprises when the entire PR is ready. Please feel free let me know if you need more explanation or a walk-through of the code.

fdb-windows-ci · 2022-02-04T01:30:08Z

Doxense CI Report for Windows 10

Commit ID: acbe336
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-02-08T01:51:25Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 66f5f40
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

fdb-windows-ci · 2022-02-08T02:03:25Z

Doxense CI Report for Windows 10

Commit ID: 66f5f40
Result: ❌ FAILED
Build Logs (available for 30 days)

foundationdb-ci · 2022-02-08T02:17:39Z

AWS CodeBuild CI Report for macOS BigSur 11.5.2

CodeBuild project: foundationdb-pr-macos
Commit ID: 66f5f40
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2022-02-10T05:08:27Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 8d15e58
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

fdb-windows-ci · 2022-02-10T05:24:02Z

Doxense CI Report for Windows 10

Commit ID: 8d15e58
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-02-10T05:36:08Z

AWS CodeBuild CI Report for macOS BigSur 11.5.2

CodeBuild project: foundationdb-pr-macos
Commit ID: 8d15e58
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2022-02-16T02:05:05Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 2443058
Result: FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

jzhou77 · 2022-03-07T18:49:18Z

fdbclient/FDBTypes.h

+
+	MappedReqAndResultRef reqAndResult;
+
+	MappedKeyValueRef() {}


Nit: use default constructor

MappedKeyValueRef() = default;

jzhou77 · 2022-03-07T19:01:25Z

bindings/java/src/main/com/apple/foundationdb/MappedRangeResultInfo.java

+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2013-2018 Apple Inc. and the FoundationDB project authors


s/2018/2022/

bindings/java/src/main/com/apple/foundationdb/MappedRangeResultDirectBufferIterator.java

bindings/java/src/main/com/apple/foundationdb/MappedRangeResult.java

bindings/c/test/unit/unit_tests.cpp

fdbserver/storageserver.actor.cpp

fdbserver/workloads/GetMappedRange.actor.cpp

jzhou77 · 2022-03-07T23:09:46Z

fdbserver/workloads/GetMappedRange.actor.cpp

+#include "flow/Error.h"
+#include "flow/IRandom.h"
+#include "flow/flow.h"
+#include "ApiWorkload.h"


Suggested change

#include "ApiWorkload.h"

#include "fdbserver/workloads/ApiWorkload.h"

jzhou77 · 2022-03-07T23:14:10Z

fdbserver/workloads/GetMappedRange.actor.cpp

+			// TODO: When n is large, split into multiple transactions.
+			state Transaction tr(cx);
+			try {
+				tr.reset();


Delete this line, and move line 100 out of the loop. Basically tr.onError does the right thing for resetting the internal states of tr and updating the timeout value.

If calling reset here, the timeout value is also reset, which is undesirable.

jzhou77 · 2022-03-07T23:16:09Z

fdbserver/workloads/GetMappedRange.actor.cpp

+		// TODO: When n is large, split into multiple transactions.
+		state Transaction tr(cx);
+		try {
+			tr.reset();


Delete. And add a loop outside try...catch block.

jzhou77 · 2022-03-08T00:47:27Z

fdbclient/ReadYourWrites.actor.cpp

+	    static void addConflictRangeAndMustUnmodified(ReadYourWritesTransaction* ryw,
+	                                                  GetMappedRangeReq<backwards> read,
+	                                                  WriteMap::iterator* it,
+	                                                  MappedRangeResult* result) {


Delete this comment block.

foundationdb-ci · 2022-03-09T01:31:02Z

AWS CodeBuild CI Report for macOS BigSur 11.5.2

CodeBuild project: foundationdb-pr-macos
Commit ID: 5c24094
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} if [ -d "foundationdb" ] && pgrep -x clang > /dev/null; then echo "There is a running macOS CI, please try again later."; exit 1; fi. Reason: exit status 2
Build Logs (available for 30 days)

fdb-windows-ci · 2022-03-09T01:56:31Z

Doxense CI Report for Windows 10

Commit ID: 5c24094
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-09T02:15:06Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 5c24094
Result: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-09T19:16:26Z

AWS CodeBuild CI Report for macOS BigSur 11.5.2

CodeBuild project: foundationdb-pr-macos
Commit ID: d884f3a
Result: FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} if [ -d "foundationdb" ] && pgrep -x clang > /dev/null; then echo "There is a running macOS CI, please try again later."; exit 1; fi. Reason: exit status 2
Build Logs (available for 30 days)

fdb-windows-ci · 2022-03-09T19:28:47Z

Doxense CI Report for Windows 10

Commit ID: d884f3a
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-09T20:00:40Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: d884f3a
Result: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

jzhou77

LGTM

jzhou77 · 2022-03-09T22:44:15Z

fdbserver/workloads/GetMappedRange.actor.cpp

-			wait(tr.onError(e));
+		loop {
+			try {
+				tr.reset();


Delete this line.

jzhou77

LGTM.

A suggestion is to move "mapper" logic, e.g., constructMappedKey, into its own files, which can be done in a separate PR.

fdb-windows-ci · 2022-03-09T23:15:31Z

Doxense CI Report for Windows 10

Commit ID: 4e40bf3
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-09T23:35:53Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: 4e40bf3
Result: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)

fdb-windows-ci · 2022-03-10T00:16:26Z

Doxense CI Report for Windows 10

Commit ID: c866c21
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-10T00:19:23Z

AWS CodeBuild CI Report for macOS BigSur 11.5.2

CodeBuild project: foundationdb-pr-macos
Commit ID: c866c21
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2022-03-10T00:58:03Z

AWS CodeBuild CI Report for Linux CentOS 7

CodeBuild project: foundationdb-pr
Commit ID: c866c21
Result: SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

… continuation

nblintao force-pushed the serializable-get-range-and-map branch from d493067 to cc4ec67 Compare January 11, 2022 22:46

nblintao force-pushed the serializable-get-range-and-map branch from cc4ec67 to 526132e Compare January 19, 2022 04:29

nblintao force-pushed the serializable-get-range-and-map branch from 526132e to ae3cbe0 Compare January 20, 2022 06:33

nblintao force-pushed the serializable-get-range-and-map branch from ae3cbe0 to 8d99e1a Compare January 25, 2022 23:59

nblintao changed the title ~~[WIP] GetRangeAndMap support serializable & check RYW~~ [WIP] GetRangeAndMap support serializable & check RYW & continuation Jan 25, 2022

nblintao force-pushed the serializable-get-range-and-map branch from 8d99e1a to acbe336 Compare February 4, 2022 00:34

nblintao requested review from halfprice and xumengpanda February 4, 2022 00:40

nblintao force-pushed the serializable-get-range-and-map branch from acbe336 to 66f5f40 Compare February 8, 2022 01:45

nblintao force-pushed the serializable-get-range-and-map branch from 66f5f40 to 8d15e58 Compare February 10, 2022 05:03

nblintao force-pushed the serializable-get-range-and-map branch from 8d15e58 to 2443058 Compare February 16, 2022 02:01

jzhou77 requested changes Mar 7, 2022

View reviewed changes

jzhou77 reviewed Mar 8, 2022

View reviewed changes

nblintao force-pushed the serializable-get-range-and-map branch from c1a1515 to 5c24094 Compare March 9, 2022 01:26

jzhou77 requested changes Mar 9, 2022

View reviewed changes

fdbserver/workloads/GetMappedRange.actor.cpp Outdated

wait(tr.onError(e));

loop {

try {

tr.reset();

Copy link

Contributor

jzhou77 Mar 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete this line.

nblintao added 4 commits March 9, 2022 14:46

GetMappedRange support serializable & check RYW & continuation

68c0559

Response to Jingyu's comments

f0b992b

Change the byte limit of the underlying query to max (10MB)

b0d30b9

Delete redundant reset

4e40bf3

nblintao force-pushed the serializable-get-range-and-map branch from d884f3a to 4e40bf3 Compare March 9, 2022 22:46

jzhou77 previously approved these changes Mar 9, 2022

View reviewed changes

Fix trace event detail name

c866c21

nblintao dismissed jzhou77’s stale review via c866c21 March 9, 2022 23:48

nblintao requested a review from jzhou77 March 10, 2022 01:03

jzhou77 approved these changes Mar 10, 2022

View reviewed changes

nblintao removed the request for review from halfprice March 10, 2022 18:05

nblintao merged commit e2c7c30 into apple:main Mar 10, 2022

nblintao deleted the serializable-get-range-and-map branch March 10, 2022 18:05

nblintao added a commit to nblintao/foundationdb that referenced this pull request Mar 11, 2022

(apple#6181 to 7.1) GetMappedRange support serializable & check RYW &…

7b4ba07

… continuation

crclark mentioned this pull request Jul 31, 2022

Add FoundationDB 7.1 Support crclark/foundationdb-haskell#52

Closed

	#include "ApiWorkload.h"
	#include "fdbserver/workloads/ApiWorkload.h"

GetMappedRange support serializable & check RYW & continuation #6181

GetMappedRange support serializable & check RYW & continuation #6181

Conversation

nblintao commented Dec 22, 2021 • edited Loading

Code-Reviewer Section

For Release-Branches

foundationdb-ci commented Dec 22, 2021

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Dec 22, 2021

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Jan 11, 2022

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Jan 11, 2022

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Jan 19, 2022

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Jan 19, 2022

AWS CodeBuild CI Report for macOS Catalina 10.15

foundationdb-ci commented Jan 20, 2022

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Jan 26, 2022

AWS CodeBuild CI Report for Linux CentOS 7

fdb-windows-ci commented Jan 26, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Feb 4, 2022

AWS CodeBuild CI Report for Linux CentOS 7

nblintao commented Feb 4, 2022 • edited Loading

fdb-windows-ci commented Feb 4, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Feb 8, 2022

AWS CodeBuild CI Report for Linux CentOS 7

fdb-windows-ci commented Feb 8, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Feb 8, 2022

AWS CodeBuild CI Report for macOS BigSur 11.5.2

foundationdb-ci commented Feb 10, 2022

AWS CodeBuild CI Report for Linux CentOS 7

fdb-windows-ci commented Feb 10, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Feb 10, 2022

AWS CodeBuild CI Report for macOS BigSur 11.5.2

foundationdb-ci commented Feb 16, 2022

AWS CodeBuild CI Report for Linux CentOS 7

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foundationdb-ci commented Mar 9, 2022

AWS CodeBuild CI Report for macOS BigSur 11.5.2

fdb-windows-ci commented Mar 9, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Mar 9, 2022

AWS CodeBuild CI Report for Linux CentOS 7

foundationdb-ci commented Mar 9, 2022

AWS CodeBuild CI Report for macOS BigSur 11.5.2

fdb-windows-ci commented Mar 9, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Mar 9, 2022

AWS CodeBuild CI Report for Linux CentOS 7

jzhou77 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzhou77 left a comment

Choose a reason for hiding this comment

fdb-windows-ci commented Mar 9, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Mar 9, 2022

AWS CodeBuild CI Report for Linux CentOS 7

fdb-windows-ci commented Mar 10, 2022

Doxense CI Report for Windows 10

foundationdb-ci commented Mar 10, 2022

AWS CodeBuild CI Report for macOS BigSur 11.5.2

nblintao commented Dec 22, 2021 •

edited

Loading

nblintao commented Feb 4, 2022 •

edited

Loading