RUST-656 Run tests against serverless in evergreen #504

abr-egn · 2021-10-28T18:14:01Z

This adds evergreen configuration to run a subset of tests on serverless, and fixes various test runner bugs that were preventing those tests from passing.

This also closes RUST-983.

abr-egn · 2021-10-28T19:23:46Z

.evergreen/run-serverless-tests.sh

+        cargo test ${DEFAULT_FEATURES} --features $FEATURE_FLAGS $1 $OPTIONS
+}
+
+cargo_test test::spec::crud


This is the subset of tests listed by the spec; running the full suite had a lot of noisy errors (unsupported operations, failures due to restrictions, db cleanup issues) that didn't seem worth individually chasing down.

abr-egn · 2021-10-28T19:24:55Z

src/coll/mod.rs

-                                    .get_or_insert_with(Default::default)
-                                    .push(BulkWriteError { index, ..err });
+                Err(e) => {
+                    let labels = e.labels().clone();


Github diff is misleading here - the only change is cloning the labels before the match to accommodate the labels field no longer being visible.

abr-egn · 2021-11-02T14:16:17Z

src/client/options/mod.rs

@@ -1190,6 +1190,9 @@ impl ClientOptions {
    /// Applies the options in other to these options if a value is not already present
    #[cfg(test)]
    pub(crate) fn merge(&mut self, other: ClientOptions) {
+        if self.hosts.is_empty() {


I'm not super thrilled about the behavior here but I can't think of a clean solution.

TestClient and EventClient have a with_additional_options constructor fn that, semantically, takes a passed-in ClientOptions and merges in the set of "test-default" options there for any unset fields, with preference given to the passed-in fields where both were set.

Prior to this change, hosts was not included in the merge, so the passed-in ClientOptions would always determine hosts. This caused the label_not_added_first_read_error / label_not_added_second_read_error tests to break on serverless because those tests were not setting hosts and the default value was not valid in those cases.

Ideally, the semantics for hosts would be the same as for the rest of the fields - if unset, it would use the "test default". However, because it's populated in the builder by default, there's no way to actually determine "if unset"; testing against the default value causes other tests to fail because those are specifically constructing a hosts vec that, coincidentally, is equal to the default value, and should not be overridden by the test default.

Again ideally, this could be fixed by changing hosts to an Option<Vec<_>> or not populating it by default so vec[] would be a reasonable signal for "unset", but either of those would be a breaking change.

Calling with_additional_options with a ClientOptions constructed via builder with the default hosts is a nasty pitfall that's easy to hit - from the call site it looks like you'll be constructing a client with only the options specified, and the default hosts value will coincidentally work in most but not all testing environments.

Existing tests with the exception of the two that broke avoid this by either explicitly populating hosts or copying the ClientOptions from the CLIENT_OPTIONS value.

.evergreen/config.yml

.evergreen/run-serverless-tests.sh

patrickfreed · 2021-11-02T17:44:41Z

src/error.rs

+
+    /// Labels categorizing the error.
+    #[serde(rename = "errorLabels", default)]
+    pub labels: Vec<String>,


Error labels are only ever actually returned at the top level of an error response, and it's merely a quirk / bug in the spec tests that they appear in the write concern error. @isabelatkinson actually just put up a PR to fix that: mongodb/specifications#1086.

For that reason, I don't think we should include public API support for this case, and if need be we can just skip the affected tests until the PR is merged. I think I remember you saying we may have to wait for a serverless change on this too to support the labels being specified in the correct place.

The serverless proxy apparently enforces that they appear here; I've removed this field from the public API but I'd prefer to leave it and the related logic in place so the tests can continue to pass.

have we confirmed that this behavior is correct/desired on the serverless end? it seems like a weird inconsistency. I think it's fine to have it within the driver right now so that we can run the tests but in the long run it would be nice not to have to maintain a special case like this

I'm in the process of following up about this on the serverless side - it looks like this isn't expected behavior. I'll make sure to send a follow-up PR for the driver (even if it's just a TODO with a bug reference) once that gets sorted out.

SGTM to leave this in temporarily until serverless fixes the issue on their end.

I've added todos here referencing CLOUDP-105256.

isabelatkinson · 2021-11-03T14:52:17Z

src/client/options/mod.rs

@@ -1232,6 +1236,10 @@ impl ClientOptions {
                original_uri
            ]
        );
+        #[cfg(test)]


Do we need #[cfg(test)] here? The method already has it

Derp, fixed.

isabelatkinson · 2021-11-03T14:57:16Z

src/test/spec/retryable_writes/mod.rs

@@ -179,6 +179,8 @@ async fn run_spec_tests() {
                    .await
                    .unwrap();
            }
+
+            let _ = client.database(&db_name).drop(None).await;


I think we should unwrap here to make sure this doesn't silently fail

patrickfreed

LGTM!

abr-egn requested review from patrickfreed and isabelatkinson November 1, 2021 19:31

abr-egn marked this pull request as ready for review November 1, 2021 19:31

abr-egn commented Nov 2, 2021

View reviewed changes

patrickfreed reviewed Nov 2, 2021

View reviewed changes

abr-egn force-pushed the RUST-656/serverless-test branch from eb834f7 to 0715c6b Compare November 2, 2021 19:53

isabelatkinson reviewed Nov 3, 2021

View reviewed changes

patrickfreed approved these changes Nov 4, 2021

View reviewed changes

abr-egn force-pushed the RUST-656/serverless-test branch from 62d8f8f to 932207f Compare November 8, 2021 17:12

isabelatkinson approved these changes Nov 8, 2021

View reviewed changes

abr-egn added 20 commits November 8, 2021 15:00

evergreen config wip

0cc16d1

the rest of the initial serverless setup

50445a8

only run tests listed in the serverless spec

bbfe3a5

fix test client srv resolution, other minor things

e2a1be7

allow error labels nested in write concerns

98dd479

merge labels into toplevel; clean up dbs

7616e77

properly set serverless credentials for unified test clients

ee9dff2

test script updates

3e74b2b

rustfmt

d790439

newline

981aa40

allow unset write concern error labels

e64f763

refix option merging

71213f5

rustfmt and clippy

9f73e72

override default hosts

abf119c

another stab at test host logic

5feeac1

export results as junit

8f5e711

env

68b26f6

rustfmt

24a6f0f

review updates

7e481c9

review updates

d3c86c2

add todos

2682541

abr-egn force-pushed the RUST-656/serverless-test branch from 932207f to 2682541 Compare November 8, 2021 20:00

abr-egn merged commit 8311c41 into mongodb:master Nov 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RUST-656 Run tests against serverless in evergreen #504

RUST-656 Run tests against serverless in evergreen #504

abr-egn commented Oct 28, 2021 •

edited

Loading

abr-egn Oct 28, 2021

abr-egn Oct 28, 2021

abr-egn Nov 2, 2021

patrickfreed Nov 2, 2021

abr-egn Nov 2, 2021

isabelatkinson Nov 3, 2021

abr-egn Nov 3, 2021

patrickfreed Nov 4, 2021

abr-egn Nov 8, 2021

isabelatkinson Nov 3, 2021

abr-egn Nov 3, 2021

isabelatkinson Nov 3, 2021

abr-egn Nov 3, 2021

patrickfreed left a comment

RUST-656 Run tests against serverless in evergreen #504

RUST-656 Run tests against serverless in evergreen #504

Conversation

abr-egn commented Oct 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickfreed left a comment

Choose a reason for hiding this comment

abr-egn commented Oct 28, 2021 •

edited

Loading