Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloud/gcp: TestFileDoesNotExist failed #135307

Closed
cockroach-teamcity opened this issue Nov 15, 2024 · 5 comments · Fixed by #135565 · May be fixed by #135595
Closed

cloud/gcp: TestFileDoesNotExist failed #135307

cockroach-teamcity opened this issue Nov 15, 2024 · 5 comments · Fixed by #135565 · May be fixed by #135595
Assignees
@jeffswenson
Labels
A-disaster-recovery branch-master Failures and bugs on the master branch. branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-disaster-recovery

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 15, 2024 •
edited by cockroach-jira-scripts
Loading

cloud/gcp.TestFileDoesNotExist failed with artifacts on master @ cd7b29726f13528f2c435cd46a9f7af2b341cb1a:

=== RUN   TestFileDoesNotExist
    gcs_storage_test.go:377: 
        	Error Trace:	pkg/cloud/gcp/gcs_storage_test.go:377
        	Error:      	Should be true
        	Test:       	TestFileDoesNotExist
--- FAIL: TestFileDoesNotExist (0.74s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-44448
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Nov 15, 2024
This was referenced Nov 15, 2024
Closed
Closed
Closed
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

cloud/gcp.TestFileDoesNotExist failed with artifacts on master @ 69fbd56c594624b45f421617728ff00462020ff5:

=== RUN   TestFileDoesNotExist
    gcs_storage_test.go:377: 
        	Error Trace:	pkg/cloud/gcp/gcs_storage_test.go:377
        	Error:      	Should be true
        	Test:       	TestFileDoesNotExist
--- FAIL: TestFileDoesNotExist (0.70s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 16, 2024
Open
@cockroach-teamcity
Copy link
Member Author

cloud/gcp.TestFileDoesNotExist failed with artifacts on master @ a5ab06046e817627cd0b64390f4ede0c4a5d82ef:

=== RUN   TestFileDoesNotExist
    gcs_storage_test.go:377: 
        	Error Trace:	pkg/cloud/gcp/gcs_storage_test.go:377
        	Error:      	Should be true
        	Test:       	TestFileDoesNotExist
--- FAIL: TestFileDoesNotExist (0.77s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

cloud/gcp.TestFileDoesNotExist failed with artifacts on master @ 864b6c3c540a7fb2039227d25e3d785cbdbc769f:

=== RUN   TestFileDoesNotExist
    gcs_storage_test.go:377: 
        	Error Trace:	pkg/cloud/gcp/gcs_storage_test.go:377
        	Error:      	Should be true
        	Test:       	TestFileDoesNotExist
--- FAIL: TestFileDoesNotExist (0.68s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@jeffswenson jeffswenson self-assigned this Nov 18, 2024
@jeffswenson
Copy link
Collaborator

Okay. I think what happened is GCP behavior changed. This started failing on all branches without a correlated back port. Here's the error the test is seeing:

open tpch-csv/sf-1/region.tbl: googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>teamcity-nightly@cockroach-ephemeral.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>"
                          "googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>teamcity-nightly@cockroach-ephemeral.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>

Instead of returning not found, its returning a permission denied error. This isn't listed in the change log for GCS: https://cloud.google.com/storage/docs/release-notes.

@jeffswenson jeffswenson removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 18, 2024
jeffswenson added a commit to jeffswenson/cockroach that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
a8aeb9e 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: cockroachdb#135307
Fixes: cockroachdb#135348
Fixes: cockroachdb#135349
Fixes: cockroachdb#135350
Fixes: cockroachdb#135351
Fixes: cockroachdb#135352
@jeffswenson jeffswenson mentioned this issue Nov 18, 2024
Merged
jeffswenson added a commit to jeffswenson/cockroach that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
1807b28 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: cockroachdb#135307
Fixes: cockroachdb#135348
Fixes: cockroachdb#135349
Fixes: cockroachdb#135350
Fixes: cockroachdb#135351
Fixes: cockroachdb#135352
@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Nov 18, 2024
craig bot pushed a commit that referenced this issue Nov 18, 2024
@dt @mgartner
@RaduBerinde @dhartunian @souravcrl @jeffswenson
Merge #126587 #134228 #135291 #135337 #135552 #135565
c7b4b54 
126587: kvpb: removed disused Path field from ExportResp r=dt a=dt

Release note: none.
Epic: none.

134228: rpc: reduce certificate-related allocations during authentication r=mgartner a=mgartner

#### rpc: add authenication benchmark

Release note: None

#### rpc: reduce certificate-related allocations during authentication

This commit avoids certificate-related allocations in the happy path of
authenticating network requests.

Release note: None

#### security: inline getCertificatePrincipals

The `getCertificatePrincipals` function has been inlined into its one
callsite to avoid allocating a slice of the results.

Fixes #133317

Release note: None

135291: logictest: enable local-mixed-24.3 config r=RaduBerinde a=RaduBerinde

#### bootstrap: add 24.3 bootstrap data

Obtained by running (on release-24.3 branch):
```
./dev build sql-bootstrap-data && bin/sql-bootstrap-data
```

Epic: REL-1322
Release note: None

#### logictest: enable local-mixed-24.3 config

Includes a minor fix to the schema changer (thanks Rafi and Annie!).

Fixes #135358

Epic: REL-1322
Release note: None


135337: cli: skip TestTenantZip under deadlock r=dhartunian a=dhartunian

We had some CPU profile failures under deadlock, skipping under that scenario since that's not relevant for this test.

Resolves: #134187

Release note: None

135552: ldapccl: support partial ldap groups mapping on authz r=souravcrl a=souravcrl

fixes #133779
Epic CRDB-33829

Currently, an LDAP user may be configured to be a member of multiple LDAP groups, and when we retrieve the group roles for the user during LDAP authZ we try to find corresponding roles for all the groups being synced and fail the roles grant operation if any of the group roles do not exist on CRDB. This is a problem as not all groups are desired to have corresponding CRDB roles and in such a case a partial roles grant should take place.

Release note(security):  This fix will add support for partial roles from ldap synced group to be mapped to crdb roles and ensure appropriate erroring for undesired behavior.

135565: gcs: remove test that asserts buckets produce not found error r=jeffswenson a=jeffswenson

The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352

Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>
Co-authored-by: Radu Berinde <radu@cockroachlabs.com>
Co-authored-by: David Hartunian <davidh@cockroachlabs.com>
Co-authored-by: souravcrl <sourav.sarangi@cockroachlabs.com>
Co-authored-by: Jeff Swenson <jeffswenson@betterthannull.com>
@craig craig bot closed this as completed in 83ece20 Nov 18, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 18, 2024

Based on the specified backports for linked PR #135565, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3, branch-release-24.3.0-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 labels Nov 18, 2024
@blathers-crl blathers-crl bot added branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 branch-release-24.3.0-rc labels Nov 18, 2024
blathers-crl bot pushed a commit that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
20d6da5 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352
@blathers-crl blathers-crl bot mentioned this issue Nov 18, 2024
Closed
blathers-crl bot pushed a commit that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
ed283b6 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352
@blathers-crl blathers-crl bot mentioned this issue Nov 18, 2024
Closed
blathers-crl bot pushed a commit that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
e57afd8 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352
@blathers-crl blathers-crl bot mentioned this issue Nov 18, 2024
Closed
blathers-crl bot pushed a commit that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
2651711 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352
@blathers-crl blathers-crl bot mentioned this issue Nov 18, 2024
Closed
blathers-crl bot pushed a commit that referenced this issue Nov 18, 2024
@jeffswenson
gcs: remove test that asserts buckets produce not found error
f7f89a0 
The behavior of GCS appears to have changed. If a bucket does not exist,
it will return a 403 forbidden. So we can't disambiguate a bucket that
does not exist from a bucket we can't access.

Overall, this is annoying but fine. The main value of not found as a
sentinel error the program can check is it indicates the name is free
and can be used. This property doesn't apply if the bucket does not
exist because CRDB will never create a bucket.

Release Justification: Test only change
Release Note: None
Fixes: #135307
Fixes: #135348
Fixes: #135349
Fixes: #135350
Fixes: #135351
Fixes: #135352
@blathers-crl blathers-crl bot mentioned this issue Nov 18, 2024
Closed
jeffswenson added a commit to jeffswenson/cockroach that referenced this issue Nov 18, 2024
@jeffswenson
cloud: randomize name of buckets that do not exist
93843e3 
Previously, we used a hard coded bucket name when tested the GCS and S3
behavior of buckets that do not exist. This made it possible for people
to create that bucket and break the test. Now, the test uses a random
bucket name.

Release Justification: Test Only Change
Release Note: None
Fixes: cockroachdb#135307
Fixes: cockroachdb#135348
Fixes: cockroachdb#135349
Fixes: cockroachdb#135350
Fixes: cockroachdb#135351
Fixes: cockroachdb#135352
@jeffswenson jeffswenson mentioned this issue Nov 18, 2024
Merged
jeffswenson added a commit to jeffswenson/cockroach that referenced this issue Nov 18, 2024
@jeffswenson
cloud: use uuid as bucket name
b0ce78a 
Previously, the GCS and S3 tests used hard coded bucket names to test
the behavior of buckets that do not exist. Now, the tests use a random
UUID. This change is necessary because someone appears to have created
the GCP bucket which causes the test to fail.

Release Justification: Test Only Change
Release Note: None
Fixes: cockroachdb#135307
Fixes: cockroachdb#135348
Fixes: cockroachdb#135349
Fixes: cockroachdb#135350
Fixes: cockroachdb#135351
Fixes: cockroachdb#135352
@jeffswenson jeffswenson mentioned this issue Nov 18, 2024
Open
craig bot pushed a commit that referenced this issue Nov 18, 2024
@kev-cao @cockroach-teamcity
@rickystewart @jeffswenson
Merge #135135 #135581 #135616 #135619
e9c8cd9 
135135: backupccl: clean up old restore syntax code r=msbutler a=kev-cao

The old restore syntax was fully removed in #133610. This commit removes all the old code that ran for the old syntax.

Epic: none

Release note: none

135581: orchestration: released CockroachDB version 24.2.5. Next version: 24.2.6 r=fantapop a=cockroach-teamcity


Release note: None
Epic: None
Release justification: non-production (release infra) change.


135616: build: allow `bazel-generate.sh` to detect changes to untracked files r=rail,RaduBerinde a=rickystewart

In some scenarios, the existing logic can be insufficient to detect changes as file generation can create untracked files, which are then not spotted by `git diff`. In addition to the `git diff`, we also run `git status` to spot these files specifically.

Closes: #135345
Epic: CRDB-17171
Release note: None

135619: Revert "gcs: remove test that asserts buckets produce not found error" r=jeffswenson a=jeffswenson

This reverts commit 83ece20.

Release Note: None
Part of: #135307
Part of: #135348
Part of: #135349
Part of: #135350
Part of: #135351
Part of: #135352

Co-authored-by: Kevin Cao <kevin.cao@cockroachlabs.com>
Co-authored-by: Justin Beaver <teamcity@cockroachlabs.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Jeff Swenson <jeffswenson@betterthannull.com>
jeffswenson added a commit to jeffswenson/cockroach that referenced this issue Nov 18, 2024
@jeffswenson
cloud: use uuid as bucket name
627c9ac 
Previously, the GCS and S3 tests used hard coded bucket names to test
the behavior of buckets that do not exist. Now, the tests use a random
UUID. This change is necessary because someone appears to have created
the GCP bucket which causes the test to fail.

Release Justification: Test Only Change
Release Note: None
Fixes: cockroachdb#135307
Fixes: cockroachdb#135348
Fixes: cockroachdb#135349
Fixes: cockroachdb#135350
Fixes: cockroachdb#135351
Fixes: cockroachdb#135352
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jeffswenson jeffswenson

Labels
A-disaster-recovery branch-master Failures and bugs on the master branch. branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 branch-release-24.3.0-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-disaster-recovery
Projects
None yet
Milestone
No milestone
2 participants
@jeffswenson @cockroach-teamcity