Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to access the gcs when use custom http client #47022

Closed
Leavrth opened this issue Sep 18, 2023 · 2 comments · Fixed by #47026
Closed

failed to access the gcs when use custom http client #47022

Leavrth opened this issue Sep 18, 2023 · 2 comments · Fixed by #47026
Assignees
Labels
affects-6.5 affects-7.1 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@Leavrth
Copy link
Contributor

Leavrth commented Sep 18, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. restore data from gcs

2. What did you expect to see? (Required)

success

3. What did you see instead (Required)

failed

Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).

4. What is your TiDB version? (Required)

master, v6.5.4

@Leavrth Leavrth added type/bug The issue is confirmed as a bug. component/br This issue is related to BR of TiDB. affects-6.5 affects-7.1 labels Sep 18, 2023
@Leavrth
Copy link
Contributor Author

Leavrth commented Sep 18, 2023

The gcs credential is saved in the Transport of http.Client. So we need to support the HTTPClient with the credential:
https://github.com/googleapis/google-api-go-client/blob/f79df4875aea4520e4aff5c5ebceb9e01b7b60a2/transport/http/dial.go#L42

@Leavrth Leavrth self-assigned this Sep 18, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 labels Sep 18, 2023
@Leavrth Leavrth removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 labels Sep 18, 2023
ti-chi-bot bot pushed a commit that referenced this issue Sep 18, 2023
@YuJuncen
Copy link
Contributor

There is another solution for this, we can wrap the transport provided in the users' HTTPClient if there are:

diff --git a/br/pkg/storage/gcs.go b/br/pkg/storage/gcs.go
index f32d4344d9..468b03dda1 100644
--- a/br/pkg/storage/gcs.go
+++ b/br/pkg/storage/gcs.go
@@ -20,6 +20,8 @@ import (
        "golang.org/x/oauth2/google"
        "google.golang.org/api/iterator"
        "google.golang.org/api/option"
+
+       htransport "google.golang.org/api/transport/http"
 )
 
 const (
@@ -292,11 +294,16 @@ func NewGCSStorage(ctx context.Context, gcs *backuppb.GCS, opts *ExternalStorage
                        clientOps = append(clientOps, option.WithCredentialsJSON([]byte(gcs.GetCredentialsBlob())))
                }
        }
-
        if gcs.Endpoint != "" {
                clientOps = append(clientOps, option.WithEndpoint(gcs.Endpoint))
        }
        if opts.HTTPClient != nil {
+               var err error
+               opts.HTTPClient.Transport, err = htransport.NewTransport(ctx, opts.HTTPClient.Transport,
+                       append(clientOps, option.WithScopes(storage.ScopeFullControl, "https://www.googleapis.com/auth/cloud-platform"))...)
+               if err != nil {
+                       return nil, errors.Trace(err)
+               }
                clientOps = append(clientOps, option.WithHTTPClient(opts.HTTPClient))
        }
        client, err := storage.NewClient(ctx, clientOps...)
diff --git a/br/pkg/storage/gcs_test.go b/br/pkg/storage/gcs_test.go
index c6d7e14b80..31c68c5ebc 100644

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 affects-7.1 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants