add S3 minimum part size defined by the user #17171
Conversation
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
vitess-bot
bot
added
NeedsBackportReason
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #17171 +/- ##
==========================================
+ Coverage 67.46% 67.52% +0.06%
==========================================
Files 1581 1581
Lines 253934 253966 +32
==========================================
+ Hits 171305 171492 +187
+ Misses 82629 82474 -155 ☔ View full report in Codecov by Sentry. |
rvrangel
force-pushed
the
s3-minimum-partsize
branch
from
44ede6d to
807c22b
Compare
rvrangel
requested review from
deepthi,
mattlord,
ajm188,
harshit-gangal,
rohit-nayak-ps,
systay and
frouioui
as code owners
rvrangel
changed the title
shlomi-noach
added
Type: Enhancement
frouioui
removed
NeedsDescriptionUpdate
rvrangel
requested review from
notfelineit,
GuptaManan100,
shlomi-noach,
derekperkins,
timvaillancourt and
arthurschreiber
as code owners
rvrangel
force-pushed
the
s3-minimum-partsize
branch
from
7f7d336 to
0f08ff4
Compare
|
Hey @rvrangel the history of that Pull Request looks a bit broken. It needs to be rebased. |
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
rvrangel
force-pushed
the
s3-minimum-partsize
branch
from
5994e0e to
9ab638a
Compare
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
| @@ -98,6 +109,7 @@ func registerFlags(fs *pflag.FlagSet) { | |||
| fs.BoolVar(&tlsSkipVerifyCert, "s3_backup_tls_skip_verify_cert", false, "skip the 'certificate is valid' check for SSL connections.") | |||
| fs.StringVar(&requiredLogLevel, "s3_backup_log_level", "LogOff", "determine the S3 loglevel to use from LogOff, LogDebug, LogDebugWithSigning, LogDebugWithHTTPBody, LogDebugWithRequestRetries, LogDebugWithRequestErrors.") | |||
| fs.StringVar(&sse, "s3_backup_server_side_encryption", "", "server-side encryption algorithm (e.g., AES256, aws:kms, sse_c:/path/to/key/file).") | |||
| fs.Int64Var(&minPartSize, "s3_backup_aws_min_partsize", manager.MinUploadPartSize, "Minimum part size to use, defaults to 5MiB but can be increased due to the dataset size.") | |||
There was a problem hiding this comment.
This new flag must also be added to vitess-operator there:
https://github.com/planetscale/vitess-operator/blob/6e2323e6d536ac5f7fe6be2256479611ca68d3a1/pkg/operator/vitessbackup/storage_s3.go#L27-L39
There was a problem hiding this comment.
is it only needed on that struct so we can load the definitions? in any case we can discuss in planetscale/vitess-operator#645
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
frouioui
removed request for
arthurschreiber,
vmg,
systay,
shlomi-noach,
timvaillancourt,
derekperkins,
ajm188,
harshit-gangal,
notfelineit,
GuptaManan100 and
rohit-nayak-ps
There was a problem hiding this comment.
Looks good to me after the latest changes.
Only comment left is the one I wrote on planetscale/vitess-operator#645 (review). We should port the operator-latest.yaml changes made in vtop to this repo's ./examples/operator/operator.yaml.
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
|
I updated the yaml file to be in sync with the other PR, let me know if it looks okay now |
| return nil, err | ||
| } | ||
|
|
||
| bh.bs.params.Logger.Infof("Using S3 upload part size: %s", humanize.IBytes(uint64(partSizeBytes))) |
There was a problem hiding this comment.
I wonder if it's worth adding a project dependency on humanize for this and the error message below. The flag is in bytes, so I think it would actually be better to log the bytes here as well to confirm that the expected/flag value is used.
There was a problem hiding this comment.
it was already an indirect dependency, so I thought I would use it :)
I can remove it tomorrow since the preference is to avoid it
There was a problem hiding this comment.
Ah, you're right! Sorry, for some reason I thought we added it to go.mod here. No reason not to use it here then.
There was a problem hiding this comment.
okay, I will leave it then and just update the comment as mentioned below!
go/vt/mysqlctl/s3backupstorage/s3.go
Outdated
| @@ -213,9 +228,11 @@ func (bh *S3BackupHandle) handleAddFile(ctx context.Context, filename string, pa | |||
| }() | |||
| } | |||
|
|
|||
| func calculateUploadPartSize(filesize int64) int64 { | |||
| // this is a helper to calculate the part size, taking into consideration the minimum part size | |||
There was a problem hiding this comment.
Nit, but the function comment should start with the function name. Some linters will complain about this.
Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
Description
This adds a new parameter to allow the operator to specify a minimum part size for the S3 uploads.
Related Issue(s)
Checklist