This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
storage: extend the ExternalStorage interface and implements
#446
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kennytm
reviewed
Jul 30, 2020
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
kennytm
reviewed
Aug 3, 2020
pkg/storage/flags.go
Outdated
Comment on lines
10
to
13
| func DefineFlags(flags *pflag.FlagSet) { | ||
| defineS3Flags(flags) | ||
| DefineS3Flags(flags) | ||
| defineGCSFlags(flags) | ||
| } |
There was a problem hiding this comment.
what is the rationale of only exposing S3 but not GCS?
|
@kennytm Shall we change the |
|
@glorv we don't really need those flags since they can be passed via URL parameters ( Given that pflag refuses to support |
overvenus
reviewed
Aug 4, 2020
Comment on lines
+373
to
+378
| // FIXME: we test in minio, when request with Range, the result.AcceptRanges is a bare string 'range', | ||
| // not sure whether this is a feature or bug | ||
| //if rangeOffset != nil && (result.AcceptRanges == nil || *result.AcceptRanges != *rangeOffset) { | ||
| // return nil, errors.Errorf("open file '%s' failed, expected range: %s, got: %v", | ||
| // name, *rangeOffset, result.AcceptRanges) | ||
| //} |
Comment on lines
+77
to
+78
| // export for test. | ||
| func NewLocalStorage(base string) (*LocalStorage, error) { |
There was a problem hiding this comment.
used in lightning tests
pkg/storage/s3.go
Outdated
| return realOffset, nil | ||
| } | ||
|
|
||
| // if seek ahead no more than 64k, we read add drop these data |
There was a problem hiding this comment.
Suggested change
| // if seek ahead no more than 64k, we read add drop these data | |
| // if seek ahead no more than 64k, we discard these data |
pkg/storage/s3.go
Outdated
| return 0, err | ||
| } | ||
|
|
||
| newReader, err := r.storage.open(context.Background(), r.name, realOffset, 0) |
There was a problem hiding this comment.
Please avoid use Background context.
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
|
LGTM |
|
/run-all-tests |
ti-srebot
approved these changes
Aug 11, 2020
|
/merge |
|
/run-all-tests |
glorv
added a commit
to glorv/br
that referenced
this pull request
Aug 17, 2020
…ap#446) * update storage * update * fix * export s3 flags * fix local path * fix s3 reader * fix range offset * fix the req range errors * fix comments * fix review comments * fix some comments * update * fix * update Co-authored-by: ti-srebot <66930949+ti-srebot@users.noreply.github.com>
3pointer
pushed a commit
that referenced
this pull request
Aug 17, 2020
…#462) * update storage * update * fix * export s3 flags * fix local path * fix s3 reader * fix range offset * fix the req range errors * fix comments * fix review comments * fix some comments * update * fix * update Co-authored-by: ti-srebot <66930949+ti-srebot@users.noreply.github.com> Co-authored-by: ti-srebot <66930949+ti-srebot@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
extend the
ExternalStorageinterface and implement the new methods for local and s3 storage. This extension is currently used in lightning to support restore from s3 storage.What is changed and how it works?
Add new methods for ExternalStorage:
Since s3 GetObject does not provide seek directly, we implement this by: 1) read and skip the bytes if the distance is small, else we open a new reader set the start position to the seek position.
Check List
Tests
Code changes
Side effects
Related changes
Release Note