Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement commitlog rotation as well as writing out, cleaning up, and bootstrapping using snapshot metadata files #1384

Merged
merged 13 commits into from
Mar 2, 2019
Merged

Implement commitlog rotation as well as writing out, cleaning up, and bootstrapping using snapshot metadata files #1384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
fc161c7
squash
Feb 15, 2019
deaf83a
rename proto field to camelcase
Feb 15, 2019
e172e49
add TODO to remove field in v1
Feb 15, 2019
bba6ec8
add issue in todo
Feb 15, 2019
2127865
rename struct and add comment
Feb 15, 2019
56753aa
rename function
Feb 15, 2019
ed8e4aa
commitlog -> commitLog
Feb 15, 2019
dd560a1
regen mocks
Feb 15, 2019
4b3e35c
clarify code and add tes case
Feb 15, 2019
a9c1409
remove extranneosu check
Feb 15, 2019
609b9a8
delete unused structs
Feb 16, 2019
fc1bc6a
only read files that were on disk before start
Feb 16, 2019
d076698
regen mocks and fix comment
Mar 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion kube/bundle.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion kube/m3dbnode-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,6 @@ data:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
Expand Down
1 change: 0 additions & 1 deletion scripts/development/m3_stack/m3dbnode.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
Expand Down
4 changes: 2 additions & 2 deletions src/cmd/services/m3dbnode/config/bootstrap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,13 @@ type BootstrapFilesystemConfiguration struct {

// BootstrapCommitlogConfiguration specifies config for the commitlog bootstrapper.
type BootstrapCommitlogConfiguration struct {
// ReturnUnfulfilledForCorruptCommitlogFiles controls whether the commitlog bootstrapper
// ReturnUnfulfilledForCorruptCommitLogFiles controls whether the commitlog bootstrapper
// will return unfulfilled for all shard time ranges when it encounters a corrupt commit
// file. Note that regardless of this value, the commitlog bootstrapper will still try and
// read all the uncorrupted commitlog files and return as much data as it can, but setting
// this to true allows the node to attempt a repair if the peers bootstrapper is configured
// after the commitlog bootstrapper.
ReturnUnfulfilledForCorruptCommitlogFiles bool `yaml:"returnUnfulfilledForCorruptCommitlogFiles"`
ReturnUnfulfilledForCorruptCommitLogFiles bool `yaml:"returnUnfulfilledForCorruptCommitLogFiles"`
}

// New creates a bootstrap process based on the bootstrap configuration.
Expand Down
5 changes: 3 additions & 2 deletions src/cmd/services/m3dbnode/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,9 @@ type CommitLogPolicy struct {
// enough for almost all workloads assuming a reasonable batch size is used.
QueueChannel *CommitLogQueuePolicy `yaml:"queueChannel"`

// The commit log block size.
BlockSize time.Duration `yaml:"blockSize" validate:"nonzero"`
// Deprecated. Left in struct to keep old YAMLs parseable.
// TODO(V1): remove
DeprecatedBlockSize *time.Duration `yaml:"blockSize"`
}

// CalculationType is a type of configuration parameter.
Expand Down
3 changes: 1 addition & 2 deletions src/cmd/services/m3dbnode/config/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
Expand Down Expand Up @@ -411,7 +410,7 @@ func TestConfiguration(t *testing.T) {
calculationType: fixed
size: 2097152
queueChannel: null
blockSize: 10m0s
blockSize: null
repair:
enabled: false
interval: 2h0m0s
Expand Down
1 change: 0 additions & 1 deletion src/cmd/services/m3dbnode/main/main_index_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: {{.DataDir}}
Expand Down
1 change: 0 additions & 1 deletion src/cmd/services/m3dbnode/main/main_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,7 +498,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: {{.DataDir}}
Expand Down
1 change: 0 additions & 1 deletion src/dbnode/config/m3dbnode-cluster-template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
1 change: 0 additions & 1 deletion src/dbnode/config/m3dbnode-local-etcd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
Expand Down
1 change: 0 additions & 1 deletion src/dbnode/config/m3dbnode-local.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ db:
queue:
calculationType: fixed
size: 2097152
blockSize: 10m

fs:
filePathPrefix: /var/lib/m3db
Expand Down
2 changes: 1 addition & 1 deletion src/dbnode/generated/mocks/generate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

// mockgen rules for generating mocks for exported interfaces (reflection mode)

//go:generate sh -c "mockgen -package=fs $PACKAGE/src/dbnode/persist/fs DataFileSetWriter,DataFileSetReader,DataFileSetSeeker,IndexFileSetWriter,IndexFileSetReader,IndexSegmentFileSetWriter,IndexSegmentFileSet,IndexSegmentFile | genclean -pkg $PACKAGE/src/dbnode/persist/fs -out $GOPATH/src/$PACKAGE/src/dbnode/persist/fs/fs_mock.go"
//go:generate sh -c "mockgen -package=fs $PACKAGE/src/dbnode/persist/fs DataFileSetWriter,DataFileSetReader,DataFileSetSeeker,IndexFileSetWriter,IndexFileSetReader,IndexSegmentFileSetWriter,IndexSegmentFileSet,IndexSegmentFile,SnapshotMetadataFileWriter | genclean -pkg $PACKAGE/src/dbnode/persist/fs -out $GOPATH/src/$PACKAGE/src/dbnode/persist/fs/fs_mock.go"
//go:generate sh -c "mockgen -package=xio $PACKAGE/src/dbnode/x/xio SegmentReader,SegmentReaderPool | genclean -pkg $PACKAGE/src/dbnode/x/xio -out $GOPATH/src/$PACKAGE/src/dbnode/x/xio/io_mock.go"
//go:generate sh -c "mockgen -package=digest -destination=$GOPATH/src/$PACKAGE/src/dbnode/digest/digest_mock.go $PACKAGE/src/dbnode/digest ReaderWithDigest"
//go:generate sh -c "mockgen -package=series $PACKAGE/src/dbnode/storage/series DatabaseSeries,QueryableBlockRetriever | genclean -pkg $PACKAGE/src/dbnode/storage/series -out $GOPATH/src/$PACKAGE/src/dbnode/storage/series/series_mock.go"
Expand Down
Loading