-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure that binary logs for PITR are in a shared directory #541
base: main
Are you sure you want to change the base?
Conversation
1824f91
to
0562d49
Compare
Signed-off-by: Matt Lord <mattalord@gmail.com>
0562d49
to
d259730
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Should we also provide the flag in yaml
files?
Signed-off-by: Matt Lord <mattalord@gmail.com>
Yeah. I think this does it. e2e5e8b |
How is the value being set, and to what specific value? |
The user would specify the flag and value in their cluster yaml definition using the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems to me like it's feature complete and can be taken out of Draft
?
Signed-off-by: Matt Lord <mattalord@gmail.com>
The flag ended up being for |
Signed-off-by: Matt Lord <mattalord@gmail.com>
pkg/operator/vttablet/pod.go
Outdated
// Ensure that binary logs are restored to/from a location that all containers | ||
// in the pod can access if no location was explicitly provided. | ||
if _, ok := vttabletAllFlags["builtinbackup-incremental-restore-path"]; !ok { | ||
vttabletAllFlags["builtinbackup-incremental-restore-path"] = vtDataRootPath | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What would happen if the path specified in --builtinbackup-incremental-restore-path
is not accessible to all containers in the pod?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am guessing it is up to the user to set the same value on all components too? mysqlctl
, vttablet
and vtbackup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same thing that happens now to every user. It doesn't work. PITR does not generally work in the operator today.
pkg/operator/vttablet/mysqlctld.go
Outdated
@@ -29,6 +29,7 @@ const ( | |||
vtRootInitScript = `set -ex | |||
mkdir -p /mnt/vt/bin | |||
cp --no-clobber /vt/bin/mysqlctld /mnt/vt/bin/ | |||
cp --no-clobber $(command -v mysqlbinlog) /mnt/vt/bin/ || true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The directory /mnt/.../
is shared across all the containers in the pod I am assuming? In which case, this line would resolve what you wrote in the PR's description:
/tmp
is not a shared mount point within the pod so whenmysqlbinlog
subsequently tries to read them from within the mysqld container it cannot find them in its container's/tmp
directory and it fails with an error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is simply about copying the mysqlbinlog binary from the vitess/lite container image to the mysqlctld/vtbackup container (if it's not already there), as it looks like we'll need to keep that around in the lite image because the MySQL images do not contain that binary and it's needed for PITR.
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
4abb9ce
to
a2d80d2
Compare
Signed-off-by: Matt Lord <mattalord@gmail.com>
a2d80d2
to
63abf80
Compare
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
c9793dd
to
e069303
Compare
This includes mysqld (of course) and mysqlbinlog But it does NOT include xtrabackup Signed-off-by: Matt Lord <mattalord@gmail.com>
e069303
to
2aeb36b
Compare
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
ae78d72
to
5e7df2b
Compare
When executing the
vtctldclient RestoreFromBackup --restore-to-pos <value>
command, thevttablet
process in thevttablet
container within thevttablet
pod — in theRestoreFromBackup
tabletmanager RPC — restores the full backup within theVTDATAROOT
(specifically/vt/vtdataroot/vt_<tabletUID>/
for the mysql data) that is shared by all containers within the pod using the configured backup engine (e.g.xtrabackup
). It orchestrates that in conjunction with themysqlctld
process that's running inside themysqld
container within the samevttablet
pod. In the end there is a runningmysqld
instance inside themysqld
container that is from the restored full backup. Then once the full backup is in place and themysqld
process is running thevttablet
process uses the OS tmp dir of/tmp
to restore the binary logs from the backup — via thebuiltinbackupengine
— for subsequent application and/tmp
is not a shared mount point within the pod so whenmysqlbinlog
subsequently tries to read them from within themysqld
container it cannot find them in its container's/tmp
directory and it fails with an error.vtctldclient
https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/cmd/vtctldclient/command/backups.go#L227-L263
vtctld
[server]https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/vtctl/grpcvtctldserver/server.go#L3260-L3286
vttablet
https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/vttablet/tabletmanager/rpc_backup.go#L173-L193
https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/vttablet/tabletmanager/restore.go#L191-L273
mysqlctld
(rather thanmysqlctl
, and which runs in themysql
container)https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/mysqlctl/backup.go#L364-L487
https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/mysqlctl/builtinbackupengine.go#L995-L1060
vttablet
builtinbackupengine
https://github.com/vitessio/vitess/blob/3ae5cf7e690e560dd5630119215bcc3f5ecf31c8/go/vt/mysqlctl/builtinbackupengine.go#L995-L1060
Related issues and PRs: