Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vtctldclient MoveTables #13015

Merged
merged 104 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
dd6cac7
Add MoveTables RPC
mattlord May 1, 2023
baa26cf
More testing
mattlord May 2, 2023
0f54fc8
You guessed it
mattlord May 2, 2023
c3185cb
Get vtctldclient in sync with a dumb copy first
mattlord May 2, 2023
ea8d4d6
Move to tabletmanager MoveTablesCreate RPC
mattlord May 2, 2023
7542015
Move to tmc.CreateVRWorkflow
mattlord May 2, 2023
16fe219
Add DeleteVRWorkflow RPC
mattlord May 2, 2023
0763cf1
Response improvements
mattlord May 2, 2023
c0df28b
Update vtctldclient --help output
mattlord May 3, 2023
28c1c84
Cleanup artifacts on workflow Cancel/Delete
mattlord May 3, 2023
6ddd57e
Migrate TrafficSwitcher to workflow package.
mattlord May 4, 2023
f0dd0b2
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord May 4, 2023
afe099d
Getting Cancel/Delete artifact cleanup working
mattlord May 5, 2023
45575dd
Add ReadVRWorkflow tabletmanager RPC
mattlord May 6, 2023
2b6e318
Finish migration of workflow delete/cancel
mattlord May 7, 2023
5a900c1
Import cleanup
mattlord May 7, 2023
67a1b60
Add Workflow/MoveTables Show
mattlord May 8, 2023
287390a
Fix vreplication_log usage in VReplicationExec
mattlord May 8, 2023
babfa66
Add MoveTables Progress
mattlord May 8, 2023
fb0a524
Different output format for Progress
mattlord May 8, 2023
adab63c
Change Progress output again
mattlord May 9, 2023
b39e17d
Add MoveTables SwitchTraffic
mattlord May 9, 2023
7d79390
Rename WorkflowProgress to WorkflowStatus and add Progress alias
mattlord May 9, 2023
e4156f2
Add SwitchTraffic support
mattlord May 11, 2023
5f41041
Add workflow stop/start
mattlord May 12, 2023
93546ef
Add MoveTables start/stop
mattlord May 12, 2023
ed42ebb
SwitchTraffic improvements
mattlord May 12, 2023
c4dbb3d
Get Workflow list working
mattlord May 13, 2023
adbe57d
Separate traffic direction and whether or not to setup reverse replic…
mattlord May 15, 2023
fa9a7c0
Still fixing...
mattlord May 15, 2023
53bac0f
Fix the UpdateVRWorkflow unit test
mattlord May 17, 2023
baedee7
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord May 18, 2023
da6ac28
Update tablet picker
mattlord May 18, 2023
f9b878d
Fix ReverseTraffic
mattlord May 18, 2023
7792ed9
Add MoveTables complete
mattlord May 19, 2023
a329f2f
Finish Complete sub-command
mattlord May 19, 2023
adcc6f8
Migrate unit tests
mattlord May 19, 2023
3ee91a3
Apply vtctlclient change from:
mattlord May 24, 2023
c3e2bd5
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jun 6, 2023
47ec1f5
Port over https://github.com/vitessio/vitess/pull/13220
mattlord Jun 6, 2023
1bd0bee
Revert "Migrate unit tests"
mattlord Jun 6, 2023
088d535
WIP Migrating unit tests
mattlord Jun 6, 2023
9b4b4cd
Move some things in the v2 e2e tests to vtctldclient
mattlord Jun 7, 2023
316f33a
Correct BLS formation
mattlord Jun 7, 2023
341e176
Test fixing...
mattlord Jun 7, 2023
c0bf2a8
Fix binlog source generation
mattlord Jun 9, 2023
14bf40d
Add workflow wait to examples
mattlord Jun 9, 2023
82ad6ee
Remove invalid vtctldclient MoveTables Create flag
mattlord Jun 9, 2023
90254bf
Try to debug local examples in CI
mattlord Jun 9, 2023
c0a25a6
Local example related updates
mattlord Jun 9, 2023
6124c9f
Debugging failures
mattlord Jun 9, 2023
d5066e6
Change MoveTables Create flow and response
mattlord Jun 10, 2023
b9584da
Now I need to redo the materializer unit tests...
mattlord Jun 10, 2023
e4e5149
Adjust journal entry error message
mattlord Jun 10, 2023
5e18f85
Adjust and improve the new wait function
mattlord Jun 10, 2023
1e8f144
Get materializer MoveTables unit tests working
mattlord Jun 12, 2023
3be48f1
Move to using more concrete types in protos
mattlord Jun 15, 2023
b2fa98f
Move to using ReadVRWorkflow RPC
mattlord Jun 16, 2023
b3072e2
Traffic switching fixes
mattlord Jun 16, 2023
f4f97d3
Merge remote-tracking branch 'vitessio/main' into vtctldclient_moveta…
mattlord Jun 16, 2023
efbbdbf
Address various minor issues
mattlord Jun 16, 2023
eeeb544
Use vreplication state proto values everywhere
mattlord Jun 19, 2023
612ad4f
Corrections and tweaks after moving to proto type for status.
mattlord Jun 19, 2023
6ed7b32
Use unshortened name for new RPCs
mattlord Jun 19, 2023
e41c885
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 5, 2023
a849a54
Fixups after merging in origin/main
mattlord Jul 5, 2023
ad4f20d
Port over changes from https://github.com/vitessio/vitess/pull/13238
mattlord Jul 5, 2023
8336055
Improve dry-run output
mattlord Jul 6, 2023
a4b2da4
Add Logf helper function
mattlord Jul 7, 2023
46a20ae
Getting more unit tests working
mattlord Jul 7, 2023
5aec49f
Make text output the default
mattlord Jul 10, 2023
86498ef
Support JSON and TEXT format for more MoveTables commands
mattlord Jul 10, 2023
53f0120
Adjust external cluster name field for clarity
mattlord Jul 10, 2023
be3f733
Minor improvements after self review
mattlord Jul 11, 2023
aa5bdab
Get unit test framework in place for testing MoveTablesCreate
mattlord Jul 11, 2023
ddf1938
Minor cleanup around onDDL defaults
mattlord Jul 12, 2023
b5925a0
Unit test improvements
mattlord Jul 12, 2023
8343c0d
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 12, 2023
6a3b978
Leave vtctl/wrangler code unchanged
mattlord Jul 13, 2023
8aee04f
Get rid of irrelevant old tests
mattlord Jul 13, 2023
d3d6ff3
Starting to think about additional unit tests
mattlord Jul 13, 2023
5a350f1
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 13, 2023
8559d42
Add more general MoveTables unit test
mattlord Jul 14, 2023
a86ea39
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 14, 2023
c15cca2
Minor tweaks
mattlord Jul 14, 2023
cc1e925
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 17, 2023
de4884e
Migrate fixes from https://github.com/vitessio/vitess/pull/13515
mattlord Jul 17, 2023
16559a9
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 17, 2023
8f0bdfb
Align and unify command capitalization and examples
mattlord Jul 18, 2023
e1e19a9
Fix docgen back links
mattlord Jul 18, 2023
f332d35
Use standard sentence punctuation for command Short output
mattlord Jul 18, 2023
5ec944e
Properly fix doxgen generation (after testing)
mattlord Jul 18, 2023
0aec5c7
Update vtctldclient help output
mattlord Jul 18, 2023
6a05959
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 19, 2023
b12aa14
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 20, 2023
6aba116
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 21, 2023
7ff6ba4
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 24, 2023
84c7590
Revert incompatible existing proto msg change
mattlord Jul 24, 2023
eeaf270
Undo errant default value change
mattlord Jul 24, 2023
fe3db85
Address review comments
mattlord Jul 25, 2023
4ddf930
Merge remote-tracking branch 'origin/main' into vtctldclient_movetables
mattlord Jul 25, 2023
b602cef
Minor changes after self review
mattlord Jul 25, 2023
ea3910a
Move format check before cli.FinishedParsing
mattlord Jul 25, 2023
659f1cb
Add nil check and handling for issue seen
mattlord Jul 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/common/env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ fi

# mysqld might be in /usr/sbin which will not be in the default PATH
PATH="/usr/sbin:$PATH"
for binary in mysqld etcd etcdctl curl vtctlclient vttablet vtgate vtctld mysqlctl; do
for binary in mysqld etcd etcdctl curl vtctlclient vtctldclient vttablet vtgate vtctld mysqlctl; do
command -v "$binary" > /dev/null || fail "${binary} is not installed in PATH. See https://vitess.io/docs/get-started/local/ for install instructions."
done;

Expand Down
29 changes: 29 additions & 0 deletions examples/common/lib/utils.sh
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,35 @@ function wait_for_healthy_shard() {
wait_for_shard_vreplication_engine "${keyspace}" "${shard}"
}

# Wait for a workflow to reach the running state. Example:
# wait_for_workflow_running customer customer2customer
function wait_for_workflow_running() {
if [[ -z ${1} || -z ${2} ]]; then
fail "A keyspace and workflow must be specified when waiting for a workflow to reach the running state"
fi

local keyspace=${1}
local workflow=${2}
local wait_secs=90
local result=""

echo "Waiting for the ${workflow} workflow in the ${keyspace} keyspace to finish the copy phase..."

for _ in $(seq 1 ${wait_secs}); do
result=$(vtctldclient Workflow --keyspace="${keyspace}" show --workflow="${workflow}" 2>/dev/null | grep "Copy phase completed")
if [[ ${result} != "" ]]; then
break
fi
sleep 1
done;

if [[ ${result} == "" ]]; then
fail "Timed out after ${wait_secs} seconds waiting for the ${workflow} workflow in the ${keyspace} keyspace to reach the running state"
fi

echo "The ${workflow} workflow in the ${keyspace} keyspace is now running. $(sed -rn 's/.*"(Copy phase.*)".*/\1/p' <<< "${result}")."
}

# Stop the specified binary name using the provided PID file.
# Example:
# stop_process "vtadmin-web" "$VTDATAROOT/tmp/vtadmin-web.pid"
Expand Down
5 changes: 4 additions & 1 deletion examples/local/202_move_tables.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,7 @@

source ../common/env.sh

vtctlclient MoveTables -- --source commerce --tables 'customer,corder' Create customer.commerce2customer
vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer create --source-keyspace commerce --tables "customer,corder"

# Wait for the workflow to reach the running state.
wait_for_workflow_running customer commerce2customer
2 changes: 1 addition & 1 deletion examples/local/203_switch_reads.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@

source ../common/env.sh

vtctlclient MoveTables -- --tablet_types=rdonly,replica SwitchTraffic customer.commerce2customer
vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer switchtraffic --tablet-types "rdonly,replica"
2 changes: 1 addition & 1 deletion examples/local/204_switch_writes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@

source ../common/env.sh

vtctlclient MoveTables -- --tablet_types=primary SwitchTraffic customer.commerce2customer
vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer switchtraffic --tablet-types primary
3 changes: 1 addition & 2 deletions examples/local/205_clean_commerce.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,4 @@

source ../common/env.sh

vtctlclient MoveTables Complete customer.commerce2customer

vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer complete
2 changes: 1 addition & 1 deletion go/cmd/internal/docgen/docgen.go
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ func restructure(rootDir string, dir string, name string, commands []*cobra.Comm
}

func newParentLinkSedCommand(parent string, file string) *exec.Cmd {
return exec.Command("sed", "-i ''", "-e", fmt.Sprintf("s:(./%s/):(../):", parent), file)
return exec.Command("sed", "-i", "", "-e", fmt.Sprintf("s:(./%s/):(../):i", parent), file)
}

func recursivelyDisableAutoGenTags(root *cobra.Command) {
Expand Down
22 changes: 22 additions & 0 deletions go/cmd/vtctldclient/cli/json.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,25 @@ func MarshalJSON(obj any) ([]byte, error) {
return data, nil
}
}

// MarshalJSONCompact works the same as MarshalJSON but elides zero value elements.
func MarshalJSONCompact(obj any) ([]byte, error) {
switch obj := obj.(type) {
case proto.Message:
m := protojson.MarshalOptions{
Multiline: true,
Indent: " ",
UseEnumNumbers: true,
UseProtoNames: true,
EmitUnpopulated: false, // elide zero value elements
}
return m.Marshal(obj)
default:
data, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return nil, fmt.Errorf("json.Marshal = %v", err)
}

return data, nil
}
}
Loading