Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VDiff: Migrate client command to vtctldclient #13976

Merged
merged 39 commits into from
Sep 23, 2023
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
152d842
Begin migrating VDiff command to vtctldclient
mattlord Sep 13, 2023
255f4e1
Create is now working
mattlord Sep 14, 2023
b031afb
vdiff show
mattlord Sep 18, 2023
fe1f296
Merge remote-tracking branch 'origin/main' into vtctldclient_vdiff
mattlord Sep 18, 2023
f1e4f26
Finish vdiff show
mattlord Sep 18, 2023
28850ac
vdiff delete
mattlord Sep 18, 2023
a59c854
vdiff create wait and vdiff resume
mattlord Sep 18, 2023
541e1fd
Fixup resume handling
mattlord Sep 18, 2023
c530050
vdiff stop
mattlord Sep 18, 2023
786d0df
Update vtctldclient help output
mattlord Sep 18, 2023
ebc117d
Minor changes
mattlord Sep 19, 2023
08413c7
Improve JSON output
mattlord Sep 19, 2023
6a39b1b
Various simplifcations
mattlord Sep 19, 2023
beebc96
Minor changes after self review
mattlord Sep 19, 2023
0e9a171
Minor changes pt2
mattlord Sep 19, 2023
3e77a9c
Address review comments
mattlord Sep 19, 2023
e7a0be8
Address more comments
mattlord Sep 19, 2023
c01b1cb
More minor changes
mattlord Sep 19, 2023
9f45ef2
Revert "More minor changes"
mattlord Sep 19, 2023
dd37e1a
Use std protobuf package name (vtrpcb)
mattlord Sep 19, 2023
22c510e
Moar tweaks
mattlord Sep 20, 2023
e17ef54
Fixes from local testing
mattlord Sep 20, 2023
2347de6
Use vtctldclient impl in e2e tests
mattlord Sep 20, 2023
342e2b9
Rebuild protos (comment only)
mattlord Sep 20, 2023
0810146
Migrate side-by-side from v1/v2 to vtctlclient/vtctldclient
mattlord Sep 20, 2023
3e38a57
Merge remote-tracking branch 'origin/main' into vtctldclient_vdiff
mattlord Sep 20, 2023
c5ea2c4
Correct errant casing for MoveTables
mattlord Sep 20, 2023
9b904cc
Migrate client side unit tests
mattlord Sep 20, 2023
7fc7725
Changes after self review
mattlord Sep 22, 2023
9700198
Merge remote-tracking branch 'origin/main' into vtctldclient_vdiff
mattlord Sep 22, 2023
5fc4671
Add deprecation note for vdiff v1
mattlord Sep 22, 2023
d57bd05
Try to address unrelated test flakes
mattlord Sep 22, 2023
41860c8
Try to deflake the across_db_versions workflow
mattlord Sep 22, 2023
3acb0c1
Use io writer in display functions
mattlord Sep 22, 2023
b292ebb
Don't use sudo with fuser so that we can only kill a proc we own
mattlord Sep 22, 2023
f3fb2e6
Try to address more flakiness
mattlord Sep 23, 2023
9145dad
Yer killing me flake
mattlord Sep 23, 2023
2fb0784
Minor simplification -- excuse to run tests again
mattlord Sep 23, 2023
a6974ac
More micro nits -- excuse to run tests again
mattlord Sep 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changelog/18.0/18.0.0/summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
- [Deleted `vtgr`](#deleted-vtgr)
- [Deleted `query_analyzer`](#deleted-query_analyzer)
- [Deprecated VTBackup stat `DurationByPhase`](#deprecated-vtbackup-stat-duration-by-phase)
- [Deprecated VDiff v1](#deprecated-vdiff-v1)
- **[New stats](#new-stats)**
- [VTGate Vindex unknown parameters](#vtgate-vindex-unknown-parameters)
- [VTBackup stat `Phase`](#vtbackup-stat-phase)
Expand Down Expand Up @@ -132,6 +133,10 @@ The `vtgr` has been deprecated in Vitess 17, also see https://github.com/vitessi

The undocumented `query_analyzer` binary has been removed in Vitess 18, see https://github.com/vitessio/vitess/issues/14054.

#### <a id="deprecated-vdiff-v1"/>Deprecated VDiff v1

[VDiff v2 was added in Vitess 15.0](https://vitess.io/blog/2022-11-22-vdiff-v2/) and marked as GA in 16.0. The [legacy v1 client command](https://vitess.io/docs/18.0/reference/vreplication/vdiffv1/) is now deprecated in Vitess 18.0 and will be **removed** in 19.0. Please switch all of your usage to the [new VDiff client](https://vitess.io/docs/18.0/reference/vreplication/vdiff/) command ASAP.

#### <a id="deprecated-vtbackup-stat-duration-by-phase"/>Deprecated VTbackup stat `DurationByPhase`

VTBackup stat `DurationByPhase` is deprecated. Use the binary-valued `Phase` stat instead.
Expand Down
20 changes: 13 additions & 7 deletions go/cmd/vtctldclient/cli/json.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ import (
"google.golang.org/protobuf/proto"
)

const (
mattlord marked this conversation as resolved.
Show resolved Hide resolved
jsonIndent = " "
jsonPrefix = ""
)

// MarshalJSON marshals obj to a JSON string. It uses the jsonpb marshaler for
// proto.Message types, with some sensible defaults, and falls back to the
// standard Go marshaler otherwise. In both cases, the marshaled JSON is
Expand All @@ -39,14 +44,14 @@ func MarshalJSON(obj any) ([]byte, error) {
case proto.Message:
m := protojson.MarshalOptions{
Multiline: true,
Indent: " ",
Indent: jsonIndent,
UseEnumNumbers: true,
UseProtoNames: true,
EmitUnpopulated: true,
}
return m.Marshal(obj)
default:
data, err := json.MarshalIndent(obj, "", " ")
data, err := json.MarshalIndent(obj, jsonPrefix, jsonIndent)
if err != nil {
return nil, fmt.Errorf("json.Marshal = %v", err)
}
Expand All @@ -55,20 +60,21 @@ func MarshalJSON(obj any) ([]byte, error) {
}
}

// MarshalJSONCompact works the same as MarshalJSON but elides zero value elements.
func MarshalJSONCompact(obj any) ([]byte, error) {
// MarshalJSONPretty works the same as MarshalJSON but elides zero value
// elements and uses ENUM names instead of numbers.
func MarshalJSONPretty(obj any) ([]byte, error) {
switch obj := obj.(type) {
case proto.Message:
m := protojson.MarshalOptions{
Multiline: true,
Indent: " ",
UseEnumNumbers: true,
Indent: jsonIndent,
UseEnumNumbers: false,
UseProtoNames: true,
EmitUnpopulated: false, // elide zero value elements
}
return m.Marshal(obj)
default:
data, err := json.MarshalIndent(obj, "", " ")
data, err := json.MarshalIndent(obj, jsonPrefix, jsonIndent)
if err != nil {
return nil, fmt.Errorf("json.Marshal = %v", err)
}
Expand Down
1 change: 1 addition & 0 deletions go/cmd/vtctldclient/command/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import (
vreplcommon "vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/common"
_ "vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/movetables"
_ "vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/reshard"
_ "vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/vdiff"
_ "vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/workflow"
"vitess.io/vitess/go/trace"
"vitess.io/vitess/go/vt/logutil"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func commandCancel(cmd *cobra.Command, args []string) error {
sort.Slice(resp.Details, func(i, j int) bool {
return resp.Details[i].Tablet.String() < resp.Details[j].Tablet.String()
})
output, err = cli.MarshalJSONCompact(resp)
output, err = cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ func commandComplete(cmd *cobra.Command, args []string) error {

var output []byte
if format == "json" {
output, err = cli.MarshalJSONCompact(resp)
output, err = cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand Down
2 changes: 1 addition & 1 deletion go/cmd/vtctldclient/command/vreplication/common/show.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func commandShow(cmd *cobra.Command, args []string) error {
return err
}

data, err := cli.MarshalJSON(resp)
data, err := cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ func commandSwitchTraffic(cmd *cobra.Command, args []string) error {

var output []byte
if format == "json" {
output, err = cli.MarshalJSONCompact(resp)
output, err = cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ func commandUpdateState(cmd *cobra.Command, args []string) error {
return resp.Details[i].Tablet.String() < resp.Details[j].Tablet.String()
})

data, err := cli.MarshalJSON(resp)
data, err := cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand Down
14 changes: 7 additions & 7 deletions go/cmd/vtctldclient/command/vreplication/common/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func RegisterCommands(root *cobra.Command) {

type SubCommandsOpts struct {
SubCommand string
Workflow string // used to specify an example workflow name for the Examples section of the help output.
Workflow string // Used to specify an example workflow name for the Examples section of the help output.
}

func SetClient(c vtctldclient.VtctldClient) {
Expand Down Expand Up @@ -150,7 +150,7 @@ func OutputStatusResponse(resp *vtctldatapb.WorkflowStatusResponse, format strin
var output []byte
var err error
if format == "json" {
output, err = cli.MarshalJSON(resp)
output, err = cli.MarshalJSONPretty(resp)
if err != nil {
return err
}
Expand All @@ -172,11 +172,11 @@ func OutputStatusResponse(resp *vtctldatapb.WorkflowStatusResponse, format strin
}

func AddCommonFlags(cmd *cobra.Command) {
cmd.Flags().StringVar(&BaseOptions.TargetKeyspace, "target-keyspace", "", "Target keyspace for this workflow exists (required).")
cmd.MarkFlagRequired("target-keyspace")
cmd.Flags().StringVarP(&BaseOptions.Workflow, "workflow", "w", "", "The workflow you want to perform the command on (required).")
cmd.MarkFlagRequired("workflow")
cmd.Flags().StringVar(&BaseOptions.Format, "format", "text", "The format of the output; supported formats are: text,json.")
cmd.PersistentFlags().StringVar(&BaseOptions.TargetKeyspace, "target-keyspace", "", "Target keyspace for this workflow.")
cmd.MarkPersistentFlagRequired("target-keyspace")
cmd.PersistentFlags().StringVarP(&BaseOptions.Workflow, "workflow", "w", "", "The workflow you want to perform the command on.")
cmd.MarkPersistentFlagRequired("workflow")
cmd.PersistentFlags().StringVar(&BaseOptions.Format, "format", "text", "The format of the output; supported formats are: text,json.")
}

func AddCommonCreateFlags(cmd *cobra.Command) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ var (
// moveTablesCreate makes a moveTablesCreate gRPC call to a vtctld.
moveTablesCreate = &cobra.Command{
Use: "create",
Short: "Create and optionally run a moveTables VReplication workflow.",
Short: "Create and optionally run a MoveTables VReplication workflow.",
Example: `vtctldclient --server localhost:15999 movetables --workflow commerce2customer --target-keyspace customer create --source-keyspace commerce --cells zone1 --cells zone2 --tablet-types replica`,
SilenceUsage: true,
DisableFlagsInUseLine: true,
Expand Down Expand Up @@ -123,7 +123,7 @@ func commandMoveTablesCreate(cmd *cobra.Command, args []string) error {

func registerCreateCommand(root *cobra.Command) {
common.AddCommonCreateFlags(moveTablesCreate)
moveTablesCreate.PersistentFlags().StringVar(&moveTablesCreateOptions.SourceKeyspace, "source-keyspace", "", "Keyspace where the tables are being moved from (required).")
moveTablesCreate.PersistentFlags().StringVar(&moveTablesCreateOptions.SourceKeyspace, "source-keyspace", "", "Keyspace where the tables are being moved from.")
moveTablesCreate.MarkPersistentFlagRequired("source-keyspace")
moveTablesCreate.Flags().StringSliceVar(&moveTablesCreateOptions.SourceShards, "source-shards", nil, "Source shards to copy data from when performing a partial moveTables (experimental).")
moveTablesCreate.Flags().StringVar(&moveTablesCreateOptions.SourceTimeZone, "source-time-zone", "", "Specifying this causes any DATETIME fields to be converted from the given time zone into UTC.")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ var (
moveTables = &cobra.Command{
Use: "MoveTables --workflow <workflow> --keyspace <keyspace> [command] [command-flags]",
Short: "Perform commands related to moving tables from a source keyspace to a target keyspace.",
Long: `moveTables commands: Create, Show, Status, SwitchTraffic, ReverseTraffic, Stop, Start, Cancel, and Delete.
Long: `MoveTables commands: Create, Show, Status, SwitchTraffic, ReverseTraffic, Stop, Start, Cancel, and Delete.
See the --help output for each command for more details.`,
DisableFlagsInUseLine: true,
Aliases: []string{"movetables"},
Expand Down
Loading
Loading