Fixing Mysqld.normalizedSchema() by formally parsing the query

[shlomi-noach]

Description

This fixes #13865. The existing query editing mechanism is a naive string search & replace, which can lead to incorrect replacement of table names. This PR will work to formally parse the query and only replace DB names.

This will be done in stages. First, we introduce unit testing to support the existing behavior. Then, introduce unit tests that break with #13865. Last, we will introduce formal parsing that passes all tests.

Related Issue(s)

Fixes #13865

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[dbussink]

There's a few other places where we use string replacements, but where really parsing should be used:

vitess/go/vt/mysqlctl/schema.go

Lines 90 to 102 in b65cde1

	func (mysqld *Mysqld) GetSchema(ctx context.Context, dbName string, request *tabletmanagerdatapb.GetSchemaRequest) (*tabletmanagerdatapb.SchemaDefinition, error) {
	sd := &tabletmanagerdatapb.SchemaDefinition{}
	backtickDBName := sqlescape.EscapeID(dbName)
	// get the database creation command
	qr, fetchErr := mysqld.FetchSuperQuery(ctx, fmt.Sprintf("SHOW CREATE DATABASE IF NOT EXISTS %s", backtickDBName))
	if fetchErr != nil {
	return nil, fetchErr
	}
	if len(qr.Rows) == 0 {
	return nil, fmt.Errorf("empty create database statement for %v", dbName)
	}
	sd.DatabaseSchema = strings.Replace(qr.Rows[0][1].ToString(), backtickDBName, "{{.DatabaseName}}", 1)

vitess/go/vt/mysqlctl/schema.go

Lines 438 to 445 in b65cde1

	for _, td := range originalSchema.TableDefinitions {
	if td.Type == tmutils.TableView {
	// Views will have {{.DatabaseName}} in there, replace
	// it with _vt_preflight
	s := strings.Replace(td.Schema, "{{.DatabaseName}}", "`_vt_preflight`", -1)
	initialCopySQL += s + ";\n"
	}
	}

vitess/go/vt/mysqlctl/tmutils/schema.go

Lines 213 to 234 in b65cde1

	// Backtick database name since keyspace names appear in the routing rules, and they might need to be escaped.
	// We unescape() them first in case we have an explicitly escaped string was specified.
	createDatabaseSQL := strings.Replace(sd.DatabaseSchema, "`{{.DatabaseName}}`", "{{.DatabaseName}}", -1)
	createDatabaseSQL = strings.Replace(createDatabaseSQL, "{{.DatabaseName}}", sqlescape.EscapeID("{{.DatabaseName}}"), -1)
	sqlStrings = append(sqlStrings, createDatabaseSQL)
	for _, td := range sd.TableDefinitions {
	if schema.IsInternalOperationTableName(td.Name) {
	continue
	}
	if td.Type == TableView {
	createViewSQL = append(createViewSQL, td.Schema)
	} else {
	lines := strings.Split(td.Schema, "\n")
	for i, line := range lines {
	if strings.HasPrefix(line, "CREATE TABLE `") {
	lines[i] = strings.Replace(line, "CREATE TABLE `", "CREATE TABLE `{{.DatabaseName}}`.`", 1)
	}
	}
	sqlStrings = append(sqlStrings, strings.Join(lines, "\n"))
	}
	}

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

	// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
	// SQL statements, but doesn't return any results, so it's only useful for SQL statements
	// that would be run for their effects (e.g., CREATE).
	// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
	// Thus it should be used only for changes that can be applied on a live instance without causing issues;
	// it shouldn't be used for anything that will require a pivot.
	// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
	func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
	filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
	if err != nil {
	return fmt.Errorf("fillStringTemplate failed: %v", err)
	}
	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
	defer cancel()
	// Need to make sure that replication is enabled since we're only applying the statement on primaries
	_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
	SQL: filledChange,
	Force: false,
	AllowReplication: true,
	SQLMode: vreplication.SQLMode,
	})
	return err
	}
	// fillStringTemplate returns the string template filled
	func fillStringTemplate(tmpl string, vars any) (string, error) {
	myTemplate := template.Must(template.New("").Parse(tmpl))
	data := new(bytes.Buffer)
	if err := myTemplate.Execute(data, vars); err != nil {
	return "", err
	}
	return data.String(), nil
	}

[shlomi-noach]

@dbussink I don't think

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

	// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
	// SQL statements, but doesn't return any results, so it's only useful for SQL statements
	// that would be run for their effects (e.g., CREATE).
	// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
	// Thus it should be used only for changes that can be applied on a live instance without causing issues;
	// it shouldn't be used for anything that will require a pivot.
	// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
	func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
	filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
	if err != nil {
	return fmt.Errorf("fillStringTemplate failed: %v", err)
	}
	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
	defer cancel()
	// Need to make sure that replication is enabled since we're only applying the statement on primaries
	_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
	SQL: filledChange,
	Force: false,
	AllowReplication: true,
	SQLMode: vreplication.SQLMode,
	})
	return err
	}
	// fillStringTemplate returns the string template filled
	func fillStringTemplate(tmpl string, vars any) (string, error) {
	myTemplate := template.Must(template.New("").Parse(tmpl))
	data := new(bytes.Buffer)
	if err := myTemplate.Execute(data, vars); err != nil {
	return "", err
	}
	return data.String(), nil
	}

is related; looks like an altogether different mechanism (using text/template, haven't seen this elsewhere in vitess code), and not part of {{.DatabaseName}} thing.

[shlomi-noach]

@dbussink After an hour of debugging a test failure, I came full circle to recognize that

vitess/go/vt/wrangler/schema.go

Lines 283 to 315 in b65cde1

	// applySQLShard applies a given SQL change on a given tablet alias. It allows executing arbitrary
	// SQL statements, but doesn't return any results, so it's only useful for SQL statements
	// that would be run for their effects (e.g., CREATE).
	// It works by applying the SQL statement on the shard's primary tablet with replication turned on.
	// Thus it should be used only for changes that can be applied on a live instance without causing issues;
	// it shouldn't be used for anything that will require a pivot.
	// The SQL statement string is expected to have {{.DatabaseName}} in place of the actual db name.
	func (wr *Wrangler) applySQLShard(ctx context.Context, tabletInfo *topo.TabletInfo, change string) error {
	filledChange, err := fillStringTemplate(change, map[string]string{"DatabaseName": tabletInfo.DbName()})
	if err != nil {
	return fmt.Errorf("fillStringTemplate failed: %v", err)
	}
	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
	defer cancel()
	// Need to make sure that replication is enabled since we're only applying the statement on primaries
	_, err = wr.tmc.ApplySchema(ctx, tabletInfo.Tablet, &tmutils.SchemaChange{
	SQL: filledChange,
	Force: false,
	AllowReplication: true,
	SQLMode: vreplication.SQLMode,
	})
	return err
	}
	// fillStringTemplate returns the string template filled
	func fillStringTemplate(tmpl string, vars any) (string, error) {
	myTemplate := template.Must(template.New("").Parse(tmpl))
	data := new(bytes.Buffer)
	if err := myTemplate.Execute(data, vars); err != nil {
	return "", err
	}
	return data.String(), nil
	}

is very much related 🤕

[dbussink]

@dbussink After an hour of debugging a test failure, I came full circle to recognize that

Ah yes, the syntax of {{.DatabaseName}} comes from it being used as a Go template variable in the first place.

[shlomi-noach]

OK, good to go! Looking for reviews. Formatting is back to String() and all tests adapted.

[deepthi]

LGTM except for the one question about a comment.

[deepthi]

Is this comment still valid? Since you reverted the call to CanonicalString?

[shlomi-noach]

Fixed

[mattlord]

LGTM! I only had a minor comment and will let you resolve that as you feel best.

[mattlord]

This comment doesn't match the behavior, does it? We only do that replacement for views AFAICT. Which one is correct, the comment or the code? I think it's the comment as the code was just moved.

[shlomi-noach]

Fixed

[mattlord]

Any reason not to consolidate these?

[shlomi-noach]

Yes - golang will give compilation errors, because SQLNode (the shared interface for all these types) does not have a DBName field (obviously as it is an interface, and each implementing type adds the fields independently).

[mattlord]

This is escaped but not qualified.

[shlomi-noach]

Not sure I follow?

[shlomi-noach]

Temporarily converted to draft because I want to confirm something before merging, and I don't want anyone else to accidentally merge in the interim. Will likely merge this shortly.

[github-actions]

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

• Push additional commits to the associated branch.
• Remove the stale label.
• Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

[github-actions]

This PR was closed because it has been stale for 7 days with no activity.

[github-actions]

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

• Push additional commits to the associated branch.
• Remove the stale label.
• Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

[github-actions]

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

• Push additional commits to the associated branch.
• Remove the stale label.
• Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

[github-actions]

This PR was closed because it has been stale for 7 days with no activity.