`engine.Reload()`: read InnoDB tables sizes including `FULLTEXT` index volume #17118

shlomi-noach · 2024-10-30T11:04:10Z

Description

Followup to #17066, and see #17119.

The changes in #17066 solve a performance issue. We note, however, that the query TablesWithSize80 does not include the file size / allocated size for InnoDB FULLTEXT indexes.

InnoDB's FULLTEXT index isn't a BTREE, and is implemented using hidden InnoDB tables. Those are visible in information_schema.innodb_tablespaces. However, in that table they are not named after the original table. Instead, they look something like mydb/fts_000000000000075e_00000000000005f9_index_2.

The key to unlocking these table identities is in information_schema.innodb_tables, which introduces this table as a third table in the TablesWithSize80, and this introduces two problems:

Performance: with large table counts (hundreds, 1000+) the query performs poorly.
Correctness: innodb_table names are encoded. The matching my-db/my-table entry in innodb_tablespaces will look like my@002ddb/my@002dtable in innodb_tables. This makes it impossible to join between the three tables. (Note: MySQL has a filename charset that can translate the two. But it was made internal only and inaccessible to queries).

In between the two problems, this PR takes a completely different approach on MySQL 8.0:

We introduce a new query which reads InnoDB tables with their sizes.
This query joins innodb_tables and innodb_tablespaces using valid conditions (which are not the table name).
The result is a list of InnoDB table sizes, for both external and hidden tables (in fact, the query will return two rows for a table that has FULLTEXT keys)
The list uses the encoded InnoDB names
On MySQL 8.0, schema.Engine reads table data without sizes, reads the InnoDB table sizes as explained, then programmatically joins the two.
To join the two we have reimplemented the MySQL filename encoding function.
The programmatic join makes sure to correctly handle partitioned tables
The programmatic join makes sure to cumulate table sizes for non-FTS plus FTS on same table

On non MySQL 8.0 the behavior remains unchanged.

This PR should pass all existing tests. We also add a unit test designed for schema.Engine.Reload() that validates the operation of the new logic.

Related Issue(s)

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…w completely split the table identities from table sizes. Introducing baseShowInnodbTableSizes() Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

…we're reading innodb table sizes; particular validation for nonzero filesize for partitioned table proves the logic is sound Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

vitess-bot · 2024-10-30T11:04:14Z

shlomi-noach · 2024-10-30T12:06:50Z

Working on the testing failures. We have pretty strict testing!

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2024-10-30T16:13:28Z

Good to review! All unit tests have been adapted (note: I'm impressed by just how much nitty gritty coverage they do for engine.Reload).

codecov · 2024-10-30T16:54:44Z

Codecov Report

Attention: Patch coverage is 75.22124% with 28 lines in your changes missing coverage. Please review.

Project coverage is 67.44%. Comparing base (4550640) to head (25949d4).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vttablet/tabletserver/schema/engine.go	76.00%	12 Missing ⚠️
go/mysql/collations/charset/filename.go	73.33%	8 Missing ⚠️
go/mysql/flavor_mysqlgr.go	0.00%	3 Missing ⚠️
go/mysql/flavor_filepos.go	0.00%	2 Missing ⚠️
go/mysql/flavor_mariadb_binlog_playback.go	0.00%	2 Missing ⚠️
go/mysql/flavor_mysql.go	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17118      +/-   ##
==========================================
+ Coverage   67.15%   67.44%   +0.28%     
==========================================
  Files        1571     1570       -1     
  Lines      252250   252285      +35     
==========================================
+ Hits       169409   170148     +739     
+ Misses      82841    82137     -704

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dbussink

Nice fix to ensure we have the right information here.

arthurschreiber · 2024-11-04T15:01:26Z

Nice! This is looking great.

mattlord

Nice work on this! Just had some minor nits and questions.

One larger thing that wasn't clear to me though... what role does TablesWithSize80 now play? Could/should we simplify that further so that it's not concerned with sizes at all?

mattlord · 2024-11-04T16:39:03Z

go/mysql/flavor_filepos.go

@@ -366,6 +366,10 @@ func (*filePosFlavor) baseShowTablesWithSizes() string {
 	return TablesWithSize56
 }

+func (filePosFlavor) baseShowInnodbTableSizes() string {
+	return ""


Shouldn't this be:

mysqlFlavor{}.baseShowInnodbTableSizes()

? I think so...

Just making sure I understand correctly, did you mean that filePosFlavor. baseShowInnodbTableSizes () should return the new query proposed in this PR, as opposed to an empty string? Why, then does filePosFlavor.baseShowTablesWithSizes() return a TablesWithSize56 result as opposed to TablesWithSize80?

In accordance with the rest of filePosFlavor behavior, I don't think it should be using 8.0-grade queries?

I'm not sure. I don't think file/position is really tied to a version like that. We can leave it like it is.

mattlord · 2024-11-04T16:41:48Z

go/mysql/flavor_mariadb_binlog_playback.go

@@ -22,6 +22,10 @@ func (mariadbFlavor) baseShowTables() string {
 	return mysqlFlavor{}.baseShowTables()
 }

+func (mariadbFlavor) baseShowInnodbTableSizes() string {
+	return ""


Same for this one, I think:

mysqlFlavor{}.baseShowInnodbTableSizes()

go/mysql/flavor_mysql.go

mattlord · 2024-11-04T16:55:37Z

go/mysql/flavor_mysqlgr.go

mysqlgr (go/vt/vtgr) is gone (#13308), but looks like we missed removing the flavor. Are you OK removing it here? We can also do that separately.

Let's please do that separately!

go/vt/vttablet/endtoend/misc_test.go