Fixes balancing of metadata table #5195
Conversation
|
Have not tested these changes. Made similar changes in 4.0 and manually tested them. We probably need an IT that test balancing the metadata table if there is not one. In 4.0 this issue was preventing the metadata table from balancing across multiple tablet servers. |
| @@ -1000,20 +1000,22 @@ private SortedMap<TServerInstance,TabletServerStatus> createTServerStatusView( | |||
| final Map<String,TableInfo> newTableMap = | |||
| new HashMap<>(dl == DataLevel.USER ? oldTableMap.size() : 1); | |||
| if (dl == DataLevel.ROOT) { | |||
| if (oldTableMap.containsKey(RootTable.NAME)) { | |||
| newTableMap.put(RootTable.NAME, oldTableMap.get(RootTable.NAME)); | |||
| if (oldTableMap.containsKey(RootTable.ID.canonical())) { | |||
There was a problem hiding this comment.
Why not make it a Map<TableId,TableInfo> instead of Map<String,TableInfo>?
There was a problem hiding this comment.
Why not make it a Map<TableId,TableInfo> instead of Map<String,TableInfo>?
The code is currently manipulating a Thrift object which can not use the TableId type. A larger refactor would be needed to make that change, could open in issue to look into that.
|
Working on adding an IT for metadata table balancing. |
|
Wrote a test and found some additional issues and made fixes to get the test to pass. Converted this PR to draft because I want to go review the changes I made to get the test to pass. |
|
After writing the new IT I saw the following problems, with the first being the initial reason I opened this PR.
For 1 the changes in 6eab50f fix this. For 2 and 3 the changes in 6de5ef9 fix these. For 2 created a new Migrations class that tracks migrations per data level so that when making balancing decisions it could always know if the previous levels had migrations. For 3 added a new method to the balancer SPI that conveys what tables need to be balanced. This new method support partitioning tables for balancing in different ways and allows balancers to delegate to other balancers with smaller partitions. May also allow different manager processes to parition tables for balancing in the future. Took this out of draft after the changes in 6de5ef9 The new IT added only test problem 1. For problems 2 and 3, I verified those by running the IT and looking at the manager logs. The test exercises the cause of problems 2 and 3 but will not fail if they occur. |
keith-turner
changed the title
No description provided.