Add identifier to split points for automated vs user created split points

[ddanielr]

Is your feature request related to a problem? Please describe.
When split points are automatically created due to tablet sizes, these split points persist until a user runs a merge.
If a merge is run automatically, then it might remove split points that were purposely set by the user.

Describe the solution you'd like
Split points created automatically by accumulo should be identified as such so they can easily be removed.

[cshannon]

I will take a look at this issue, I think it makes sense to add a new column to track the split point type to the tablet metadata

[cshannon]

@keith-turner - I've started to look into this and prototype something with a new column qual that will be added to the split family that was added and a few questions come to mind on how we want to handle marking the tablets:

1. Should we always require a split type to be set or is absent valid?
2. This relates to my first question and I'm wondering if it makes sense to always store the split type (either user or system) or if it makes more sense to just mark a tablet if it's user generated and for system splits there would just be an absent marker. This could depend on if we ever plan to have more than 2 types at some point.
3. For new tables I was thinking we'd either just mark them as system or absent depending on if absent is valid. For merges I am guessing we'd do the same? after the merge mark it as system generated or unset it etc.

[keith-turner]

Should we always require a split type to be set or is absent valid?

The pro for absence==system split is less data in the metadata table for a common case. The con is absence can mean two things, its kinda like using null in java. It can mean there is a bug and it was not set when it was intended to be set or it can mean its a system split. So maybe we should only go w/ absence==system split if we feel confident enough in our tests to ensure that user split marker is always being set when intended.

For new tables I was thinking we'd either just mark them as system or absent depending on if absent is valid.

The create table API has a mechanism to set initial splits on a table. Those could be marked as user initially. The default tablet at the end of every table seems like it would always be marked system, but not sure. The default tablet is always there when you have zero or more splits.

For merges I am guessing we'd do the same?

I can not remember what merge does w.r.t. adding splits. Reviewing all of your merge changes, I remember something about deleterows inserting a split point possibly but not sure. But, maybe this was removed in the 4.0 changes? If merge is still adding splits, it was just doing that to make things work so not sure those should be marked as user.

[keith-turner]

Would the following be a good way to handle upgrade?

1. Upgrade marks all splits as system
2. There is per table prop that will enable automatic merges based on size. The default value for this property is disables automatic merging.
3. The add split API call is modified such that if a user calls it passes in an existing split that is a system split it will remark it as a user split.

Then after upgrade user have a chance to mark splits that should never be automatically removed before enabling the auto merge feature.

[cshannon]

Would the following be a good way to handle upgrade?

1. Upgrade marks all splits as system
2. There is per table prop that will enable automatic merges based on size. The default value for this property is disables automatic merging.
3. The add split API call is modified such that if a user calls it passes in an existing split that is a system split it will remark it as a user split.

Then after upgrade user have a chance to mark splits that should never be automatically removed before enabling the auto merge feature.

I was wondering about the upgrade case, i like the idea of using absent for system splits as most splits will be system splits. However, one downside is any existing splits that a user created would be incorrectly marked as a system split on upgrade unless there were manual steps done to mark them as user during upgrade like mentioned above.

I think this is fine but I'm wondering how we make it easy for a user to mark splits. We might need a new command line tool or something to make it simple to mark them. In fact, I could use a use case where a user decides maybe there's some tablets they don't want merged (even if originally system created) and they could use the tool not just for upgrades but for marking those to prevent merging too.

[keith-turner]

I think this is fine but I'm wondering how we make it easy for a user to mark splits. We might need a new command line tool or something to make it simple to mark them. In fact, I could use a use case where a user decides maybe there's some tablets they don't want merged (even if originally system created) and they could use the tool not just for upgrades but for marking those to prevent merging too.

Also wondering about unmarking them. The TabletInformation API couild be used to get information about the markings. Then just need to figure out what the APIs and shell commands are for marking and unmarking them. Could be similar to APIs and shell commands for makring tablet availability.

This could depend on if we ever plan to have more than 2 types at some point.

Maybe instead of marking how a split was created we mark a splits mergability. That could have the following three states.

1. ALWAYS : User or system initiated merge can merge this split away
2. USER : Only a user initiated merge can merge this split away
3. NEVER : Nothing can merge this split away and if a user merge operations is initiated on it then the merge operation would fail

[cshannon]

I talked to @keith-turner offline and we think it would be good to have a broader discussion with others first before deciding the best way forward for this. The main use case for this feature in the first place was to handle automatic merges. After talking to Keith about it being able to mark mergeability separately and not have it tied to how splits are created could have some advantages (but increases complexity). There's more potential use cases and features we came up with in regards to merging such as being able to mark a split as a candidate for auto merge or not regardless how the split was created, marking a tablet as never being able to merge, change the marking, etc and it would be good to see if anyone else has any opinions on it.

[cshannon]

Several of us talked and came up with a path forward for now, the tentative plan to try is:

1. There will be a new property created that is a percentage or ratio of the split threshold that can be used for automatic merges. The intent is that tablets can be merged together once they are below the threshold (ie maybe 10% or 25% of the split threshold). This prevents automatic merging from creating a tablet too big that will just immediately split.
2. A new column in metadata for the tablet that will store mergeability state of a table (if a tablet is eligible to be automatically merged). There will be 3 possible states and these will be represented by a duration that is relative to manager steady time. The exact names are still to be determined but we could represent it with something like NEVER (-1), NOW (0), FUTURE (timestamp). Future would mean that the tablet is eligible after some period of time for merging relative to the steady time.
3. There will need to be an updated API to set the mergeability information. A new method in the API called putSplits() can be created that will either create or update existing splits to set the mergeability. The existing addSplits() method can be deprecated and just default to never. The create table API can also be created with splits and this will also be able to set it.
4. There would need to be some way for a user to read the mergeability state of the tablet so needs to be returned as part of a new API call or modify an existing API.
5. By default, all user created tablets could be marked with a state of NEVER and system spits would be marked as NOW (or whatever we call it) so they can automatically merge. Pre-created splits for system tables like Fate, and Metadata, etc would be marked as NEVER by default.
6. On upgrade we would mark all existing tablets as NEVER.

One thing I am still trying to figure out is how the automatic merging should work if we reach the threshold we set (say smaller than 10% of the split threshold) and if we should only ever merge together 2 tablets or try and merge as many as possible etc.

@keith-turner - Is there anything I missed or got wrong after the conversation and conclusions we came to?

[keith-turner]

One thing I am still trying to figure out is how the automatic merging should work if we reach the threshold we set (say smaller than 10% of the split threshold) and if we should only ever merge together 2 tablets or try and merge as many as possible etc.

I like merging as many tablets that fit into the merge threshold. So if we have five tablets and the sum of their existing sizes is less than the merge threshold then can merge all five of them.

[cshannon]

My plan is to break this into multiple PRs, I'm thinking 3 parts for now:

1. The first PR will be to add a new Mergeability column to the metadata table and to have Accumulo start populating it based on system splits vs user created splits. Pre split tables like Fate will be marked to not merge. Some basic inital tests will be added.
2. The next PR will handle the API changes which will allow setting the Mergeability of a table/tablet at table creation time or when creating/updating splits later. There will also be API updates to allow viewing the values. These changes will make it easier to add a lot more tests.
3. The last PR will implement the automatic merge feature to handle finding tablets that need merging and handling that. I'm envisioning this will be done with FATE and set up very similar to how splits work now. There will be a new property as well for the merge threshold, like splits. This last PR could be broken done even further into more than one PR if the scope becomes too large.