[WIP] Store should not update template on every restart.

[njd5475]

Summary

This may be an alternative to trying to track version numbers for templates. Instead of trying to store, read, compare and track a version number. We can get the template properties, compare them to the codes properties with order agnostic hashes.

If anyone changes the templateProperties the template would be updated. As well as when it does not exist.

This would close #28540

Relevant details from that PR

This PR implements changes per feedback from the original Task Manager PR, that slipped through the cracks during review #24356 (comment):

@pickypg on Dec 7, 2018 Member
...
But also we should not be setting this unless we need to set it. If it already exists, then we're unnecessary causing a cluster state change on the ES side every time that Kibana starts up.

As an aside, the ES Template API supports version values and we should be using it based on the version of Kibana so that we do not replace a newer version of Kibana's template with our own. And, given a newer template, we should consider not marking the task manager as initialized.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• This was checked for cross-browser compatibility, including a check against IE11
• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately
• This includes a feature addition or change that requires a release note and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-platform

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[pickypg]

This does prevent overwriting it upon every restart, but it also means that if an older version happens to be restarted, then it will trample a newer version's template.

[njd5475]

I don't see this as an issue. Because index templates don't change existing indices. If you delete the index, it will then get created with the new mappings from the older templated version but before task manager would run and index a task it would update the template.

We could double check the mappings once per run before indexing or handle indexing error appropriately to determine the index was changed and doesn't match.

[tsullivan]

I think @pickypg is right in that updates need to be controlled in that they must either install the template from scratch, upgrade it, or do nothing. If they can't upgrade it, they'll potentially break things in ways that will be hard to troubleshoot for the user: the outcome of the problem will be some kind of TypeError when an old instance of Kibana claims a new task and fields it depends on have moved.

Consider:

1. Kibana 0.1.0 starts up, installs the template from scratch
2. Kibana 1.0.0 starts up, overwrites the template
3. Kibana 1.0.0 schedules a task
4. Kibana 0.1.0 claims the task
5. Boom.

Edit: to add more thoughts, the original PR we're working on handles this in two ways:

1. Numeric versioning ensures old templates do not overwrite new templates, provides helpful log messaging to get users to upgrade
2. API versioning ensures older instances are only able to claim tasks they understand