Add ML Jobs to Sample Data Sets

[alexfrancoeur]

We plan on introducing a sample data set to Kibana soon. We'll be starting with flight data, but as time progresses I'm sure we'll add more. I'd like to add ML jobs to these sample data sets at some point as well to show off the functionality available.

For the flight data set specifically, I am generating the data from a script so we can easily add anomalies for specific fields.

For dashboards, visualizations and saved searches we are currently using the import/export API that beats modules use in order to ingest the new dashboards. I don't believe ML jobs are stored the same way, so it's possible we may need another mechanism.

Phase 1 is to merge this PR / issue: #17807 / #16473. Once this is complete, we can begin to add enhancements such as this.

cc: @grabowskit @sophiec20 @nreese @asawariS

[elasticmachine]

Pinging @elastic/ml-ui

[lcawl]

@alexfrancoeur Would there be difficulties associated with licensing? i.e. The sample data is available with the basic license but you must have at least a trial license to create machine learning jobs.

[nreese]

What are machine learning jobs? Are they Saved objects? There is a mechanism to add save objects to sample data sets. This is used by Canvas to add workpads when canvas is enabled. Could ML add jobs in the same fashion?

[peteharverson]

@nreese ML jobs are not saved objects, but are currently stored in the clusterstate. In 6.6 this is being changed, moving them out of clusterstate metadata and instead the jobs will be stored as documents in an index (.ml-config). REST endpoints exist for adding jobs and their associated datafeeds, and then opening the jobs and starting the datafeeds (see x-pack/plugins/ml/server/client/elasticsearch_ml.js.

For the ML modules, we ship pre-configured ML jobs as JSON files and then use the endpoints to add and start the jobs, and also insert Kibana saved objects for drilldown dashboards using the saved object client. The JSON config files used for this process are kept under x-pack/plugins/ml/server/models/data_recognizer/modules.

[nreese]

@peteharverson Thanks for the clarification. What if the saved object registry exposed a way to attach a handler that would get called when a sample data set is installed and another handler that is called when a sample data set is uninstalled? ML could register functions to add/set up jobs on install and then register a function to clean up jobs on uninstall.

[peteharverson]

@nreese the approach you suggest where the saved object registry exposes a way to attach handlers sounds viable. The sample data set would need to store a ML module ID which can be passed to the ML endpoint when it is installed / uninstalled.

[peteharverson]

This was addressed by the addition of ML modules for the ecommerce and weblogs sample data sets in 7.2 in #35138. Closing issue.