Module configuration variants

[ycombinator]

Resolves #8969.

This PR introduces the concept of a module configuration variant, as detailed in #8969. This works with filebeat as well as metricbeat modules.

Usage examples

Listing modules and their variants

$ ./metricbeat modules list
Enabled:
system

Disabled:
...
docker
dropwizard
elasticsearch
elasticsearch:xpack
envoyproxy
etcd
...

$ ls -l  modules.d/elasticsearch*
-rw-r--r--  1 shaunak  staff  402 Nov 15 16:39 modules.d/elasticsearch.xpack.yml.disabled
-rw-r--r--  1 shaunak  staff  286 Nov 15 16:39 modules.d/elasticsearch.yml.disabled

Enabling a module variant configuration

$ ./metricbeat modules enable elasticsearch:xpack
Enabled elasticsearch:xpack

$ ls -l  modules.d/elasticsearch*
-rw-r--r--  1 shaunak  staff  402 Nov 15 16:39 modules.d/elasticsearch.xpack.yml
-rw-r--r--  1 shaunak  staff  286 Nov 15 16:39 modules.d/elasticsearch.yml.disabled

Source config files for a module

ls -l module/elasticsearch/_meta/config*
-rw-r--r--  1 shaunak  staff  377 Nov 15 11:17 module/elasticsearch/_meta/config.reference.yml
-rw-r--r--  1 shaunak  staff  276 Nov 15 11:34 module/elasticsearch/_meta/config.xpack.yml
-rw-r--r--  1 shaunak  staff  160 Oct 25 14:26 module/elasticsearch/_meta/config.yml

[ycombinator]

I still need to write tests, docs and a changelog entry for the functionality in this PR but just wanted to put it out there and get some early feedback.

[kvch]

Please port the Python script to Golang.

[ycombinator]

@jsoriano and @kvch Thanks for your LGTMs. I've updated this PR with unit tests and CHANGELOG entries. Additionally, given that Filebeat now uses the same module config collection script as Metricbeat, the Filebeat module configuration files now each contain a header with a link to the module's documentation. Given these changes, you might want to re-review this PR.

@dedemorton I could use your guidance on where best to document the functionality introduced by this PR. Somewhere in https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-modules.html seems appropriate to me but I'll defer to your opinion.

Everyone: Let's now have the naming discussion 😄. Are we okay calling these alternative out-of-the-box module configurations as "variants"? Or do folks like "flavors" better as used by @jsoriano and @ruflin in #8969? Or something else?

[ruflin]

Could we split up this PR in two parts? 1 that moves the python files + adds new headers to config files and a second one that does the actual changes with flavors etc? Like this the flavor / variants PR will be much smaller.

For the naming: There is also a discussion around flavor for datasets. Meaning it can have different inputs but the output data structure will be identical. Here I think flavor fits well. For the module I'm asking myself if it the same thing? The x-pack one is a bit special as it creates a completely different data structure. What is a flavor of a module in a more generic case? It is just the same module with different default config options?

[ycombinator]

Could we split up this PR in two parts? 1 that moves the python files + adds new headers to config files and a second one that does the actual changes with flavors etc? Like this the flavor / variants PR will be much smaller.

I've made a new PR for just the python file reuse + adding new headers: #9150. And I have rebased this PR (#9118) on top of the new one (#9150). Once the new one is reviewed and merged, I will rebase this PR on master so it becomes smaller.

[ycombinator]

#9150 has been merged now so I've rebased this PR on master. It now only contains the changes for implementing module configuration variants.

[dedemorton]

@ycombinator

I could use your guidance on where best to document the functionality introduced by this PR. Somewhere in https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-modules.html seems appropriate to me but I'll defer to your opinion.

That seems like the right place, but we need to make sure we show the syntax in other places where we show the enable command.

How common will the variants/flavors be? If they are likely to be fairly common (on most/all modules), maybe we should document the variant as part of the main syntax: filebeat modules enable apache2:variant. We'd mention that variant is optional. If no variant is specified, Filebeat uses the the default configuration.

I suspect that a lot users don't read the topic about specifying which modules to run. They see command examples in other places (like the getting started and the docs for each module), so we need to make sure they see the new syntax there, too.

Re naming:

IMO, the word "variant" isn't great, but I don't have a better suggestion. I do not like "flavor" very much either. :-P If you compare the actual definitions of the two terms, I think "variant" is more accurate:

variant - a form or version of something that differs in some respect from other forms of the same thing or from a standard.

flavor - an indication of the essential character of something.

[ycombinator]

@dedemorton

That seems like the right place, but we need to make sure we show the syntax in other places where we show the enable command.

I've added a section on configuration variants to that page in 501c8b6. Please take a look when you get a chance.

How common will the variants/flavors be? If they are likely to be fairly common (on most/all modules), maybe we should document the variant as part of the main syntax: filebeat modules enable apache2:variant. We'd mention that variant is optional. If no variant is specified, Filebeat uses the the default configuration.

I suspect that a lot users don't read the topic about specifying which modules to run. They see command examples in other places (like the getting started and the docs for each module), so we need to make sure they see the new syntax there, too.

I don't expect module configuration variants to be very common, at least initially. The specific use case that prompted this feature was for X-Pack Monitoring with Metricbeat so we'll make sure to add the variant syntax to those docs. (cc: @lcawl - I'll put up the PRs but just so you have context for them, this is it). Besides that, let's document the variant for a module when we introduce it, i.e. in follow up PRs to this one.

Re naming:

IMO, the word "variant" isn't great, but I don't have a better suggestion. I do not like "flavor" very much either. :-P If you compare the actual definitions of the two terms, I think "variant" is more accurate:

variant - a form or version of something that differs in some respect from other forms of the same thing or from a standard.

flavor - an indication of the essential character of something.

Naming is hard :) Unless someone feels strongly about the name, I'm inclined to go with "variants" given your finding that it's more accurate.

@ruflin I saw your note about "flavor" above but I wasn't sure if you were advocating for or against it as compared to "variant". Putting aside the case of the xpack variant (as it's a bit special), other examples that @jsoriano came up with were system:raid or kubernetes:events, as these sections are completely commented out in the respective module configuration files currently.

[kvch]

Why not call it String? I think it is more common to use functions named String to get the string representation of a type.

[ycombinator]

@kvch Made this change in 9ba27c7.

[ycombinator]

jenkins, test this