Doc: Working with Logstash Plugins on ECK

[karenzone]

Fixes: https://github.com/elastic/ingest-dev/issues/3023

Additional iterations based on excellent work in @robbavey's feature branch.

PREVIEW: https://cloud-on-k8s_bk_7702.docs-preview.app.elstc.co/guide/en/cloud-on-k8s/master/k8s-logstash-plugins.html

[robbavey]

This is great stuff! I've added some thoughts and comments

[kaisecheng]

The docs look mature, really awesome.

[kaisecheng]

suggestion: add a link to environment variables to point user how to use it. https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash-configuration.html#k8s-logstash-pipelines-es

[karenzone]

To make this work, I added a new heading called "elasticsearchRefs for establishing a secured connection". I'm linking to the new section from the elastic_integration filter section.

@karenzone note to self: Restructure to make this flow better in the next iteration

[kaisecheng]

Would you also consider adding elasticsearch-output plugin in this session? I know it is duplicated to https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash-configuration.html#k8s-logstash-pipelines-es
As I see elastic_integration-filter has a touch on setting role and permission, it seems the same category for es-ouput to add role and permission setup.

"cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get",],
  "indices": [
    {
      "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ],
      "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]
    }

[karenzone]

Please see if I implemented this as you pictured it.

[kaisecheng]

btw, the source code in the original note is missing ] at the end and having an extra , after "cluster:admin/ingest/pipeline/get" which will cause syntax error :)

This is the correct yaml

"cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get"],
"indices": [
  {
    "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ],
    "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]
  }
]

[karenzone]

@robbavey, please LMKWYT and if we can move to "Ready for Review" status.

[robbavey]

I think we're really close!

[karenzone]

I'll resync base branch again when we're done making changes.

[robbavey]

LGTM

[karenzone]

@robbavey, thank you for your work and approval on this.
As for next steps... Should I request someone from the ECK team to review? And if so, who should that be?

[robbavey]

@karenzone Yes, we should get a review from the ECK team:

@pebrc @thbkrkr @barkbay This documentation PR, to give more information on how to work with plugins to Logstash on ECK users, is ready for review.

[pebrc]

LGTM

[karenzone]

Updated docs are here: https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash-plugins.html