Plugin fails when specifying buffer chunk keys

[davide-bolcioni]

What happened:
The following buffer configuration for a match with type @splunk_plugin_hec (v 1.2.4)

              <buffer tag,time>
                @type file
                path  /var/log/fluentd-buffers/kubernetes.splunk.buffer
                ...
              </buffer>

failed with the following exception

2020-09-10 00:45:17 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="this plugin 'Fluent::Plugin::SplunkHecOutput' cannot handle arguments for <buffer ...> section"

What you expected to happen:
No exception and separate chunk buffers on disk.

How to reproduce it (as minimally and precisely as possible):
See above configuration.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.17 and 1.18
• Ruby version (use ruby --version): 2.5.0
• OS (e.g: cat /etc/os-release): fluentd Docker image fluent/fluentd:v1.11 with Splunk plugin added
• Splunk version: 8
• Others:

[rockb1017]

Hello, Davide,
Glad to see you on github.

Unfortunately, our plugin doesn’t support buffer key options. so this is expected that it throws exception.

[jasonzlai]

Hey @rockb1017! Thanks for the quick reply.

I've glanced a bit at the source code, it seems like this plugin has been using a specific version of Fluentd plugin plumbing that doesn't allow buffer key options. Do you mind sharing a bit more specifics on why it needs to be this way?

[rockb1017]

Hello, Jason!

yea, it inherits this class "Fluent::BufferedOutput" which doesn't support. I am not sure on why. It was before I joined Splunk.

Do you want to shard chunks by tag because you want to make chunk of events for the same index?

[jasonzlai]

We tag log events by their originating Kubernetes namespaces. Due to the heterogeneous log emission patterns by different services, we would prefer that log events without corresponding Splunk indexes (mapped by K8s namespaces) wouldn't interfere with those that have indexes.

Is this something addressable at upstream?

[rockb1017]

hello,

in Splunk Connect for Kubernetes chart, we have fluentd configmap for getting index name from k8s annotations("splunk.com/index") and if empty, ingest them to default index. I think this would be helpful for you.
https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/templates/configMap.yaml

so you can configure your pipeline to have a default index name to be used so that it would be collected in the default index instead of dropping them. Then other valid logs in the same batch wouldn't be dropped as well.

in the SCK chart, we also have "splunk.com/exclude" annotation to not ingest any logs from pods or namespaces.

[davide-bolcioni]

Imagine a wilder west, where there is no control over the distributed action of "trying to send to an index" ...

[hvaghani221]

It should be fixed when #208 is merged.
EDIT: PR has been merged, so it should work from fluentd-hec v1.2.10