splunk output with splunk_send_raw on is using the wrong endpoint

[pmeier]

Bug Report

Describe the bug

When using the splunk output with Splunk_Send_Raw On the data is send to the /services/collector/raw endpoint rather than /services/collector/event:

fluent-bit/plugins/out_splunk/splunk.h

Lines 25 to 26 in c3d1280

	#define FLB_SPLUNK_DEFAULT_URI_RAW "/services/collector/raw"
	#define FLB_SPLUNK_DEFAULT_URI_EVENT "/services/collector/event"

This seems to be wrong. To the best of my understanding, the term "raw" for Splunk means "raw log message without metadata". It is not entirely clear from their documentation, but in the examples they are only ever explicitly using the /services/collector/raw endpoint when sending pure messages. This is also supported by our observations that when we are using Splunk_Send_Raw On, the whole record is showing up as JSON string as message, rather than it being parsed out.

This was introduced in aeb18f7 (cc @edsiper) and first released with v1.8.0. The commit message states

In addition this patch fixes the raw mode endpoint.

without further indication what exactly was supposed to be fixed by this. In contrast, this likely broke the raw mode.

[agup006]

We should make the URi configurable with send raw @cosmo0920

[pmeier]

It would solve my issue, but I'm not sure why you want to ever hit the /raw endpoint from fluent-bit. This endpoint is for sending plain strings and attaching the timestamp on when the message was received to it. This is equivalent of sending the following JSON object to the /event endpoint

{
  "event": "<message sent to the /raw endpoint>",
  "time": "<timestamp when the message was received>"
}

This wrapping is exactly what the splunk output in fluent bit is doing by default, i.e. Splunk_Send_Raw Off: the internal record is nested under the event key and the timestamp is inserted as the time key.

Thus, there is never a need to use the /raw endpoint from fluent-bit:

• Either you don't want to perform the wrapping yourself. In this case you use the default behavior (or set Splunk_Send_Raw Off explicitly) and leave this to fluent-bit
• Or you want to have specific control over what is being sent to splunk. In this case you set Splunk_Send_Raw On and format the record to whatever shape you want it to be.

In both cases however, the output from fluent-bit is JSON object and thus has to go to the /event endpoint and never to the /raw endpoint.