Telegraf to Splunk [[outputs.http]]

[abehn-lbi]

Relevant telegraf.conf

# Global tags relate to and are available for use in Splunk searches
# Of particular note are the index tag, which is required to match the
# configured metrics index name and the cluster tag which should match the
# value of Vault's cluster_name configuration option value.

[global_tags]
  index = "metrics"
  datacenter = "test"
  role = "server"
  cluster = "test"

# Agent options around collection interval, sizes, jitter and so on
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
  logtarget = "file"
  logfile = "/var/log/telegraf/telegraf.log"
  logfile_rotation_interval = "0d"
  logfile_rotation_max_size = "50MB"
  logfile_rotation_max_archives = 5

# An input plugin that listens on UDP/8125 for statsd compatible telemetry
# messages using Datadog extensions which are emitted by Vault
[[inputs.statsd]]
  protocol = "udp"
  service_address = ":8125"
  metric_separator = "."
  datadog_extensions = true

# An output plugin that can transmit metrics over HTTP to Splunk
# You must specify a valid Splunk HEC token as the Authorization value
[[outputs.http]]
  url = "https://splunk:8088/services/collector"
  data_format = "splunkmetric"
  splunkmetric_hec_routing = true
  [outputs.http.headers]
    Content-Type = "application/json"
    Authorization = "Splunk tokengoeshere"
    X-Splunk-Request-Channel = "tokengoeshere"

# Read metrics about cpu usage using default configuration values
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

# Read metrics about memory usage
[[inputs.mem]]
  # No configuration required

# Read metrics about swap memory usage
[[inputs.swap]]
  # No configuration required

# Read metrics about disk usage using default configuration values
[[inputs.disk]]
  ## By default stats will be gathered for all mount points.
  ## Set mount_points will restrict the stats to only the specified mount points.
  ## mount_points = ["/"]
  ## Ignore mount points by filesystem type.
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.diskio]]
  # devices = ["sda", "sdb"]
  # skip_serial_number = false

[[inputs.kernel]]
  # No configuration required

[[inputs.linux_sysctl_fs]]
  # No configuration required

[[inputs.net]]
  # Specify an interface or all
  # interfaces = ["enp0s*"]

[[inputs.netstat]]
  # No configuration required

[[inputs.processes]]
  # No configuration required

[[inputs.procstat]]
 pattern = "(vault)"

[[inputs.system]]
  # No configuration required

Logs from Telegraf

telegraf -debug
2022-07-13T21:00:54Z I! Using config file: /etc/telegraf/telegraf.conf
2022-07-13T21:00:54Z I! Starting Telegraf 1.23.2
2022-07-13T21:00:54Z I! Loaded inputs: cpu disk diskio kernel linux_sysctl_fs mem net netstat processes procstat statsd swap system
2022-07-13T21:00:54Z I! Loaded aggregators: 
2022-07-13T21:00:54Z I! Loaded processors: 
2022-07-13T21:00:54Z I! Loaded outputs: http
2022-07-13T21:00:54Z I! Tags enabled: cluster=test datacenter=test host=test index=metrics role=server
2022-07-13T21:00:54Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"test", Flush Interval:10s
2022-07-13T21:00:54Z D! [agent] Initializing plugins
2022-07-13T21:00:54Z D! [agent] Connecting outputs
2022-07-13T21:00:54Z D! [agent] Attempting connection to [outputs.http]
2022-07-13T21:00:54Z D! [agent] Successfully connected to outputs.http
2022-07-13T21:00:54Z D! [agent] Starting service inputs
2022-07-13T21:00:54Z I! [inputs.statsd] UDP listening on "[::]:8125"
2022-07-13T21:00:54Z I! [inputs.statsd] Started the statsd service on ":8125"
2022-07-13T21:01:04Z D! Can not parse value:  2:02 for key: uptime_format
2022-07-13T21:01:05Z D! [outputs.http] Wrote batch of 27 metrics in 122.797122ms
2022-07-13T21:01:05Z D! [outputs.http] Buffer fullness: 0 / 10000 metrics
2022-07-13T21:01:14Z D! Can not parse value:  2:02 for key: uptime_format
2022-07-13T21:01:15Z D! [outputs.http] Wrote batch of 30 metrics in 67.778451ms
2022-07-13T21:01:15Z D! [outputs.http] Buffer fullness: 0 / 10000 metrics

System info

Telegraf 1.23.2-1, Oracle Linux 8.6

Docker

No response

Steps to reproduce

1. Install any 1.23.x version on your Oracle Linux (RHEL) 8.6 node "yum install telegraf -y"
2. Data is sent to Splunk, but nothing is being index for whatever reason.
3. Downgrade: "yum install telegraf-1.22.4-1 -y"
4. It works!
   ...

Expected behavior

Metrics work on the downgraded version, same config. Not sure why 1.23.x is having an issue.

Actual behavior

Data appears to be sent.

Additional info

No response

[powersj]

Hi,

Thanks for taking the time to file an issue. There was one change to the splunk metric serializer I think in this time frame via #11237. This came from users of Splunk where they were expecting the "event" field. Do you think this is relevant to your use-case?

Otherwise, it would be good to see if you could do the following:

1. Narrow down the inputs to what is causing the issue/difference
2. Use the[[outputs.file]] plugin to output the data to stdout and then compare between the two versions

Thanks!

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!

[pgeler]

need to open this one, and I can confirm it's related to #11237 @powersj I have exactly the case, we were using 1.20 after the update to 1.24 hec import declined to work(splunk cloud), the current state is updated to 1.22 as that is the latest version without "event" field