[Inputs.prometheus] Double Accept header using `http_headers` #12906

qistoph · 2023-03-19T12:19:46Z

Relevant telegraf.conf

[[inputs.prometheus]]
  urls = ["http://server:9300/metrics"]
  name_prefix = "authentik_"
  http_headers = {"Accept" = "text/plain", "Accept-Too" = "test/asdf"}

Logs from Telegraf

2023-03-19T12:13:46Z I! Loading config file: /etc/telegraf/telegraf-debug.conf
2023-03-19T12:13:46Z W! DeprecationWarning: Option "response_timeout" of plugin "inputs.prometheus" deprecated since version 1.26.0 and will be removed in 2.0.0: use 'timeout' instead
2023-03-19T12:13:46Z I! Starting Telegraf 1.26.0
2023-03-19T12:13:46Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-03-19T12:13:46Z I! Loaded inputs: prometheus
2023-03-19T12:13:46Z I! Loaded aggregators:
2023-03-19T12:13:46Z I! Loaded processors: converter
2023-03-19T12:13:46Z I! Loaded secretstores:
2023-03-19T12:13:46Z I! Loaded outputs: influxdb (4x)
2023-03-19T12:13:46Z I! Tags enabled: host=bashful
2023-03-19T12:13:46Z W! Deprecated inputs: 0 and 1 options
2023-03-19T12:13:46Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"bashful", Flush Interval:10s
2023-03-19T12:13:46Z D! [agent] Initializing plugins
2023-03-19T12:13:46Z D! [agent] Connecting outputs
2023-03-19T12:13:46Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-19T12:13:46Z D! [agent] Successfully connected to outputs.influxdb
2023-03-19T12:13:46Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-19T12:13:46Z D! [agent] Successfully connected to outputs.influxdb
2023-03-19T12:13:46Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-19T12:13:46Z D! [agent] Successfully connected to outputs.influxdb
2023-03-19T12:13:46Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-19T12:13:46Z D! [agent] Successfully connected to outputs.influxdb
2023-03-19T12:13:46Z D! [agent] Starting service inputs
2023-03-19T12:13:56Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:13:56Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:13:56Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:13:56Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:01Z E! [inputs.prometheus] Error in plugin: error reading metrics for "http://server:9300/metrics": reading metric family protocol buffer failed: proto: cannot parse invalid wire-format data
2023-03-19T12:14:06Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:06Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:06Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:06Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
^C2023-03-19T12:14:08Z D! [agent] Stopping service inputs
2023-03-19T12:14:08Z D! [agent] Input channel closed
2023-03-19T12:14:08Z D! [agent] Processor channel closed
2023-03-19T12:14:08Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-03-19T12:14:08Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:08Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:08Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:08Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-19T12:14:08Z I! [agent] Stopping running outputs
2023-03-19T12:14:08Z D! [agent] Stopped Successfully

System info

Telegraf 1.26.0 (git: HEAD@064650c8) in Docker 20.10.21

Docker

No response

Steps to reproduce

Start an Authentik instance
Configure Telegraf to read its metrics

Expected behavior

Authentik metrics are read (using Prometheus interface)

Actual behavior

Looks a lot like issue #12749

telegraf    | 2023-03-18T22:07:30Z E! [inputs.prometheus] Error in plugin: error reading metrics for "http://server:9300/metrics": reading metric family protocol buffer failed: proto: cannot parse invalid wire-format data

Additional info

The configured HTTP Accept Header seems to be added instead of replacing the original. Authentik probably only uses the first.

Headers in the request

Host: server:9300
User-Agent: Telegraf/1.26.0 Go/1.20.2
Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3
Accept: text/plain
Accept-Too: test/asdf
Accept-Encoding: gzip

Tcpdump

22:07:30.031580 IP 192.168.0.3.43244 > 192.168.0.2.9300: Flags [P.], seq 1:287, ack 1, win 502, options [nop,nop,TS val 751130760 ecr 2090594872], length 286
        0x0000:  4500 0152 82cf 4000 4006 3581 c0a8 0003  E..R..@.@.5.....
        0x0010:  c0a8 0002 a8ec 2454 013d 7207 4cae 67ca  ......$T.=r.L.g.
        0x0020:  8018 01f6 829a 0000 0101 080a 2cc5 5888  ............,.X.
        0x0030:  7c9b f238 4745 5420 2f6d 6574 7269 6373  |..8GET./metrics
        0x0040:  2048 5454 502f 312e 310d 0a48 6f73 743a  .HTTP/1.1..Host:
        0x0050:  2073 6572 7665 723a 3933 3030 0d0a 5573  .server:9300..Us
        0x0060:  6572 2d41 6765 6e74 3a20 5465 6c65 6772  er-Agent:.Telegr
        0x0070:  6166 2f31 2e32 362e 3020 476f 2f31 2e32  af/1.26.0.Go/1.2
        0x0080:  302e 320d 0a41 6363 6570 743a 2061 7070  0.2..Accept:.app
        0x0090:  6c69 6361 7469 6f6e 2f76 6e64 2e67 6f6f  lication/vnd.goo
        0x00a0:  676c 652e 7072 6f74 6f62 7566 3b70 726f  gle.protobuf;pro
        0x00b0:  746f 3d69 6f2e 7072 6f6d 6574 6865 7573  to=io.prometheus
        0x00c0:  2e63 6c69 656e 742e 4d65 7472 6963 4661  .client.MetricFa
        0x00d0:  6d69 6c79 3b65 6e63 6f64 696e 673d 6465  mily;encoding=de
        0x00e0:  6c69 6d69 7465 643b 713d 302e 372c 7465  limited;q=0.7,te
        0x00f0:  7874 2f70 6c61 696e 3b76 6572 7369 6f6e  xt/plain;version
        0x0100:  3d30 2e30 2e34 3b71 3d30 2e33 0d0a 4163  =0.0.4;q=0.3..Ac
        0x0110:  6365 7074 3a20 7465 7874 2f70 6c61 696e  cept:.text/plain
        0x0120:  0d0a 4163 6365 7074 2d54 6f6f 3a20 7465  ..Accept-Too:.te
        0x0130:  7374 2f61 7364 660d 0a41 6363 6570 742d  st/asdf..Accept-
        0x0140:  456e 636f 6469 6e67 3a20 677a 6970 0d0a  Encoding:.gzip..
        0x0150:  0d0a                                     ..
22:07:30.031643 IP 192.168.0.2.9300 > 192.168.0.3.43244: Flags [.], ack 287, win 507, options [nop,nop,TS val 2090594876 ecr 751130760], length 0
        0x0000:  4500 0034 c0a6 4000 4006 f8c7 c0a8 0002  E..4..@.@.......
        0x0010:  c0a8 0003 2454 a8ec 4cae 67ca 013d 7325  ....$T..L.g..=s%
        0x0020:  8010 01fb 817c 0000 0101 080a 7c9b f23c  .....|......|..<
        0x0030:  2cc5 5888                                ,.X.
22:07:30.032980 IP 192.168.0.2.9300 > 192.168.0.3.43244: Flags [P.], seq 1:4097, ack 287, win 507, options [nop,nop,TS val 2090594878 ecr 751130760], length 4096
        0x0000:  4500 1034 c0a7 4000 4006 e8c6 c0a8 0002  E..4..@.@.......
        0x0010:  c0a8 0003 2454 a8ec 4cae 67ca 013d 7325  ....$T..L.g..=s%
        0x0020:  8018 01fb 917c 0000 0101 080a 7c9b f23e  .....|......|..>
        0x0030:  2cc5 5888 4854 5450 2f31 2e31 2032 3030  ,.X.HTTP/1.1.200
        0x0040:  204f 4b0d 0a43 6f6e 7465 6e74 2d54 7970  .OK..Content-Typ
        0x0050:  653a 2061 7070 6c69 6361 7469 6f6e 2f76  e:.application/v
        0x0060:  6e64 2e67 6f6f 676c 652e 7072 6f74 6f62  nd.google.protob
        0x0070:  7566 3b20 7072 6f74 6f3d 696f 2e70 726f  uf;.proto=io.pro
        0x0080:  6d65 7468 6575 732e 636c 6965 6e74 2e4d  metheus.client.M
        0x0090:  6574 7269 6346 616d 696c 793b 2065 6e63  etricFamily;.enc
        0x00a0:  6f64 696e 673d 6465 6c69 6d69 7465 640d  oding=delimited.
        0x00b0:  0a44 6174 653a 2053 6174 2c20 3138 204d  .Date:.Sat,.18.M
        0x00c0:  6172 2032 3032 3320 3232 3a30 373a 3330  ar.2023.22:07:30
        0x00d0:  2047 4d54 0d0a 5472 616e 7366 6572 2d45  .GMT..Transfer-E
        0x00e0:  6e63 6f64 696e 673a 2063 6875 6e6b 6564  ncoding:.chunked
        0x00f0:  0d0a 0d0a 3830 300d 0af3 010a 1761 7574  ....800......aut
        0x0100:  6865 6e74 696b 5f6d 6169 6e5f 7265 7175  hentik_main_requ
        0x0110:  6573 7473 1228 5468 6520 746f 7461 6c20  ests.(The.total.
        0x0120:  6e75 6d62 6572 206f 6620 636f 6e66 6967  number.of.config
        0x0130:  7572 6564 2070 726f 7669 6465 7273 1804  ured.providers..
        0x0140:  22ab 010a 0c0a 0464 6573 7412 0463 6f72  "......dest..cor
        0x0150:  653a 9a01 0861 1100 0000 007c 7921 411a  e:...a.....|y!A.
        0x0160:  0b08 0011 7b14 ae47 e17a 743f 1a0b 0800  ....{..G.zt?....
        0x0170:  117b 14ae 47e1 7a84 3f1a 0b08 0011 9a99  .{..G.z.?.......
        0x0180:  9999 9999 993f 1a0b 0800 119a 9999 9999  .....?..........
        0x0190:  99a9 3f1a 0b08 0011 9a99 9999 9999 b93f  ..?............?
        0x01a0:  1a0b 0800 1100 0000 0000 00d0 3f1a 0b08  ............?...
        0x01b0:  0011 0000 0000 0000 e03f 1a0b 0800 1100  .........?......
        0x01c0:  0000 0000 00f0 3f1a 0b08 0011 0000 0000  ......?.........
        0x01d0:  0000 0440 1a0b 0800 1100 0000 0000 0014  ...@............
       **SNIP***

The text was updated successfully, but these errors were encountered:

qistoph · 2023-03-19T13:29:16Z

Tweaking the Prometheus plugin to use Set for custom headers instead of Add seems to fix the issue.

Would that cause issues with existing use of http_headers?

diff --git a/plugins/inputs/prometheus/prometheus.go b/plugins/inputs/prometheus/prometheus.go
index 03c0b3811..ebb01b0c2 100644
--- a/plugins/inputs/prometheus/prometheus.go
+++ b/plugins/inputs/prometheus/prometheus.go
@@ -364,7 +364,7 @@ func (p *Prometheus) gatherURL(u URLAndAddress, acc telegraf.Accumulator) error

        if p.HTTPHeaders != nil {
                for key, value := range p.HTTPHeaders {
-                       req.Header.Add(key, value)
+                       req.Header.Set(key, value)
                }
        }

Headers with above change:

Host: server:9300
User-Agent: Telegraf/1.27.0-1ccad43d Go/1.20.2
Accept: text/plain
Accept-Too: test/asdf
Accept-Encoding: gzip

fixes: influxdata#12906

powersj · 2023-03-20T15:47:42Z

Tweaking the Prometheus plugin to use Set for custom headers instead of Add seems to fix the issue.

yeah looks like add appends to a header while set would overwrite. I would expect if a user is setting a value, to overwrite anything else.

Can you try the PR artifacts found in #12907 in 20-30mins to confirm the change?

qistoph · 2023-03-21T06:28:54Z

Tested the Linux artifacts in #12907 and they fix the issue with Authentik. The Accept header is overwritten as proposed

redbaron · 2023-03-21T14:34:10Z

It is OK for Accept header to be specified multiple times, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept

powersj · 2023-03-21T14:43:18Z

It is OK for Accept header to be specified multiple times, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept

Agreed, then is there a better way to resolve this issue for the user? If a user is explicit about what they want to accept, it seems wise to follow through and only accept that.

We seem to have a mix of set and add throughout telegraf as it is today.

redbaron · 2023-03-21T14:49:42Z

This particular problem can probably be resolved with:

http_headers = {"Accept" = "text/plain;version=0.0.4;q=1" }

if server is standard compliant.

as for general set vs add I think set is better fit because it allows users better control over request headers to overcome server side bugs/limitations.

qistoph added the bug unexpected problem or unintended behavior label Mar 19, 2023

powersj added a commit to powersj/telegraf that referenced this issue Mar 20, 2023

fix(inputs.prometheus): use set over add for custom headers

18ce9ff

fixes: influxdata#12906

powersj mentioned this issue Mar 20, 2023

fix(inputs.prometheus): use set over add for custom headers #12907

Merged

powersj added the waiting for response waiting for response from contributor label Mar 20, 2023

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Mar 21, 2023

srebhan closed this as completed in #12907 Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Inputs.prometheus] Double Accept header using `http_headers` #12906

[Inputs.prometheus] Double Accept header using `http_headers` #12906

qistoph commented Mar 19, 2023 •

edited

Loading

qistoph commented Mar 19, 2023 •

edited

Loading

powersj commented Mar 20, 2023

qistoph commented Mar 21, 2023

redbaron commented Mar 21, 2023

powersj commented Mar 21, 2023

redbaron commented Mar 21, 2023

[Inputs.prometheus] Double Accept header using http_headers #12906

[Inputs.prometheus] Double Accept header using http_headers #12906

Comments

qistoph commented Mar 19, 2023 • edited Loading

Relevant telegraf.conf

Logs from Telegraf

System info

Docker

Steps to reproduce

Expected behavior

Actual behavior

Additional info

qistoph commented Mar 19, 2023 • edited Loading

powersj commented Mar 20, 2023

qistoph commented Mar 21, 2023

redbaron commented Mar 21, 2023

powersj commented Mar 21, 2023

redbaron commented Mar 21, 2023

[Inputs.prometheus] Double Accept header using `http_headers` #12906

[Inputs.prometheus] Double Accept header using `http_headers` #12906

qistoph commented Mar 19, 2023 •

edited

Loading

qistoph commented Mar 19, 2023 •

edited

Loading