Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP result - Returning 502 instead of 200 #8993

Closed
Venkat2694 opened this issue Mar 16, 2021 · 4 comments
Closed

HTTP result - Returning 502 instead of 200 #8993

Venkat2694 opened this issue Mar 16, 2021 · 4 comments
Labels
area/prometheus bug unexpected problem or unintended behavior waiting for response waiting for response from contributor

Comments

@Venkat2694
Copy link

Host operating system: output of uname -a

Linux EUWBOMON01 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.17.0 (branch: HEAD, revision: 1bc7680)
build user: root
build date: 20200619-11:54:41
go version: go1.14.4

What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: true
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
preferred_ip_protocol: ip4
tcp_connect:
prober: tcp
timeout: 5s
tcp:
preferred_ip_protocol: ip4
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
http_2xx_home:
prober: http
timeout: 5s
http:
method: GET

fail_if_body_not_matches_regexp:

  • "OK"

fail_if_not_ssl: true

preferred_ip_protocol: ip4

http_2xx_tls:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4

What is the prometheus.yml scrape config.

job_name: 'blackbox-Product_Template'
scrape_interval: 30s
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:

  • files: ['./blackboxtargets/blackbox_product_template_targets.yml']
    relabel_configs:
  • source_labels: [address]
    target_label: __param_target
  • source_labels: [__param_target]
    target_label: instance
  • target_label: address
    replacement: Blackbox_server_IP
    What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2021-01-07T13:29:02.789489148Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=info msg="Beginning probe" probe=http timeout_seconds=25
ts=2021-01-07T13:29:02.789667447Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-01-07T13:29:02.804001657Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolved target address" ip=104.18.186.229
ts=2021-01-07T13:29:02.804057257Z caller=client.go:252 module=http_2xx target=https://www2.trustnet.com level=info msg="Making HTTP request" url=https://104.18.186.229 host=www2.trustnet.com
ts=2021-01-07T13:29:02.930748562Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Received HTTP response" status_code=502
ts=2021-01-07T13:29:02.930772962Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=502
ts=2021-01-07T13:29:02.930826762Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Response timings for roundtrip" roundtrip=0 start=2021-01-07T13:29:02.804120156Z dnsDone=2021-01-07T13:29:02.804120156Z connectDone=2021-01-07T13:29:02.806239843Z gotConn=2021-01-07T13:29:02.815712583Z responseStart=2021-01-07T13:29:02.930606663Z end=2021-01-07T13:29:02.930792862Z
ts=2021-01-07T13:29:02.930893061Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=error msg="Probe failed" duration_seconds=0.141338714

Metrics that would have been returned:

HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 0.01433891

HELP probe_duration_seconds Returns how long the probe took to complete in seconds

TYPE probe_duration_seconds gauge

probe_duration_seconds 0.141338714

HELP probe_failed_due_to_regex Indicates if probe failed due to regex

TYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

HELP probe_http_content_length Length of http content response

TYPE probe_http_content_length gauge

probe_http_content_length 1477

HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects

TYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="connect"} 0.002119787
probe_http_duration_seconds{phase="processing"} 0.11489408
probe_http_duration_seconds{phase="resolve"} 0.01433891
probe_http_duration_seconds{phase="tls"} 0.011592527
probe_http_duration_seconds{phase="transfer"} 0.000186199

HELP probe_http_redirects The number of redirects

TYPE probe_http_redirects gauge

probe_http_redirects 0

HELP probe_http_ssl Indicates if SSL was used for the final redirect

TYPE probe_http_ssl gauge

probe_http_ssl 1

HELP probe_http_status_code Response HTTP status code

TYPE probe_http_status_code gauge

probe_http_status_code 502

HELP probe_http_uncompressed_body_length Length of uncompressed response body

TYPE probe_http_uncompressed_body_length gauge

probe_http_uncompressed_body_length 1477

HELP probe_http_version Returns the version of HTTP of the probe response

TYPE probe_http_version gauge

probe_http_version 2

HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 3.641314862e+09

HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

TYPE probe_ip_protocol gauge

probe_ip_protocol 4

HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime

TYPE probe_ssl_earliest_cert_expiry gauge

probe_ssl_earliest_cert_expiry 1.6131312e+09

HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp seconds

TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge

probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10

HELP probe_success Displays whether or not the probe was a success

TYPE probe_success gauge

probe_success 0

HELP probe_tls_version_info Contains the TLS version used

TYPE probe_tls_version_info gauge

probe_tls_version_info{version="TLS 1.3"} 1

Module configuration:
prober: http
timeout: 25s
http:
valid_http_versions:

  • HTTP/1.0
  • HTTP/1.1
  • HTTP/2.0
    preferred_ip_protocol: ip4
    ip_protocol_fallback: true
    tls_config:
    insecure_skip_verify: true
    tcp:
    ip_protocol_fallback: true
    icmp:
    ip_protocol_fallback: true
    dns:
    ip_protocol_fallback: true

What did you expect to see?

Expect to see result of 200
We have looked into the page inspection and App Insights and both are returning the correct 200 value.

What did you see instead?

Getting an incorrect result of 502
BBE failed

When we check in other API we are getting correct HTTP response. Can you please help me to resolve this,?

Thanks,

@Venkat2694 Venkat2694 added the bug unexpected problem or unintended behavior label Mar 16, 2021
@sspaink
Copy link
Contributor

sspaink commented Mar 16, 2021

@Venkat2694 thank you for opening this issue, couple of questions. What version of Telegraf are you using? Are you able to share your Telegraf config? Are you using the InfluxDB Listener Input Plugin for monitoring http?

On another note, It is a bit hard to read the configs and logs you provided in the current format, can you update this issue by wrapping your config and logs with ``` to create a code block? It will be easier to read were one begins and one ends. (for more info you can refer to this document)

@sspaink sspaink added the waiting for response waiting for response from contributor label Mar 18, 2021
@Venkat2694
Copy link
Author

Venkat2694 commented Mar 21, 2021

Hi sspaink,

Thanks for your response.

I am using blackbox exporter for monitoring http not telegraf. I am using telegraf for windows servers.

Blackbox Module

modules:
http_2xx:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: true
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
preferred_ip_protocol: ip4
tcp_connect:
prober: tcp
timeout: 5s
tcp:
preferred_ip_protocol: ip4
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
http_2xx_home:
prober: http
timeout: 5s
http:
method: GET

fail_if_body_not_matches_regexp:

"OK"
fail_if_not_ssl: true

preferred_ip_protocol: ip4

http_2xx_tls:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4

Prometheus job

job_name: 'blackbox-Product_Template'
scrape_interval: 30s
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:

files: ['./blackboxtargets/blackbox_product_template_targets.yml']
relabel_configs:
source_labels: [address]
target_label: __param_target
source_labels: [__param_target]
target_label: instance
target_label: address
replacement: Blackbox_server_IP

Logs for the probe

ts=2021-01-07T13:29:02.789489148Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=info msg="Beginning probe" probe=http timeout_seconds=25
ts=2021-01-07T13:29:02.789667447Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-01-07T13:29:02.804001657Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolved target address" ip=104.18.186.229
ts=2021-01-07T13:29:02.804057257Z caller=client.go:252 module=http_2xx target=https://www2.trustnet.com level=info msg="Making HTTP request" url=https://104.18.186.229 host=www2.trustnet.com
ts=2021-01-07T13:29:02.930748562Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Received HTTP response" status_code=502
ts=2021-01-07T13:29:02.930772962Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=502
ts=2021-01-07T13:29:02.930826762Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Response timings for roundtrip" roundtrip=0 start=2021-01-07T13:29:02.804120156Z dnsDone=2021-01-07T13:29:02.804120156Z connectDone=2021-01-07T13:29:02.806239843Z gotConn=2021-01-07T13:29:02.815712583Z responseStart=2021-01-07T13:29:02.930606663Z end=2021-01-07T13:29:02.930792862Z
ts=2021-01-07T13:29:02.930893061Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=error msg="Probe failed" duration_seconds=0.141338714

Metrics that would have been returned:

HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 0.01433891

HELP probe_duration_seconds Returns how long the probe took to complete in seconds

TYPE probe_duration_seconds gauge

probe_duration_seconds 0.141338714

HELP probe_failed_due_to_regex Indicates if probe failed due to regex

TYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

HELP probe_http_content_length Length of http content response

TYPE probe_http_content_length gauge

probe_http_content_length 1477

HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects

TYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="connect"} 0.002119787
probe_http_duration_seconds{phase="processing"} 0.11489408
probe_http_duration_seconds{phase="resolve"} 0.01433891
probe_http_duration_seconds{phase="tls"} 0.011592527
probe_http_duration_seconds{phase="transfer"} 0.000186199

HELP probe_http_redirects The number of redirects

TYPE probe_http_redirects gauge

probe_http_redirects 0

HELP probe_http_ssl Indicates if SSL was used for the final redirect

TYPE probe_http_ssl gauge

probe_http_ssl 1

HELP probe_http_status_code Response HTTP status code

TYPE probe_http_status_code gauge

probe_http_status_code 502

HELP probe_http_uncompressed_body_length Length of uncompressed response body

TYPE probe_http_uncompressed_body_length gauge

probe_http_uncompressed_body_length 1477

HELP probe_http_version Returns the version of HTTP of the probe response

TYPE probe_http_version gauge

probe_http_version 2

HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 3.641314862e+09

HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

TYPE probe_ip_protocol gauge

probe_ip_protocol 4

HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime

TYPE probe_ssl_earliest_cert_expiry gauge

probe_ssl_earliest_cert_expiry 1.6131312e+09

HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp seconds

TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge

probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10

HELP probe_success Displays whether or not the probe was a success

TYPE probe_success gauge

probe_success 0

HELP probe_tls_version_info Contains the TLS version used

TYPE probe_tls_version_info gauge

probe_tls_version_info{version="TLS 1.3"} 1

Expect to see result of 200
We have looked into the page inspection and App Insights and both are returning the correct 200 value.

What did you see instead?

Getting an incorrect result of 502
BBE failed

![image](https://user-images.githubusercontent.com/71388748/111898297-27369180-8a4b-11eb-9060-ac5fb7afd866.png)

@Venkat2694
Copy link
Author

@Venkat2694 thank you for opening this issue, couple of questions. What version of Telegraf are you using? Are you able to share your Telegraf config? Are you using the InfluxDB Listener Input Plugin for monitoring http?

On another note, It is a bit hard to read the configs and logs you provided in the current format, can you update this issue by wrapping your config and logs with ``` to create a code block? It will be easier to read were one begins and one ends. (for more info you can refer to this document)

Hi @sspaink ,
I hope i have provided the required details. Please let me know if anymore.
Can you please help me on this issue.?

Thanks,
Venkatraman N

@sspaink
Copy link
Contributor

sspaink commented Mar 29, 2021

This reads like this might not be an issue with Telegraf or one of its plugins? 502 means there is a bad gateway and most likely an environment issue and not a code issue. I also don't have any experience with blackbox exporter and this is not the best place to ask for help with that tool. I'd recommend either asking for help in our Telegraf community slack channel for support or looking for help in the prometheus/blackbox project.

Closing this issue because I don't think there is anything we can change in Telegraf to help you.

@sspaink sspaink closed this as completed Mar 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/prometheus bug unexpected problem or unintended behavior waiting for response waiting for response from contributor
Projects
None yet
Development

No branches or pull requests

2 participants