metrics-hijacker should scrape app metrics on localhost

[lahabana]

What happened?

2022-09-23T11:23:05.035Z	ERROR	metrics-hijacker	failed call	{"name": "application", "path": "/metrics", "port": 5000, "error": "Get \"http://10.42.0.17:5000/metrics\": dial tcp 127.0.0.6:0->10.42.0.17:5000: connect: connection refused"}

This enables folks to bind their "troubleshooting" listener to localhost only and increase their security.

[jakubdyszkiewicz]

Triage: so the app listener is on "public IP", but the metrics listener is on localhost?

[lahabana]

I need to double check this because I think I was maybe wrong.

I think you can force it by setting "address" my gut feeling is that this should be the default.

[jakubdyszkiewicz]

Before we start reproducing this.

This enables folks to bind their "troubleshooting" listener to localhost only and increase their security

But they can't with transparent proxy. We recently fixed the behavior that you cannot bind the app to localhost and use a transparent proxy.

[lahabana]

But this is the "troubleshooting" listener right? It's separate from the actual app listener no?

[jakubdyszkiewicz]

Triage: we want to keep it as is for now. For the majority of cases, the metrics listener is public. If needed, we can reopen.
The only case that this would be useful is if you expose metrics and other important endpoints like heap dump on the same listener and you want to hide it.

[lukidzi]

Looks like there might be a use case to create hijacking metrics from applications binding to localhost. We should implement this.