How to integrate PushProx's service discovery into Promgen?

[saulp]

Hi all,

I am trying to feed PushProx's service discovery into Promgen. (For those new to it, PushProx is a proxy that serves as a messaging proxy between Prometheus, and hosts behind an inaccessible network setup.)

So far I have both PushProx and Promgen working (yay!), and now I'm trying to sort out how to make Promgen aware of the hosts reported by PushProx's service discovery (SD).

PushProx publishes the targets it's aware of in (an admittedly barebones) JSON format, e.g.

[{"targets":["mybox.domain.com"],"labels":null}]

(The above is exposed via PushProx's /clients endpoint, but the dev-recommended way to use it is to copy the above via a cronjob somewhere file_sd_configs can read it.)

What would be a recommended way to automatically import the above targets into Promgen? I'm aware I could bypass Promgen and simply use Prometheus to absorb these targets, but I really want Promgen to be aware of them so my team can use it to configure rules and alerts. And I'm also aware we could skip the PushProx SD and enter the targets manually into Promgen... but I'm also hoping that won't be necessary as the manual effort won't be negligible due to the expected number of hosts we will be adding/updating.

Any and all advice will be welcome. Thanks!

[kfdm]

Hmm, that's an interesting use case that has not come up yet. I am familiar with PushProx but have not used it myself

Right now, Promgen does have an import method, but that was intended to import from other Promgen instances (in the case you wanted to move a group from a beta environment to a release environment or something like that)

It currently expects that there would be labels for service,project,job so that promgen would know where to group things.

1. If you wanted to just manage rules + alerts, that's a viable option even without importing the list
2. If you know a bit of Django, it would also not be difficult to write a small cli command to import the list

I can write a more detailed explanation about either of those if that sounds like something that would work in your environment. I will also take another glance at PushProx to refresh myself and see what it does in terms of labels

[saulp]

Thanks for the prompt reply, Paul! Option 1 is sounding more attractive at the moment, if only because I am not super familiar with Django... but now that I think about it, the CLI option might the most flexible way to add the labels I'd be missing. If you are willing to share details on both options I'm all ears :)

…

On Tue, Jul 30, 2019, 1:23 AM Paul Traylor ***@***.***> wrote: Hmm, that's an interesting use case that has not come up yet. I am familiar with PushProx but have not used it myself Right now, Promgen does have an import method, but that was intended to import from other Promgen instances (in the case you wanted to move a group from a beta environment to a release environment or something like that) It currently expects that there would be labels for service,project,job so that promgen would know where to group things. 1. If you wanted to just manage rules + alerts, that's a viable option even without importing the list 2. If you know a bit of Django, it would also not be difficult to write a small cli command to import the list I can write a more detailed explanation about either of those if that sounds like something that would work in your environment. I will also take another glance at PushProx to refresh myself and see what it does in terms of labels — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#158?email_source=notifications&email_token=ACEOMQWWIYZV7CGAIXHZHGLQB7F5NA5CNFSM4IHYNHQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3CZX4A#issuecomment-516266992>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACEOMQWRZFJPBM6CHGG6FEDQB7F5NANCNFSM4IHYNHQA> .

[saulp]

Hi again Paul,

I'm thinking that for now I'll keep it simple, and will just try to add the proxied targets manually.

I've run into an issue, though: how can I set properties such as "proxy_url" and "params" when adding a host manually in Promgen?

For instance, this is the relevant portion of my current prometheus.yml file for a job that scrapes the Node Exporters of hosts server1 and server2 through the PushProx proxy. (PushProx is running on the same server as Prometheus on port 8080) :

  - job_name: proxied_node
    proxy_url: 'http://127.0.0.1:8080'
    params:
      collect[]:
        - filesystem
        - systemd
        - cpu
        - meminfo
    static_configs:
    - targets: ['server1:9100']
    - targets: ['server2:9100']

[kfdm]

Sorry for the delay.
proxy_url and params can be set in the Prometheus configuration. Currently there is not a way to set those in Promgen. Params could be a good option to add to Promgen (and actually metrics_path is set in a similar way) but there's not a good way to allow Promgen to manage proxy_url since that's entirely handled by Prometheus.

[saulp]

Hey Paul,

Thanks again - I have sorted my proxy_url and params needs via the Prometheus config, along with 'relabel_configs' and 'keep'.

I am sort of back to square one regarding my discovery needs, though... We'll actually be using Ansible to deploy monitoring to new hosts, and our goal is to end up with a script (cli or otherwise) that Ansible can kick off to add hosts to a Promgen farm.

I'll appreciate it if you can point me in the right direction.. I'm new to Django but not to Python, so I may not need much more than a little bit of info to get started ;)

[kfdm]

Right now, there's not a simple way to do this via ansible.

There is a promgen import command, that expects a list of targets in the file_sd_config format, but that's probably not easy to generate from Ansible.

One possible option, that probably would not require much code, and would probably be easy to use from ansible, is to add a CLI command that allows you to add a monitoring target. You can look in promgen/management/commands for some examples, but perhaps a command like

promgen register-job <project> <job> <port> [path]
could be paired with a command to register a host
promgen register-host <project> <host>

though there's probably a few minor things regarding farms that I haven't thought through yet.

Once Promgen had commands for that, it should be (reasonably) easy to call those command from your ansible scripts

[kfdm]

I wrote #192 as an example command to handle the first half, I will add the next command shortly. There's a possibility I may want to rename the commands slightly in the future but hopefully that would not be a major issue

[saulp]

Thanks Paul! This is appreciated. Looking forward to the next command.

[kfdm]

Added a register-host command in #193 so hopefully that will take care of that part.