Log file path optional parameter

[oxedions]

Hi,

I am using the ipmi_exporter (works like a charm !) on some huge systems, and sometime when a range of nodes are down (including BMC), I can face thousands of error.

Currently, all logs are sent to system log, which "pollute" it. Is there a simple way to add a login file path to parameters ?

My current systemctl file is:

[Unit]
Description=IPMI Exporter
After=network.target

[Service]
User=ipmiexporter
Group=ipmiexporter
Type=simple
ExecStart=/usr/local/bin/ipmi_exporter \
  --config.file=/etc/ipmi_exporter/ipmi_config.yml

[Install]
WantedBy=multi-user.target

I could find --log.level or --log.format, but nothing related to an optional log file path.

Please consider this not as a critical issue, but only asking if some kind of mechanism exist for this tool to store logs in a dedicated file.

Best regards

Ox

[bitfehler]

Hi and thanks for raising this. I understand your motivation, but I think this won't be easily solved in the exporter itself. Proper logging is serious business, so if we start going down this route, the following questions will come up:

1. Can we add an option to send some logs to stdout, but others to file?
2. The log file becomes to big, can we add an option to rotate it?
3. I only log errors, but sometimes there is too many, can we add rate-limiting?
4. ...

Things will become arbitrarily complex, with every user having a different preference based on their use case. Hence, the best thing to do (in my opinion at least) is to keep the interface simple (write to stdout) and leave everything else to software designed for these things, such as systemd/journald.

Now, I don't just want to brush this off, so let's talk about your specific problem. The solution you are asking for could be easily achieved by having systemd redirect the stdout elsewhere, e.g. using StandardOutput=file:/var/log/ipmi_exporter.log (see the systemd man page).

But then you end up with this log file that might be blowing up over time. As long as you use journald, at least it does some rotating and such for you. So maybe an even better option would be to take a closer look at journald's rate limiting capabilities or log size limitation settings to see if you can tune them to something that suits your needs (note that unfortunately AFAIK journald settings are global and apply to all services)?

Maybe you can elaborate a little. What were the actual problems you observed? Did the amount of logs slow down journald and/or the host? Did they fill the file system? Or were they just a nuisance? If you were to write the logs to a file, what would you do with that file? How would you make sure it doesn't grow too big?

I hope you understand that I am not to keen to add advanced logging options to the exporter. But I am genuinely interested in making this work for you, so I'd appreciate if you let me know if any of the suggestions work for you or else provide a few more details, e.g. answers to the questions above. I am sure this will be useful to others.

[oxedions]

Dear @bitfehler ,

Many thanks for this very detailed answer.
I didn't knew systemd was able to redirect output to a dedicated file in addition to journalctl (I should have guess however, my bad...).

I understand your concerns, and on my side your solution based on StandardOutput is enough.
I will add the logrotate (xz based) on these files. Dealing with logs of multiple gigabytes is common in our field and disks are sized according to this. I just desired the "main" message log to be cleaner. So it was just a nuisance.

Regarding the context:

I am working on an opensource stack (https://bluebanquise.com/, https://github.com/oxedions/bluebanquise), and the stack is used on HPC (High Performance Computing) clusters. We are currently integrating Prometheus monitoring (metrics and alerts).
HPC clusters can have thousands of nodes, and when using the ipmi_exporter in production on a cluster with only 400 nodes, I faced a log spam in /var/log/messages when half of the configuration went down for maintenance.
Note that Prometheus scarping interval was high due to the maintenance context, I should get much less error messages in normal time.

I will let you know once I made the systemctl file modification with StandardOutput if it works as expected.

[oxedions]

Dear @bitfehler,

As promised, a feedback.

Tests were made on an RHEL 7.7.

I couldn't have systemctl to log files, as expected in the documentation.

I used this service file:

[root@]# cat /etc/systemd/system/ipmi_exporter.service
[Unit]
Description=IPMI Exporter
After=network.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/ipmi_exporter \
  --config.file=/etc/ipmi_exporter/ipmi_config.yml
StandardOutput=append:/var/log/ipmi_exporter.log
StandardError=inherit
#StandardError=append:/var/log/ipmi_exporter.log

[Install]
WantedBy=multi-user.target
[root@]#

I tried using both StandardOutput/StandardError, or StandardError only (commented here), as I have seen ipmi exporter output in stderr when using command line. But /var/log/ipmi_exporter.log remains empty.

This is not a bug on your side, and so I will ask RedHat about this.

Will keep you informed once I get an answer.

Best regards
Ox