Abstract matplotlib figure logging from individual logger API

[Haydnspass]

🚀 Feature

Implement default log_figure for all implemented loggers

Motivation

Currently, logging figures relies on calling the API of the individual logger directly. This is not really convenient for a variety of reasons:

• It is cumbersome to change loggers
• It is cumbersome to disable logging (e.g. for debugging) --> if self.logger is not None: \n[...]
• It is not really nice if you have multiple loggers

Pitch

I propose something like
logger.log_figure(figure_name, plt.figure, step, close_figure, kwargs)
where the kwargs are passed on to the respective logger implementation (i.e. if one wants something specific).

Additional context

Should a log_image method also be considered? Should it rather be log_figures (plural, i.e. passing multiple figures)?

[awaelchli]

This is a very difficult feature to implement. There is a great disparity in the API between the loggers regarding the logging of images and graphs and it would be quite the challenge to bring this under a common denominator without sacrificing functionality.

You mention that kwargs could be used to pass down specific arguments to a logger but that will defeat the purpose because changing the logger externally will break the LightningModule as it would today anyway.

One other point one has to consider is that if the third party api changes our wrappers will also break very easily and adds maintenance cost.

[Haydnspass]

I agree that the kwargs option kind of contradicts the purpose of changing loggers and is difficult for multiple loggers, it still solves the temporary disabling logger issue though. But I agree, if one wants more than a default implementation one should possibly ask the user to still us the actual API via logger.experiment.[]

However, I don't quite agree with the API. To me, it is not really different than the differences in API for metrics among the different loggers. Parsing a matplotlib figure is a pretty standard thing which is supported by almost all of the loggers and there's not much up for discussion what to do with the figure is it?

I think figure logging is an important point to debug and monitor the performance of a model and should be well supported.

[awaelchli]

I misunderstood, I didn't know you were only considering "matplotlib" figures.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[Haydnspass]

Can someone help me with MLFlow. How do you usually log figures over multiple epochs there?
My suggestion would be something like

filename = name + f"_step{step}.png"
figure.savefig(filename)
self.experiment.log_artifact(?)

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!