feat(doctest): SSIM metric doctest

[ydcjeff]

ref #2230

Description:

• Add doctest for SSIM metric
• Add doctest gh job
• Sphinx makes doctest easy by importing modules we need in doctest_global_setup

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[vfdev-5]

Can't it work with code-block ? I'm not a big fan of >>>

[ydcjeff]

It does not work AFAIK. If >>> is not preferred, we need to use .. testcode:: and .. testoutput:: which is more verbose.

[vfdev-5]

With >>> when code is copied it is with these useless >>>. I think it would be better to use other tags instead of >>> which requires each line modification.

[ydcjeff]

sphinx-copybutton can be modified to strip >>> in conf.py.

[sdesrozis]

I agree the @vfdev-5's remark about >>>. Maybe we could explore an alternative and see ?

[ydcjeff]

@vfdev-5 @sdesrozis changed to using .. testcode:: and .. testoutput::. If we are using those, we need to use print to test the expected output.

[sdesrozis]

If we are using those, we need to use print to test the expected output.

Not sure to understand, sorry.

Does the rendering of the doc show the output ?

[ydcjeff]

I mean if we are using .. testcode:: and .. testoutput::, we need to use print like print(state.metrics['ssim']) to test the expected output.

Does the rendering of the doc show the output ?

Yes, it shows.

[sdesrozis]

@ydcjeff I just would like to see the rendering and that’s fine on my side !

[ydcjeff]

@sdesrozis https://deploy-preview-2241--pytorch-ignite-preview.netlify.app/generated/ignite.metrics.ssim

[sdesrozis]

The current rendering

[sdesrozis]

We could hide the output cell adding the option :hide:

.. testoutput::
    :hide:

    0.9218971...

[sdesrozis]

Or an alternative would be to mix doctest and sphinx extension.

.. testcode::

    def process_function(engine, batch):
        y_pred, y = batch
        return y_pred, y
    engine = Engine(process_function)
    metric = SSIM(data_range=1.0)
    metric.attach(engine, 'ssim')
    preds = torch.rand([4, 3, 16, 16])
    target = preds * 0.75
    state = engine.run([[preds, target]])

.. doctest::

    >>> state.metrics['ssim']
    0.9218971...

[sdesrozis]

Each examples above are tested by make doctest. I think a section title would be valuable before the output (Expected output(s) ?).

What do you think ? Everything works fine though !

[ydcjeff]

Mostly python doctest is used alongside with output. So it's better to show output as well maybe with "Output" section if we want.

[sdesrozis]

@ydcjeff Thanks ! Now we can refactor every docstring example.

[sdesrozis]

@vfdev-5 Is it fine for you with this PR ?

[vfdev-5]

@sdesrozis @ydcjeff I'm OK with this change however can you estimate how much work it would require to rework all doc strings and even if this is feasible for all doc strings ? Will we do a mix of tested and non-tested doc strings ?
Please, update #2230 with more details and maybe make the issue as "help wanted"/Hacktoberfest to bring more contributors on that. Thanks!

[ydcjeff]

From the first glance, I think most examples can be tested, but I'm not sure for distributed ones.

[sdesrozis]

First of all, having the metrics and param schedulers tests in the doc validated would be great. Except for idist, I'm not sure we have that many tests distributed in the docstring.

[vfdev-5]

@sdesrozis @ydcjeff I let you open new tracking issue or update existing one to see clearly the task and how we can involve contributors to that. Sylvain, I let you guide this work and merge this PR or wait for something. Thanks!