Cannot use f1/recall/precision arguments in CombinedEvaluations.compute

[fcakyon]

This works:

metric=evaluate.load('f1')
metric.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], average=None)

This won't work:

metric=evaluate.combine(["f1"])
metric.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], average=None)

Reason:
average is not included in f1 score features:

evaluate/metrics/f1/f1.py

Line 112 in eaf34a7

features=datasets.Features(

and CombinedEvaluations.compute ignore if argument is not included in features:

evaluate/src/evaluate/module.py

Line 858 in eaf34a7

batch = {input_name: batch[input_name] for input_name in evaluation_module._feature_names()}

Is this expected or a bug?

[fcakyon]

@lvwerra do you have any opinion on that?

[lvwerra]

Yes, this is a current limitation of combine: you can't pass any settings to compute only the features. Rather than fixing this in combine we aim to enable changing the settings when the metric is loaded in #169. This should be coming in the next few weeks.

[m-movahhedinia]

@lvwerra Do you mind if I ask whether you have an estimate on when this issue will be closed? I reviewed #188 and noticed that it seems to be passing for pip release. Is there a chance we can git it in the next couple of days?

[falcaopetri]

@lvwerra I also need to pass custom kwargs to my metrics. My current workaround is to overwrite CombinedEvaluations .compute in a child class and remove

evaluate/src/evaluate/module.py

Line 858 in eaf34a7

batch = {input_name: batch[input_name] for input_name in evaluation_module._feature_names()}

This works in my specific use case because all my (custom) metrics accept **kwargs in their compute method, which means they will just ignore extra args.

[ernestchu]

Has this been resolved?

[lvwerra]

Not yet. There is an issue with the sync mechanism between the hub and the library which is why we had to roll back #169. Merging it will break pre 0.3.0 installs so we need to wait for sufficient adaptation.