Enable ROUGEScore to evaluate hypotheses against multiple references.

[stancld]

🚀 Feature

Enable ROUGEScore to evaluate hypotheses against multiple references.

Motivation

In the original paper, Lin (2004) proposes the evaluation of a hypothesis against the multiple references, and eventually, the maximum pairwise score is used.

Pitch

Enable ROUGEScore to evaluate hypotheses against multiple references as it is the case for other text metrics.

Alternatives

Leave it as it is.

Additional context

Ideal for #new-contributors

[ashutoshml]

I can take a look at this

[ashutoshml]

I started working on it. The requirements say: "maximum pairwise score is used". Depending on rouge_types - there are 1, 2, L, different highest values, and consequently precision, recall and fmeasure. Which rouge_types' highest value should be used? 1, 2, L and which metric precision, recall and fmeasure?

Also, should we have an avg. version also (instead of just maximum pairwise)? It can be passed as an argument during init.

I can take a look at this

[ashutoshml]

@stancld I have written the code for maximum pairwise.
I wanted to know if we can quickly test just the test_rouge.py using pytest. make test seems to be taking a lot of time.
I'll run the full test once test_rouge.py works.

[stancld]

Hi @ashutoshml, thanks a lot for your effort! O:] I'll have a look tomorrow, but you can run:

pytest tests/text/test_rouge.py

in the project directory to run ROUGE test only :]

[ashutoshml]

Currently, we have

def update(self, preds: Union[str, List[str]], targets: Union[str, List[str]]) -> None:

Should we convert it into

def update(self, preds: Union[str, List[str]], targets: Union[str, List[str], List[List[str]]]) -> None:

?

For handling cases where we have list of predictions and list of list of references for that ?

[stancld]

Hi @ashutoshml, yes I think we may start with something like this. Feel free to open a draft issue once ready and we'll have a proper look :]

[ashutoshml]

@stancld During testing, we compare our score against the scores given by the rouge-score=0.0.4 package. It does not have a multiple-reference version. How do we write the test cases for it i.e., what would be the baseline?

[ashutoshml]

@stancld Opened a draft pull-request. Kindly check