Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi Reference ROUGEScore #680

Merged
merged 30 commits into from
Dec 17, 2021
Merged

Multi Reference ROUGEScore #680

merged 30 commits into from
Dec 17, 2021

Conversation

ashutoshml
Copy link
Contributor

@ashutoshml ashutoshml commented Dec 12, 2021

What does this PR do?

Fixes #667

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃
Yes.

@codecov
Copy link

codecov bot commented Dec 12, 2021

Codecov Report

Merging #680 (f6b6ff9) into master (4f8015f) will decrease coverage by 0%.
The diff coverage is 91%.

@@          Coverage Diff          @@
##           master   #680   +/-   ##
=====================================
- Coverage      95%    95%   -0%     
=====================================
  Files         166    166           
  Lines        6379   6413   +34     
=====================================
+ Hits         6074   6103   +29     
- Misses        305    310    +5     

@ashutoshml ashutoshml marked this pull request as draft December 12, 2021 05:20
@ashutoshml ashutoshml changed the title Multirouge [WIP] Multi Reference ROUGEScore Dec 12, 2021
@Borda Borda added the enhancement New feature or request label Dec 12, 2021
@Borda Borda added this to the v0.7 milestone Dec 12, 2021
Copy link
Contributor

@stancld stancld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear @ashutoshml, thank you very much for your contribution. :] I left some minor comments. I will also need to consider averaging methods a bit more, but they're likely okay.

Also, I think you can update CHANGELOG.md with a note that we allow a user to evaluate against multiple references.

torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
@ashutoshml
Copy link
Contributor Author

Based on the current checks, it seems like the build is literally failing because of the absence of "Literal" typing in python<=3.7. Kindly let me know if it needs to be removed.

@stancld
Copy link
Contributor

stancld commented Dec 16, 2021

@ashutoshml Hi, Literal is not a part of typing for python<=3.7. You can, however, import Literal from typing_extensions, which is compatible with all python versions supported by TM.

@ashutoshml
Copy link
Contributor Author

@ashutoshml Hi, Literal is not a part of typing for python<=3.7. You can, however, import Literal from typing_extensions, which is compatible with all python versions supported by TM.

Done. Hopefully, the errors would be resolved now.

tests/text/test_rouge.py Outdated Show resolved Hide resolved
@Borda Borda self-requested a review December 16, 2021 18:32
tests/text/test_rouge.py Outdated Show resolved Hide resolved
tests/text/test_rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
@mergify mergify bot added the ready label Dec 16, 2021
@ashutoshml ashutoshml requested a review from Borda December 17, 2021 08:11
Copy link
Contributor

@stancld stancld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashutoshml Nice, it looks good to me now. Also, thanks a lot for replacing List type annotations for more general Sequence annotations "]

CHANGELOG.md Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
Copy link
Member

@SkafteNicki SkafteNicki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/text/rouge.py Outdated Show resolved Hide resolved
torchmetrics/text/rouge.py Outdated Show resolved Hide resolved
mergify bot and others added 2 commits December 17, 2021 13:12
@ashutoshml
Copy link
Contributor Author

Made the necessary changes from review comments. Not sure why the PyTorchLightning.metrics build is failing though.

@Borda Borda enabled auto-merge (squash) December 17, 2021 14:13
@Borda Borda merged commit 8a8ac9e into Lightning-AI:master Dec 17, 2021
@ashutoshml ashutoshml deleted the multirouge branch December 17, 2021 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable ROUGEScore to evaluate hypotheses against multiple references.
4 participants