Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/lambada evaluator #1845

Merged
merged 6 commits into from
Jan 9, 2023

Conversation

bmosaicml
Copy link
Contributor

@bmosaicml bmosaicml commented Dec 22, 2022

What does this PR do?

This PR implements a composer-native evaluator for Lambada.

What issue(s) does this change relate to?

Previously we were dependent on the Eleuther eval harness to run Eval, which was implemented inefficiently and couldn't support multi GPU. We now support LAMBADA with multi-gpu and enjoy linear scaling w/ num GPUs.

This is related to the following JIRA https://mosaicml.atlassian.net/browse/RESEARCH-345

Before submitting

  • Have you read the contributor guidelines?
  • Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
  • Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
  • Did you update any related docs and document your change?
  • Did you update any related tests and add any new tests related to your change? (see testing)
  • Did you run the tests locally to make sure they pass?
  • Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from 13838c1 to 6a3de37 Compare December 22, 2022 23:55
@bmosaicml bmosaicml marked this pull request as ready for review December 25, 2022 22:15
@bmosaicml bmosaicml requested review from a team, knighton and karan6181 as code owners December 25, 2022 22:15
Copy link
Contributor

@dakinggg dakinggg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a bunch of comments, mostly about integrating everything nicely with composer, and other clean code stuff like tests and docstrings. The implementation itself looks correct to me. Happy to help with any of the comments related to integrating it with composer if you want!

composer/models/huggingface.py Outdated Show resolved Hide resolved
composer/models/huggingface.py Outdated Show resolved Hide resolved
composer/models/huggingface.py Outdated Show resolved Hide resolved
composer/trainer/trainer.py Outdated Show resolved Hide resolved
composer/trainer/trainer.py Outdated Show resolved Hide resolved
composer/datasets/in_context_learning_evaluation.py Outdated Show resolved Hide resolved
composer/datasets/in_context_learning_evaluation.py Outdated Show resolved Hide resolved
@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from 6a3de37 to 94f5ecb Compare January 2, 2023 21:36
@bmosaicml bmosaicml requested a review from dakinggg January 2, 2023 21:55
@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from 94f5ecb to 9744e9f Compare January 3, 2023 18:41
tests/metrics/test_nlp_metrics.py Outdated Show resolved Hide resolved
tests/metrics/test_nlp_metrics.py Outdated Show resolved Hide resolved
tests/metrics/test_nlp_metrics.py Show resolved Hide resolved
composer/metrics/nlp.py Outdated Show resolved Hide resolved
composer/metrics/nlp.py Outdated Show resolved Hide resolved
composer/datasets/in_context_learning_evaluation.py Outdated Show resolved Hide resolved
tests/metrics/test_nlp_metrics.py Outdated Show resolved Hide resolved
composer/models/huggingface.py Outdated Show resolved Hide resolved
@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from 9744e9f to 76e9d37 Compare January 9, 2023 20:24
@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from 76e9d37 to e934acc Compare January 9, 2023 20:46
@bmosaicml bmosaicml force-pushed the feature/lambada_evaluator branch from e934acc to 677228e Compare January 9, 2023 20:59
@bmosaicml bmosaicml merged commit 62bf8ba into mosaicml:dev Jan 9, 2023
@bmosaicml bmosaicml deleted the feature/lambada_evaluator branch January 9, 2023 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants