Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raising a better warning if train or eval did not process any data. #3656

Merged
merged 9 commits into from
Oct 17, 2024

Conversation

ethantang-db
Copy link
Contributor

What does this PR do?

Added warnings to the user if a train or eval epoch ended up processing no data. This can happen if somehow the data sampling ends up not feeding a process data or an empty dataset is passed in. This warning gives more visibility to the user that it has happened and that further issues or crashes can happen down the line.

Before submitting

  • [ x] Have you read the contributor guidelines?
  • [ N/A] Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
  • [ N/A] Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
  • [ N/A] Did you update any related docs and document your change?
  • [ x] Did you update any related tests and add any new tests related to your change? (see testing)
  • [ x] Did you run the tests locally to make sure they pass?
  • [ x] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

@ethantang-db ethantang-db requested review from a team as code owners October 17, 2024 04:49
@ethantang-db ethantang-db force-pushed the ethantang-db/no_batch_crash_fix branch from 9360e77 to e08e52c Compare October 17, 2024 04:50
Copy link

@mihir-db mihir-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, minor comments

tests/trainer/test_trainer.py Outdated Show resolved Hide resolved
tests/trainer/test_trainer.py Outdated Show resolved Hide resolved
@ethantang-db ethantang-db force-pushed the ethantang-db/no_batch_crash_fix branch 2 times, most recently from fe69841 to 74eb09e Compare October 17, 2024 17:08
@ethantang-db ethantang-db requested a review from mihir-db October 17, 2024 17:09
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with exception of small nits

pyproject.toml Outdated Show resolved Hide resolved
@ethantang-db ethantang-db force-pushed the ethantang-db/no_batch_crash_fix branch from 74eb09e to 80de437 Compare October 17, 2024 18:37
@ethantang-db ethantang-db merged commit 32caf5b into main Oct 17, 2024
14 checks passed
@ethantang-db ethantang-db deleted the ethantang-db/no_batch_crash_fix branch October 17, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants