Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Modal #237

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add support for Modal #237

wants to merge 1 commit into from

Conversation

azliu0
Copy link

@azliu0 azliu0 commented Oct 23, 2024

What does this implement/fix? Explain your changes.

Adds support for running the evaluation harness on Modal Labs.

Adding a --modal flag to the normal run command will run the instances remotely rather than locally. The current images resolves ~290 tasks on Lite.

Benefits of running this way:

  • users don't have to set up their own infrastructure, everything runs in the cloud ☁️
  • everything runs in parallel, so building and running larger numbers of instances is effortless. Lite builds and runs from scratch in 20-30min and should work out of the box
  • Modal's image caching is built-in and follows essentially the same layered abstraction as the images already in the repo

The only downside for now is that the images are slightly less reliable/well tested than the local images.

@wenting-zhao
Copy link
Contributor

I am curious if Modal has the plan to fix the remaining 10/300?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants