Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle chunking of long lists of job submissions for users #197

Open
jhkennedy opened this issue Mar 15, 2023 · 0 comments
Open

handle chunking of long lists of job submissions for users #197

jhkennedy opened this issue Mar 15, 2023 · 0 comments

Comments

@jhkennedy
Copy link
Contributor

Currently, HyP3 only allows submitting up to 200 jobs at a time.

Most of the SDK submit functions only submit a single job, but vail the prepare methods a long list of prepared jobs can be created and submitted via submit_prepared_jobs.

The SDK provides a helper function to "chunk" submissions into sets of 200 jobs for long submissions:
https://github.com/ASFHyP3/hyp3-sdk/blob/develop/src/hyp3_sdk/util.py#L40
and error handling needs to happen for each submitted chunck.

Therefore, typical user code for submitting a large number of jobs may look like:

import hyp3_sdk as sdk
import hyp3_sdk.util
import hyp3_sdk.exceptions

hyp3 = sdk.HyP3()

jobs = [...]  # A list of jobs > 200 in length

submitted_jobs = sdk.Batch()
failed_to_submit_jobs = []
for batch in sdk.util.chunk(jobs):
    try:
        submitted_jobs += hyp3.submit_prepared_jobs(batch)
    except sdk.exeptions.HyP3SDKError:
        failed_to_submit_jobs.extend(batch)
        print('Some failure message')

It would be nice to handle the chunking for users directly in the submit_prepared_jobs function. We have yet to do so up to this point because error handling is problematic in this space (e.g., do we return a list of failed jobs too?) and we've not determined a good way to handle failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant