You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, HyP3 only allows submitting up to 200 jobs at a time.
Most of the SDK submit functions only submit a single job, but vail the prepare methods a long list of prepared jobs can be created and submitted via submit_prepared_jobs.
Therefore, typical user code for submitting a large number of jobs may look like:
importhyp3_sdkassdkimporthyp3_sdk.utilimporthyp3_sdk.exceptionshyp3=sdk.HyP3()
jobs= [...] # A list of jobs > 200 in lengthsubmitted_jobs=sdk.Batch()
failed_to_submit_jobs= []
forbatchinsdk.util.chunk(jobs):
try:
submitted_jobs+=hyp3.submit_prepared_jobs(batch)
exceptsdk.exeptions.HyP3SDKError:
failed_to_submit_jobs.extend(batch)
print('Some failure message')
It would be nice to handle the chunking for users directly in the submit_prepared_jobs function. We have yet to do so up to this point because error handling is problematic in this space (e.g., do we return a list of failed jobs too?) and we've not determined a good way to handle failures.
The text was updated successfully, but these errors were encountered:
Currently, HyP3 only allows submitting up to 200 jobs at a time.
Most of the SDK submit functions only submit a single job, but vail the prepare methods a long list of prepared jobs can be created and submitted via
submit_prepared_jobs
.The SDK provides a helper function to "chunk" submissions into sets of 200 jobs for long submissions:
https://github.com/ASFHyP3/hyp3-sdk/blob/develop/src/hyp3_sdk/util.py#L40
and error handling needs to happen for each submitted chunck.
Therefore, typical user code for submitting a large number of jobs may look like:
It would be nice to handle the chunking for users directly in the
submit_prepared_jobs
function. We have yet to do so up to this point because error handling is problematic in this space (e.g., do we return a list of failed jobs too?) and we've not determined a good way to handle failures.The text was updated successfully, but these errors were encountered: