Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable retries for prefect tasks #111

Closed
sgoodm opened this issue Dec 9, 2022 · 1 comment
Closed

Enable retries for prefect tasks #111

sgoodm opened this issue Dec 9, 2022 · 1 comment

Comments

@sgoodm
Copy link
Member

sgoodm commented Dec 9, 2022

Enable retries for prefect tasks via config args to set number of retries and delay between retries.

In addition, we need to implement a separate error wrapper for prefect tasks which allows prefect to manage task failures until the retry cap is reached (and then generates and error wrapper for logging, etc). See: https://github.com/aiddata/geo-datasets/blob/master/malaria_atlas_project/run_tasks.py#L87

A good point that @jacobwhall brought up is by wrapping prefect tasks we ultimately prevent identifying failed tasks directly via prefect (i.e., the UI) since the final task run always seems to succeed due to the except statement. Not wrapping prefect tasks has implications on the use of prefect futures to generate dependencies across tasks, and would likely require adding code to manage logging for failed prefect tasks.

Using the wrapper for prefect seems like the best short term option, but open to exploring other routes long term.

@jacobwhall
Copy link
Member

Resolved in #119

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants