improve exponential backoff while job hunting

[terrazoon]

Description

1. Shorten exponential back-off times so that we don't time out just because we're doing back offs
2. Do an initial check to see if it even looks like the job is in the bucket before we start any back offs
3. Use global variable/singleton so we aren't endless recreating boto resources.

Security Considerations

N/A

[ccostino]

Thanks, @terrazoon!

I'm not quite sure how I feel about the global but I do see what you're doing with reusing resources and such, and saw the commented out line in the test too. For the sake of trying to nail this down since it's an active production issue, I'm good to go with this as it is though. 🙂

[A-Shumway42]

LGTM! Fingers crossed!