Object Store Logger Race Condition + EMA Fix

[mvpatel2000]

Fixes CO-1097. Object logger calls join in post_close. At this point, the interpreter has shutdown so new futures cannot be scheduled. So, if the worker for the last checkpoint save has not scheduled the future yet, the upload will error out. To avoid this, we need to wait for all workers to finish after each Trainer function call to ensure all futures are scheduled.

This is a partial fix because anything logged in close by another callback won't (potentially) be logged. I don't think this is possible to support though since once you're in the close phase, you can't create futures

Note that we can't really add a test for this since it's a race condition.

[mvpatel2000]

Update: It looks like there were actually two (!!) bugs going on here. The previous implementation of checkpoint_save_interval was not idempotent, so when EMA was modified last week to move the EMA weights into the default weight slot on a checkpoint event, it broke checkpointing for end of training. Fixed checkpoint_save_interval to be idempotent.

[hanlint]

LGTM, thanks for tracing down this nasty race condition!

[mvpatel2000]

Red button merging because Evan is on vacation.