Add train finished run event

[jjanezhang]

Add train finished run event

• Adds timestamp when model is finished training to the run metadata
• Will create a new TRAIN_FINISHED event in MAPI

Testing

1. Added an assert in the unit test to make sure train finished is logged.
2. Confirmed that train finished time is added to run metadata with same timestamp as the last batch finishes

[eracah]

The code looks good! But I am curious if we want to log when the actual training finished or when everything is finished and about to shutdown. The reason is that as the code stands now, the train_finish_time wouldn't get logged until the last checkpoint was sucessfully uploaded to the cloud, which can take an extra 30min-1hour after the last batch_end depending on the size of the checkpoint.

[eracah]

The code looks good! But I am curious if we want to log when the actual training finished or when everything is finished and about to shutdown. The reason is that as the code stands now, the train_finish_time wouldn't get logged until the last checkpoint was sucessfully uploaded to the cloud, which can take an extra 30min-1hour after the last batch_end depending on the size of the checkpoint.

Perhaps running this log event on the last batch_end or making sure MosaicMLLogger comes before RemoteUploaderDownloader in the list of loggers would ensure we log after training is done, but before all checkpoints are uploaded. @dakinggg, wdyt?

[dakinggg]

Yeah @eracah I think we want to log this before the wait for checkpoint upload. Is it simplest to make sure the mosaic logger runs before the RUD?

[eracah]

Yeah @eracah I think we want to log this before the wait for checkpoint upload. Is it simplest to make sure the mosaic logger runs before the RUD?

Yup I think so! And any other RUDs that callbacks create (but I think those will always be added to the end of the list of callbacks)

[eracah]

LGTM. Thanks, @jjanezhang !

[eracah]

LGTM. Thanks, @jjanezhang !

Maybe add a comment describing why RUD has to be last?