Adding logging and force flushing for run events

[jjanezhang]

Adding logging and force flushing for run events

We are seeing run events inconsistently sent to run metadata for finetuning runs. Adding logs to help debug and force flushing metadata when run event related metadata is being written to the run.

Testing

[j316chuck]

QQ: are these dditional logs generally helpful for future debugging or are they just useful for debugging the issues we are seeing in the current run?

[jjanezhang]

QQ: are these dditional logs generally helpful for future debugging or are they just useful for debugging the issues we are seeing in the current run?

I think if we get logging to occur consistently, then we can remove these.

[j316chuck]

@jjanezhang should we not merge this change then and simply set the integration in the RunConfig when debugging the issue?

- integration_type: git_repo
  git_repo: mosaicml/composer
  git_branch: jane/re-logs
  pip_install: '--upgrade -e .[all]'

Alternatively if you think this improves the mosaicml_logger generally then we should merge it.

[dakinggg]

@j316chuck not a lot of harm in adding debug logs :)

[jjanezhang]

@jjanezhang should we just dev/test on this branch and not merge this then by setting the

- integration_type: git_repo
  git_repo: mosaicml/composer
  git_branch: jane/re-logs
  pip_install: '--upgrade -e .[all]'

I can't add integrations to a finetune yaml so the best I can do is submit a normal run with a ft generated yaml and then add the integrations. If this is really a concern I can just test with the later, but I would prefer to be able to check the logs from a finetune run.

[dakinggg]

Approved with one minor comment

[mvpatel2000]

LGTM -- agree with Daniel to make sure logs look OK and not spammy. Spammy logs = no one reads anything