Add default compression, and warning about local paths to dataframe_to_mds

[srowen]

Description of changes:

Set more appropriate defaults for compression in dataframe_to_mds.

In a distributed Spark env, it's probably worth saving I/O with compression, as many data types in MDS files will compress well (tokens). This enables default zstd compression if not specified.

Warn about local-only paths

In Databricks in particular, it's possible to provide a path to remote storage as a FUSE-mounted 'local' path likes /Volumes or /dbfs. It is interpreted as a local path, not remote. This works, but, in practice seems to be suboptimal as MDSWriter will causes lots of writes to the 'local' file. Instead it's probably better and more in line with default logic in CloudUploader to specify a temp dir as the local path, and the FUSE path as remote.

Because this is not necessarily a problem, it generates a warning only, not an error.

Issue #, if available:

None.

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

[XiaohanZhangCMU]

LGTM. Thanks for contributing.

[snarayan21]

lgtm but before approval would like @XiaohanZhangCMU to verify comment

[snarayan21]

@srowen can you run pre-commit run --all-files and push again? seeing linting issues.

[XiaohanZhangCMU]

I guess there are some test failures from test_utils/test_merge_index you need to adjust. The ground truth there are different now.

[srowen]

Yeah I pushed linter fixes, but it is still complaining about the changed file. Hm.

I will look at the test fixes yes

[snarayan21]

lgtm

[srowen]

OK I think it might be ready to go.
Thanks for the attention and patience, I know it was a simple change.
I think this will make a difference to users, getting these right.

[snarayan21]

@srowen of course, thanks for getting this one in! Will definitely make a difference for a lot of users.

Apologies for my comma suggestions haha. Resolved those.