glob("*") does not support matching non-utf8 filenames

[lilyball]

glob::glob() does not have any support right now for matching non-utf8 filenames. Not only are its patterns restricted to strings, but it also explicitly skips any non-utf8 filenames it encounters (which should at least be able to match a * pattern).

Tasks that need to be done:

• glob() needs to accept both strings and byte-vectors. It can do this using std::path::BytesContainer
• glob() needs to process its pattern as a byte vector instead of a string, which will allow it to process filenames as byte vectors. This includes matching non-utf8 filenames against * and ? tokens (for the latter, matching a single byte is appropriate; ideally, it would match however many bytes are supposed to be consumed to create a U+FFFD REPLACEMENT CHARACTER as per the unicode standard)

This is a sub-task of #9639.

[lilyball]

An alternative approach is to wait until std::str::from_utf8() has the capability of replacing invalid byte sequences with U+FFFD REPLACEMENT CHARACTER, then simply using that to match against the string pattern. This is deficient in two ways:

1. You cannot specify a pattern that intentionally wants to match against a particular non-utf8 sequence, and
2. Patterns that embed literal U+FFFD REPLACEMENT CHARACTERs should not match arbitrary non-utf8 sequences.

For this reason, the approach outlined in the issue description is recommended.

[lilyball]

Test case from @flaper87:

#[test]
#[cfg(not(windows))]
fn test_non_utf8_glob() {
    let dir = tempfile::TempDir::new("").unwrap();
    let p = dir.path().join(&[0xFFu8]);
    fs::mkdir(&p, S_IRWXU as u32);

    let pat = p.with_filename("*");
    assert_eq!(glob(pat.as_str().expect("tmpdir is not utf-8")).collect::<~[Path]>(), ~[p])
}

This also needs to be disabled on OS X, although perhaps we should do the opposite and simply enable it for linux.

[flaper87]

@kballard Thanks for putting this together. As discussed on IRC, I'm setting the mentor tag on you 😄

[flaper87]

@kballard I'll work on this

[alexcrichton]

cc @nick29581, could this move to rust-lang/globs?

[rust-highfive]

This issue has been moved to the RFCs repo: rust-lang/glob#23

[alexcrichton]

Thanks!