Add test cases for non-utf8 paths

[flaper87]

closes #9406

[flaper87]

@huonw r?

[huonw]

This is still a UTF-8 path: it's just the 0xE5'th codepoint, i.e. two bytes (any and every string in Rust is UTF-8).

I guess .push([0xff]) may work.

[flaper87]

I can't push [0xff] unless it's converted to str which would end up on the same case I submitted. The point of this specific test is to create that directory and then call glob. Current version creates a directory named å under /tmp.

[huonw]

You can push things other than strings, and you have to push something other than a string for this to be a valid test, because otherwise you have a UTF-8 path.

[huonw]

(I guess it may have to be &[0xFF] or it may not be able to find the BytesContainer impl.)

[flaper87]

not ready to be merged yet. The glob test fails! Will work on it later!

[lilyball]

Yay, the glob test fails! That's because it's finally correct ;)

Glob itself doesn't match properly against UTF-8 pathnames. It ignores them, and that's because the code was written to use strings. Line 136 of libglob/lib.rs contains a // FIXME explaining that it's ignoring non-utf8 filenames but that it really should be able to match them against a *.

It's also possible that fs::readdir() doesn't handle utf-8 filenames properly either. I don't know if there are any tests specifically on that functionality. Maybe you should duplicate your test_non_utf8_glob and modify it for fs::readdir (and stick it in the right mod).

[lilyball]

After investigating some more, my issue with fs::readdir() is a red herring. The actual problem I was experiencing is HFS+ doesn't allow non-unicode filenames, so the call to mkdir() was actually creating a folder named "%FF" instead of \xFF.

[flaper87]

Since glob is still broken, I'll abandon this PR and submit one with tests just for fs::

I copied the glob test to the issue as a reference code for testing the fix to come.

[lilyball]

New issue filed for glob() and non-utf8 paths as #11916.