UTF16 and possibly other UTF encodings support

[phiresky]

is currently broken. At least UTF16LE with BOM should be auto detected and correctly parsed like in normal ripgrep. Caused by my own binary file detection

ripgrep-all/src/adapters/spawning.rs

Line 23 in 5901cdc

if fourk.contains(&0u8) {

Probably need to ask encoding_rs about encoding before checking for null bytes?

[phiresky]

Other encodings apart from UTF8 and UTF16 with BOM are not automatically detected by ripgrep either, though they can be parsed using --encoding=xyz. Maybe useful reference: BurntSushi/ripgrep#1

[phiresky]

Decoding should now be on parity with ripgrep: 29b8f1d

There is the chardet crate that should be able to detect more types of encodings, but I don't think I care enough about other encodings right now to integrate it.

Also, it's currently not possible to disable BOM parsing / transcoding, pending BurntSushi/ripgrep#1305