Unhelpful Error Messages When Trying to Compile UTF16 Files

[JakobDegen]

Saving a Hello World program as UTF-16:

fn main(){
    println!("Hello World!");
}

and trying to compile it causes about as many errors as there are characters in the file complaining of unknown start of token: \u{0}. Instead, using some heuristics to determine that the file is saved as UTF16 and printing a more helpful error message would be much friendlier to new users who are most likely to run into this issue.

Meta

rustc --version --verbose:

rustc 1.46.0-nightly (50fc24d8a 2020-06-25)
binary: rustc
commit-hash: 50fc24d8a172a853b5dfe40702d6550e3b8562ba
commit-date: 2020-06-25
host: x86_64-unknown-linux-gnu
release: 1.46.0-nightly
LLVM version: 10.0

[Stupremee]

@rustbot claim

[Stupremee]

There's still a question. Should rustc throw an error and exit, or read the file using UTF 16?

[JakobDegen]

@Stupremee I do not think rustc should parse the file as UTF-16, as this would introduce an ambiguity. I opened this issue only to address the poor UX of the current error messages.

[Stupremee]

@rustbot release-assignment
This seems more complicated than expected, because I dont know any reliable way to check if the file is utf-16.

[tesuji]

Maybe add a help message when output error "unknown start of token: \u{0}".