Skip to content

Commit

Permalink
Handle white spaces as described in ECMAScript specs (#155)
Browse files Browse the repository at this point in the history
* Handle white spaces as described in ECMAScript specs
  • Loading branch information
evomassiny authored and jasonwilliams committed Oct 19, 2019
1 parent b7afdae commit 3b1074d
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion src/lib/syntax/lexer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,15 @@ impl<'a> Lexer<'a> {
'\r' => {
self.column_number = 0;
}
' ' => (),
// The rust char::is_whitespace function and the ecma standard use different sets
// of characters as whitespaces:
// * Rust uses \p{White_Space},
// * ecma standard uses \{Space_Separator} + \u{0009}, \u{000B}, \u{000C}, \u{FEFF}
//
// Explicit whitespace: see https://tc39.es/ecma262/#table-32
'\u{0020}' | '\u{0009}' | '\u{000B}' | '\u{000C}' | '\u{00A0}' | '\u{FEFF}' |
// Unicode Space_Seperator category (minus \u{0020} and \u{00A0} which are allready stated above)
'\u{1680}' | '\u{2000}'..='\u{200A}' | '\u{202F}' | '\u{205F}' | '\u{3000}' => (),
_ => panic!(
"{}:{}: Unexpected '{}'",
self.line_number, self.column_number, ch
Expand Down

0 comments on commit 3b1074d

Please sign in to comment.