tokenizer doesn't handle numeric literals if they start with 0

[Y-Nak]

What is wrong?

Tokenizer doesn't tokenize correctly if numeric literals start with 0 followed by non zero value.
For example,

02

is tokenized into

    [
        Token {
            typ: NUMBER,
            string: "0",
            span: Span {
                start: 0,
                end: 1,
            },
            line: "02\n",
        },
        Token {
            typ: NUMBER,
            string: "2",
            span: Span {
                start: 1,
                end: 2,
            },
            line: "02\n",
        },
        Token {
            typ: NEWLINE,
            string: "\n",
            span: Span {
                start: 2,
                end: 3,
            },
            line: "02\n",
        },
        Token {
            typ: ENDMARKER,
            string: "",
            span: Span {
                start: 3,
                end: 3,
            },
            line: "",
        },
    ],

Also, it would be better to discuss spec of number literals(sorry if it has been discussed already).
i.e. currently, tokenizer allows hex/bin/octal/decimal numbers while analyzer allows only decimal numbers. This seems inconsistent between them.

EDIT: opened #333 for further discussion of numeric literal representations.

How can it be fixed

It would be fixed by tweaking the code below

fe/parser/src/tokenizer/regex.rs

Lines 18 to 21 in 3a7fa34

	pub const HEXNUMBER: &str = r"0[xX](?:_?[0-9a-fA-F])+";
	pub const BINNUMBER: &str = r"0[bB](?:_?[01])+";
	pub const OCTNUMBER: &str = r"0[oO](?:_?[0-7])+";
	pub const DECNUMBER: &str = r"(?:0(?:_?0)*|[1-9](?:_?[0-9])*)";

.