[Lllama] Update tokenization code to ensure parsing of the special tokens [core]

[ArthurZucker]

What does this PR do?

Adresses the issues with the fast tokenizer of LLama. Namely:

• nit making it return token type ids.
• the added tokens are not correctly encoded.

There seems to be an issue with the conversion: before the python layer, just loading the tokenizer_config.json file with the rust backend still produced: tokenizer.encode("this is not<s>").tokens, ['<s>', '▁This', '▁is', '▁not', '</', 's', '>']

[ArthurZucker]

Ok, narrowed it down to this line:

        # Check all our special tokens are registered as "no split" token (we don't cut them) and are in the vocab
        added_tokens = tokenizer.sanitize_special_tokens()

When converting the model from a slow one, the tokenizer correctly processes the inputs up until this point. Meaning that before, the special tokens where already registered as special tokens, but adding them once more most probably breaks the internal regex. Still checking but should be this.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

After debugging with @Narsil it seems that the special tokens have to be not normalised, otherwise the normalizer prepends a space when adding it, which is why the token is not recognized. I suspect that there is another bug, as I tried with special tokens set to normalized = True (when calling from_slow=True+commenting self._sanitize_special_tokens) but the current should fix the conversion.

A big discrepancy is that the default AddedTokens imported from tokenizers will set normalized to !special, so if you add tokens as special tokens, normalized will be False. But in transformers this is not the case, which explains why the call to sanitize is a source of problem.

[ArthurZucker]

We have to update the online models to change the tokenizer.json, (people might be confused because the normalized param is also in the slow files but always ignored)

[sgugger]

Thanks for the fix!