Mistral template is wrong

[normster]

I don't think the mistral conversation template is implemented correctly:

>>> from fastchat.conversation import get_conv_template
>>> t = get_conv_template('mistral')
>>> t.append_message(t.roles[0], 'Hello, how are you?')
>>> t.append_message(t.roles[1], 'Doing well, thanks')
>>> t.append_message(t.roles[0], 'Great to hear')
>>> t.get_prompt()
'[INST] Hello, how are you?  [/INST] Doing well, thanks </s>[INST]  Great to hear'

But according to the official docs the prompt format should be:

<s>[INST] Instruction [/INST] Model answer</s>[INST] Follow-up instruction [/INST]

i.e. your implementation adds an extra space before the first [/INST], and before </s>, and after the second [INST].

[merrymercy]

Could you send a fix?

[normster]

Sure: #2529

[irexyc]

@normster

Hi, according to this page: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", use_default_system_prompt=False)
messages = [
     {"role": "user", "content": "A"},
     {"role": "assistant", "content": "B"},
     {"role": "user", "content": "C"},
     {"role": "assistant", "content": "D"},
     {"role": "user", "content": "E"},
     {"role": "assistant", "content": "F"},
     {"role": "user", "content": "G"},
]
encodeds = tokenizer.apply_chat_template(messages, tokenize=False)
print(encodeds)

# <s>[INST] A [/INST] B </s><s>[INST] C [/INST] D </s><s>[INST] E [/INST] F </s><s>[INST] G [/INST]

It seems the sep2 should be ' </s><s>' and the ret seems lacking the leading <s>

[normster]

The output there seems to be in llama2 format, but you can see in this section the separator should be </s> with no space: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1#instruction-format

…

On Mon, Oct 9, 2023 at 1:28 AM Chen Xin ***@***.***> wrote: @normster <https://github.com/normster> Hi, according to this page: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 from transformers import AutoModelForCausalLM, AutoTokenizertokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", use_default_system_prompt=False)messages = [ {"role": "user", "content": "A"}, {"role": "assistant", "content": "B"}, {"role": "user", "content": "C"}, {"role": "assistant", "content": "D"}, {"role": "user", "content": "E"}, {"role": "assistant", "content": "F"}, {"role": "user", "content": "G"}, ]encodeds = tokenizer.apply_chat_template(messages, tokenize=False)print(encodeds) # <s>[INST] A [/INST] B </s><s>[INST] C [/INST] D </s><s>[INST] E [/INST] F </s><s>[INST] G [/INST] It seems the sep2 should be ' </s><s>' and the ret seems lacking the leading <s> — Reply to this email directly, view it on GitHub <#2520 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTAZJSK3XL5BMZBLJL6WODX6OYUDAVCNFSM6AAAAAA5VPLT7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJSGU2TGNBSGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[irexyc]

They delete the chat template when I run the code. But I found they added it back.
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/commits/main

With current chat template,

"chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}",

the output is
'<s>[INST] A [/INST]B</s> [INST] C [/INST]D</s> [INST] E [/INST]F</s> [INST] G [/INST]'

While with FastChat and #2529, the output is
[INST] A [/INST] B</s>[INST] C [/INST] D</s>[INST] E [/INST] F</s>[INST] G [/INST]

Should sep2 be set as "</s> " ? Moreover, there is an extra space before assistant output (B D F) and the final ret lack the leading <s>.

Is this difference important, or can we ignore it? I'm not familiar with language model.