Default Chat Template Format Error (and Fix).

[Phil209]

Bug Report

There are chat template formatting issue(s) for various models. I'll use Mistral Small Instruct as an example here.

Steps to Reproduce

1. Load Mistral Small Instruct and reset the chat template.
2. An immediate error in the chat template will appear in red (see image).
3. Can fix by removing the colon after messages 1.
4. An additional issue is until the colon is removed all other attempted changes to the chat template aren't applied, nor saved to GPT4All.ini, making it harder to track down formatting issues.

Expected Behavior

No errors in the default chat template, and any edits to the chat template will be applied and saved, even if there's an error (in this case an extra colon).

Your Environment

• GPT4All version: 3.5.1
• Operating System: Windows 11

[Phil209]

Here's a screenshot of the Gemma 2 27b error, followed by the fix ChatGPT gave me.

Edit: Here it is as text so it can be copied and pasted.

{{ bos_token }}
{% if messages[0]['role'] == 'system' %}
{{ raise_exception('System role not supported') }}
{% endif %}
{% for message in messages %}
{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
{{ raise_exception('Conversation roles must alternate user/assistant/user/...') }}
{% endif %}
{% if message['role'] == 'assistant' %}
{% set role = 'model' %}
{% else %}
{% set role = message['role'] %}
{% endif %}
<start_of_turn>{{ role }}
{{ message['content'] | trim }}<end_of_turn>
{% endfor %}
{% if add_generation_prompt %}
<start_of_turn>model
{% endif %}

[Phil209]

Even the default chat templates for the most downloaded LLMs ever published to HF, including Llama 3.1 8/70b Instruct, have immediate errors out of the box preventing them from working (and commonly more than one error).

Is there any way AI can be used to scan the chat templates? AI can at least ensure that the chat templates are formatted correctly so they're easier to read (e.g. not just one long line of code), plus AI can detect obvious errors like using apostrophes to comment out lines of code (as seen in the second example posted above).

I just don't think expecting non-technical users to produce a working Jinja template for every model, or fix one with errors, is reasonable. And some of the chat templates are absurdly long and complex (example pasted below). Perhaps an advance prompt template optional button can be added, but by default a simple chat template is applied (i.e. system prompt, then alternative between user and assistant).

I mean what is an average windows user going to do with the following default chat template for Llama 3.1 8b Instruct, especially when it errors out by default and requires multiple changes to even use the model?

{{- bos_token }}
{%- if custom_tools is defined %}
{%- set tools = custom_tools %}
{%- endif %}
{%- if not tools_in_user_message is defined %}
{%- set tools_in_user_message = true %}
{%- endif %}
{%- if not date_string is defined %}
{%- set date_string = "26 Jul 2024" %}
{%- endif %}

{#- This block extracts the system message, so we can slot it into the right place. #}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}

{#- System message + builtin tools #}
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
{%- if builtin_tools is defined or tools is not none %}
{{- "Environment: ipython\n" }}
{%- endif %}
{%- if builtin_tools is defined %}
{{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
{%- endif %}
{{- "Cutting Knowledge Date: December 2023\n" }}
{{- "Today Date: " + date_string + "\n\n" }}
{%- if tools is not none and not tools_in_user_message %}
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{%- endif %}
{{- system_message }}
{{- "<|eot_id|>" }}

{#- Custom tools are passed in a user message with some extra guidance #}
{%- if tools_in_user_message and not tools is none %}
{#- Extract the first user message so we can plug it in here #}
{%- if messages | length != 0 %}
{%- set first_user_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
{%- endif %}
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
{{- "Given the following functions, please respond with a JSON for a function call " }}
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{{- first_user_message + "<|eot_id|>"}}
{%- endif %}

{%- for message in messages %}
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
{%- elif 'tool_calls' in message %}
{%- if not message.tool_calls|length == 1 %}
{{- raise_exception("This model only supports single tool-calls at once!") }}
{%- endif %}
{%- set tool_call = message.tool_calls[0].function %}
{%- if builtin_tools is defined and tool_call.name in builtin_tools %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- "<|python_tag|>" + tool_call.name + ".call(" }}
{%- for arg_name, arg_val in tool_call.arguments | items %}
{{- arg_name + '="' + arg_val + '"' }}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- ")" }}
{%- else %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- '{"name": "' + tool_call.name + '", ' }}
{{- '"parameters": ' }}
{{- tool_call.arguments | tojson }}
{{- "}" }}
{%- endif %}
{%- if builtin_tools is defined %}
{#- This means we're in ipython mode #}
{{- "<|eom_id|>" }}
{%- else %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- elif message.role == "tool" or message.role == "ipython" %}
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
{%- if message.content is mapping or message.content is iterable %}
{{- message.content | tojson }}
{%- else %}
{{- message.content }}
{%- endif %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

[SINAPSA-IC]

"
I just don't think expecting non-technical users to produce a working Jinja template for every model, or fix one with errors, is reasonable. And some of the chat templates are absurdly long and complex (example pasted below). Perhaps an advance prompt template optional button can be added, but by default a simple chat template is applied (i.e. system prompt, then alternative between user and assistant).

I mean what is an average windows user going to do with the following default chat template[...]
"

^this.
User-centric, user-oriented program design, UX first and foremost. The 101 of programming for Common Users aka Your Program's Users aka The User Base, not for programmers.

A major change Needed=Wanted by maybe 5% of the users, who are either nerds, or programmers who are ok with CopyPasting such litanies from the GUI to their code where they test these preposterous PTs, or the developers themselves.

[brankoradovanovic-mcom]

Let me first document two more models where I've encountered errors in the out-of-the-box chat templates in 3.5.1:

• Llama-3.2-3B-Instruct
• qwen2.5-coder-7b-instruct (oddly - or not, it's hard to tell - coder-14b and coder-32b are fine)

(I won't bother with the screenshots, I can post them if anyone is interested.)

Chat templates were always a bit of a black art, and it is one area I can genuinely say I never fully understood. This has just gone to a completely different level, as debugging (or even understanding, for that matter) these templates has to be an order of magnitude more difficult now.

I've upgraded to 3.5.1 on my laptop, but I don't think I'll be upgrading my desktop to 3.5.x for the time being because I have 30+ models there, and chances are many of them will be affected. It's unclear to me right now what sort of effort will be needed on my part to get them in order, and this will be using Python and transformers (as documented in the chat templates help page), which is the option many users don't realistically have. So, the situation is indeed unfortunate.

[Phil209]

@SINAPSA-IC The benefits of the new Jinja chat template are pronounced, allowing far more control over the AI models, so I'm glad the change was made. But you're right that common/regular users like me need something that works reliably out of the box without becoming proficient in Jinja.

Hopefully in time the default templates work reliably so the app functions for non-technical users, while also giving advanced users more control.

[brankoradovanovic-mcom]

@SINAPSA-IC The benefits of the new Jinja chat template are pronounced, allowing far more control over the AI models, so I'm glad the change was made. But you're right that common/regular users like me need something that works reliably out of the box without becoming proficient in Jinja.

Hopefully in time the default templates work reliably so the app functions for non-technical users, while also giving advanced users more control.

Agree fully. I haven't taken a closer look, but Jinja templates seem to be function calling-capable, and that would be a big thing.

Let me, however, also note this: per instructions, if the sideloaded model also happens to be a gated model on HuggingFace, one needs to have a HF account and request and (hopefully) receive access to the repo to get the template. I have a HF account, but thus far I didn't need to worry about access at all because GGUFs are not gated, of course. My worry is that what was once super simple has maybe become fairly onerous.

[SINAPSA-IC]

Fully agree with the above users. However, all in all, in short, the simple Users do not need=want something that programmers only are interested in and can handle. This current version is for programmers, developers, coders. Make a version for the developers and one for the simple Users. This versions fully shows the concept of Making A Program For Yourself.

[manyoso]

Fully agree with the above users. However, all in all, in short, the simple Users do not need=want something that programmers only are interested in and can handle. This current version is for programmers, developers, coders. Make a version for the developers and one for the simple Users. This versions fully shows the concept of Making A Program For Yourself.

Our non-technical uses can and do download the curated models we publish and recommend that work out of the box. The models that come from huggingface are not guaranteed to work out of the box and never have been. Same goes for sideloaded models.

"Common/regular" users aren't searching for models on huggingface and/or side loading them. Those that do either work in the tech/ml industry or are the more sophisticated tech users who can/should be able to deal with the difficulties.

The two contributors you are talking about and are labeled as dictators are responsible for more than 98% of this completely free and open source application you are receiving at no cost and with no warranty. The decision to switch to these new chat templates was made after considerable reflection and tremendous effort and attention was paid to making it as easy and painless as possible. It is not perfect and we are now in the process of working very hard to fix the problems that are being experienced.

It doesn't help or motivate at all though when users of this free and open source application hurl invective and abuse on those who are trying their best to help and provide you with free software.

Users who continue to pile on invective and abuse in lieu of trying to civilly help us improve the application are simply not wanted and won't be tolerated. No one is forcing anyone to use the app. If you're encountering a problem then by all means help us fix it in a constructive fashion. But stop with the abuse.

[manyoso]

@SINAPSA-IC The benefits of the new Jinja chat template are pronounced, allowing far more control over the AI models, so I'm glad the change was made. But you're right that common/regular users like me need something that works reliably out of the box without becoming proficient in Jinja.

Hopefully in time the default templates work reliably so the app functions for non-technical users, while also giving advanced users more control.

Yes, there are many advanced features that this change to chat templates will unlock. It is a painful change and the pain was anticipated but we believe the benefits outweigh the cons.

The models that you should expect to work out of the box are the ones that are curated by us and are at the top of the explore models page. Those that are downloaded via huggingface and or sideloaded should not have such an expectation of guarantee.

Thank you for understanding and the constructive feedback :)