Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove binary state from high-level API and use Jinja templates #3147

Merged
merged 80 commits into from
Nov 25, 2024
Merged

Remove binary state from high-level API and use Jinja templates #3147

merged 80 commits into from
Nov 25, 2024

Conversation

cebtenzzre
Copy link
Member

@cebtenzzre cebtenzzre commented Oct 28, 2024 •
edited
Loading

This PR replaces GPT4All's templates that use QString.arg to format a user/assistant pair, with a HuggingFace-style Jinja template that formats the entire conversation.

This allows more fine-grained control of how the extra data (such as LocalDocs context) is passed to the model (for {# version 1#} templates).

@manyoso manyoso force-pushed the remove-binarystate branch from 19e772f to 5c3d83d Compare November 2, 2024 17:21
cebtenzzre and others added 27 commits November 7, 2024 10:43
@cebtenzzre
initial implementation of Jinja templates
840673b 
- Python bindings use `jinja2`
- server.cpp is not implemented
- chatapi.cpp is not implemented

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
add back trimming of trailing whitespace
78c214f 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
separate applyJinjaTemplate from chat model
2a273eb 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
implement chat naming and follow-up questions
65db824 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
fix chat save/restore
904adcf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
implement the regenerate button
0e84395 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
add TODO comment
e4cdd17 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
fix off-by-one error
dac23a5 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
implement system prompt and fix equality check
5365b79 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@manyoso @cebtenzzre
gcc 12 apparently does not include the cend method for std::span.
70da368 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso @cebtenzzre
Remove extra information from jinja.
4b6044b 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso @cebtenzzre
Revert "Remove extra information from jinja."
2fa8f82 
This reverts commit 15e8fba.
@manyoso @cebtenzzre
Update the sources in chatllm directly and make it thread safe to do so.
d7fa1c7 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso @cebtenzzre
Add a fixme for a potential crasher.
aeb3aa1 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@cebtenzzre
remove unused EOS token from params, fix hardcoded BOS token
5eb10c2 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
jinja helpers: fix key casing
91f7988 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llamamodel: restore add_special=true
a2507da 
We still need this for models that don't include bos_token in their chat
template. Llama 3.1 8B Instruct sets this to false.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
get the local server working again
7c154d2 
Importantly, the non-chat completions endpoint (`/v1/completions`) no
longer uses a system prompt or LocalDocs, as those are not applicable.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
tests: fix the expected output of the non-completions test
9e6a82f 
Before this PR, GPT4All inserted a system message into non-chat
completions, and it attempted to use LocalDocs with them. It no longer
does either of these things because they do not make sense here.

This changes the output slightly, so the test needs to be updated.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
tests: simplify comparison
aacf1c8 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
server: fix missing 'else'
ccd91b6 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llamamodel: remove debugging offset
56e7140 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
do not generate name or follow-up questions for server chats
0dbb486 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
remove views::enumerate for GCC 12 compat
3265025 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
modellist: use blank default template and system prompt
4f651dd 
Alpaca is obsolete. We would be much better off defaulting to the chat
template that is shipped with the model.

The default system prompt exists in order to demonstrate how it should
be templated with Alpaca, but this is no longer applicable because this
is now part of the Jinja template.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
chatapi: implement Jinja output parsing
f462b6a 
As a bonus, system prompts can now be used with remote models.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
chatllm: fix prompt error handling
402e1b2 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: add compatibility with new models3.json
23aff08 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
qml: s/framing/control/
9a2f327 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
WIP: interaction buttons on chat messages
89ae98e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
implement redo of earlier responses
689b056 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
implement editing of aribitrary prompts
b40ec78 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
ask for confirmation before using edit to clear most of chat
7835979 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
also ask for confirmation when resetting chat
3e9f1bf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
docs: fix bulleted lists
c1340de 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
docs: improve explanation
8c75848 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
replace the pencil (edit) icon
74b0e81 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
do not store peerIndex
baf0f5f 
The new tool calling PR shows that we can achieve the desired UI with
the same adjacent prompt/response pairs that we have now, by having
child items instead of items in between.

This assumption allows us to simplify the on-disk state by always
computing the peer index lazily.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@manyoso
Copy link
Collaborator

manyoso commented Nov 23, 2024 •
edited
Loading

image

This is what I see when installing llama 3.2 models

And when I click on the documentation I see

image

Also, it doesn't build for me by default because you're using 'cbegin' and 'cend' in chatllm.cpp for the std::span

manyoso
manyoso requested changes Nov 23, 2024
View reviewed changes
gpt4all-chat/src/chatllm.cpp Outdated Show resolved Hide resolved
@cebtenzzre
fix spelling errors
f1c966f 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
fix GCC 12 compat
76ae72e 
This subrange is already const, so C++23 cbegin/cend isn't necessary.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
override Llama 3.2 templates
87e8872 
The 'set tools = none' line is not recognized by Jinja2Cpp.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
Copy link
Member Author

cebtenzzre commented Nov 23, 2024 •
edited
Loading

And when I click on the documentation I see [image: broken page]

You won't see the new documentation page until this PR is merged. If you run the doc site locally with mkdocs serve, you will see that the Chat Templates page is at the expected URL.

@cebtenzzre cebtenzzre requested a review from manyoso November 23, 2024 19:46
@manyoso
Fix chat template for llama 3.2 models.
31ef065 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso
Another small fix for llama 3.2 chat templates.
0be296c 
Signed-off-by: Adam Treat <treat.adam@gmail.com>
@manyoso manyoso merged commit 225bf6b into main Nov 25, 2024
3 of 18 checks passed
@cebtenzzre cebtenzzre mentioned this pull request Dec 2, 2024
Merged
cebtenzzre added a commit that referenced this pull request Dec 5, 2024
@cebtenzzre
add changelog entries for #3147
7a46549 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre mentioned this pull request Dec 5, 2024
Merged
cebtenzzre added a commit that referenced this pull request Dec 6, 2024
@cebtenzzre
changelog: add more changes from #3147
062fb79 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
manyoso pushed a commit that referenced this pull request Dec 6, 2024
@cebtenzzre
changelog: add more changes from #3147 (#3226)
6b18abb 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
This was referenced Dec 10, 2024
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@manyoso manyoso manyoso approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cebtenzzre @manyoso