Local server not remembering previous messages

[cebtenzzre]

From this Discord message:

Possibly an issue with the way n_past is managed, or maybe the effective context length is too small? Or possibly we never even attempted to preserve message history for the server—I haven't looked at the code (or attempted to reproduce this issue, for that matter).

[cosmic-snow]

I'm no expert on all the server internals, but there are two URLs (paths): /v1/completions and /v1/chat/completions. The former is for one-offs.

It's not clear from the screenshot which one was used.

[cosmic-snow]

Or possibly we never even attempted to preserve message history for the server

Now that I've spent some more time on it, I think it's supposed to be stateless, and when using the /v1/chat/completions path, the "messages": [] array is used to keep track of and reintroduce previous exchanges.

Although from what I've seen, I guess the local API server just concatenates these messages instead of handling them properly.

Useful/related:

• OpenAI API docs 'Text generation models': https://platform.openai.com/docs/guides/text-generation
• Python package:
  • Usage: https://github.com/openai/openai-python?tab=readme-ov-file#usage
  • web API spec: https://github.com/openai/openai-openapi/blob/master/openapi.yaml
  • Python API: https://github.com/openai/openai-openapi/blob/master/openapi.yaml

[ThiloteE]

@cosmic-snow
do you see

// don't remember any context
resetContext();

in the code and is it relevant?

[cebtenzzre]

This was fixed in #2929 (v3.3.0) but it was broken again by #3147 (v3.5.0) - oops.