Providing the previous conversation

[kaanaksit]

I hope that I am using the right platform for asking this question. Using the Python bindings, I would like to provide a previous conversation with the large language model. However, I need clarification as I need to figure out how to do that in my code. Here is how I prompt the model in my class:

    def prompt(self, prompt, max_tokens = 200, temp = 0.7, top_k = 40, top_p = 0.4, repeat_penalty = 1.18, repeat_last_n = 64, streaming = False):
        with self.current_session:
            self._set_context()
            response = self.model.generate(
                                           prompt, 
                                           max_tokens = max_tokens, 
                                           temp = temp, 
                                           top_k = top_k,
                                           top_p = top_p,
                                           repeat_penalty = repeat_penalty,
                                           repeat_last_n = repeat_last_n,
                                           streaming = streaming
                                         )
            self.chat_session = self.model.current_chat_session
        return response

In the above code, I have self._set_context. That is the function that feeds data from a chat history. For example, self.chat_session is extracted from a previous conversation and formatted like this:

        self.chat_session = [
                             {'role' : 'user', 'content' : 'hello'},
                             {'role' : 'assistant', 'content' : 'Hello! How can I assist you today?'},
                            ]

And my self._set_context function is like this:

    def _set_context(self):
        for message in self.chat_session:
            self.model.current_chat_session.append(message)

But for some reason, the language model answers my following prompt as if there was no prior conversation. Indeed, I verified that by asking things from a previous conversation. I appreciate any help to help me resolve this issue. Thanks!

[cebtenzzre]

Setting the chat session doesn't actually feed anything to the model, you need to call .generate in order to actually put things in its memory.

[kaanaksit]

Please correct me if I am wrong. Does that mean calling 'self.model.generate' for each 'user' input in history? If so wouldn't that translate to regenerate an entire chat session? It could take a while for a long history. In other ways, what is the way to save and load a chat session?

[niranjanadas]

try out this , but its simple you can objectify it :
with model.chat_session('You are a geography expert.\nBe terse.', '### Instruction:\n{0}\n### Response:\n'): while True: response = model.generate(input("User:"), temp=0, streaming=True) for token in response: print(token, end='', flush=True) print("\n")

[torrinworx]

Could we possibly get an answer to this? This appears to still be an issue I'm also facing

[cebtenzzre]

This feature request is tracked at #1959.