Is there a way to use llm-app with SentenceTransformers + OpenAI gpt-3.5-turbo / Anthropic Claude to chat with private documents??

[lalchandaniaadarsh]

How to use this library with SentenceTransformers as Embedding Model and OpenAI gpt-3.5-turbo as chat engine? WITH CONTEXT and CHAT HISTORY

[mdmalhou]

It is possible to use SentenceTransformers and OpenAI in the same example. Take a look at how to use SentenceTransformerTask in pipelines/local example and OpenAIChatGPTModel in pipelines/contextful. Those examples are with context. For chat history, one way to get it is by introducing a session_id and time columns in the queries, then you could groupby the session_id and filter+sort using time column. Another way to achieve this is you can have a stateful front end (e.g. streamlit) and pass the whole chat history in the query.

[lalchandaniaadarsh]

I didn't get the point with session_id and time columns in the query, could you please elaborate that one? Also, according to the other way, if we pass the whole chat history in the queries, that would directly go with the prompt, right? We do not want to do that as that would be expensive with a paid API while make local models proces the prompt slower otherwise.

[mdmalhou]

Maybe I misunderstood what you meant by Chat History. When using OpenAI for instance, you can pass chat history as follows:

messages = [
  dict(role='system', content='you are a helpful assistant'),
  dict(role='user', content='prompt 0'),
  dict(role='assistant', content='answer 0'),
  dict(role='user', content='prompt 1'),
]

The amount you pay for is #tokens(messages) +#tokens(answer) so including history.

[lalchandaniaadarsh]

I'm aware of the payment structure, I'm just asking if there's a more accurate way to provide chat history to the chatbot other than providing the whole list of conevrsation between the user and the bot. Isn't there a way to store the chat history in a compressed format like pickle or parquet and refer to its contents whenever needed? Just like we do it with the documents, same with the chat history. Although I'd need further explanation about your suggestion having session_id and time columns in the query as it somehow made sense but I didn't quite understand it, please consider my suggestion above about the compressed chat history and correct me if I'm wrong.