I didn't get the point with session_id and time columns in the query, could you please elaborate that one? Also, according to the other way, if we pass the whole chat history in the queries, that would directly go with the prompt, right? We do not want to do that as that would be expensive with a paid API while make local models proces the prompt slower otherwise.

Maybe I misunderstood what you meant by Chat History. When using OpenAI for instance, you can pass chat history as follows:

messages = [
  dict(role='system', content='you are a helpful assistant'),
  dict(role='user', content='prompt 0'),
  dict(role='assistant', content='answer 0'),
  dict(role='user', content='prompt 1'),
]

The amount you pay for is #tokens(messages) +#tokens(answer) so including history.

I'm aware of the payment structure, I'm just asking if there's a more accurate way to provide chat history to the chatbot other than providing the whole list of conevrsation between the user and the bot. Isn't there a way to store the chat history in a compressed format like pickle or parquet and refer to its contents whenever needed? Just like we do it with the documents, same with the chat history. Although I'd need further explanation about your suggestion having session_id and time columns in the query as it somehow made sense but I didn't quite understand it, please consider my suggestion above about the compressed chat history and correct me if I'm wrong.