Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Design new LLM database schema #556

Open
simonw opened this issue Aug 26, 2024 · 6 comments
Open

Design new LLM database schema #556

simonw opened this issue Aug 26, 2024 · 6 comments
Labels
design

Comments

@simonw
Copy link
Owner

simonw commented Aug 26, 2024

The LLM database schema for storing responses needs to evolve to support some new features:

  • "Mock" responses. Sometimes it's useful to run a prompt that contains several previous messages which didn't actually come from a model - for example, prompting Claude 3 Haiku with a bunch of examples that are input as previous user messages. Differentiating these from actual text returned by a model is important to support LLM's goal of recording responses from models over time so you can analyze them later.
  • Another Claude feature: prompt caching. This is a larger feature to decide how to build, but since one of the uses of prompt caching is for longer conversations it's important to be able to remember at what point in the conversation the "cache_control": {"type": "ephemeral"} rule was injected.
  • Images! For prompts that use images one of the following three needs to be stored: the image URL (for images that were referenced by their URL) the image file path (for images that were a file on disk) or the raw bytes of the image itself (for cases where we want that stored in the database as opposed to living separately on disk)
  • Other multi-modal data: Gemini Pro accepts PDF and audio and video, fer example
  • That Claude trick where you can pre-fill the prefix for the response, passing <!doctype html><html> to ensure an HTML response for example. If that trick is used it should be recorded in the database as well.
@simonw simonw added the design label Aug 26, 2024
@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024

Current schema:

llm/docs/logging.md

Lines 146 to 168 in e867e13

CREATE TABLE [conversations] (
[id] TEXT PRIMARY KEY,
[name] TEXT,
[model] TEXT
);
CREATE TABLE [responses] (
[id] TEXT PRIMARY KEY,
[model] TEXT,
[prompt] TEXT,
[system] TEXT,
[prompt_json] TEXT,
[options_json] TEXT,
[response] TEXT,
[response_json] TEXT,
[conversation_id] TEXT REFERENCES [conversations]([id]),
[duration_ms] INTEGER,
[datetime_utc] TEXT
);
CREATE VIRTUAL TABLE [responses_fts] USING FTS5 (
[prompt],
[response],
content=[responses]
);

Repository owner deleted a comment Aug 26, 2024
Repository owner deleted a comment Aug 26, 2024
Repository owner deleted a comment Aug 26, 2024
Repository owner deleted a comment Aug 26, 2024
@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024

Since some of these features are specific to Claude, it may make sense to have some kind of JSON column that specific LLM plugins can use to record information that only they care about - maybe using JSON keys with their name in e.g. "llm-claude-3:prefix": "<!doctype html><html>".

@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024

Does it even make sense for a conversation that was "real" (user submits a prompt, gets a model response, sends a reply, gets another model response) to be stored in the same schema as a conversation with simulated aspects (user sends an example-prompt/example-response/example-prompt/example-resoponse/real-prompt sequence)?

I think it does, especially since any follow-up prompts after that will still send the full sequence that's been stored just as it would if it wasn't full of mocked data.

Terminology here is hard. Are these "fake" responses? "mock" responses? Something else?

@simonw simonw mentioned this issue Aug 26, 2024
Open
@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024

Worth considering tools here too, since those may require special thought about how to construct the schema.

@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024

I think "extras" might be a good way to model extra things that are specific to certain models. I could have a extras_json column for storing these. For places where a model needs to be passed extra custom arguments I could use -x/--extra key value in the llm prompt CLI options.

Not sure they should have the same name though, the extras_json column may store things that aren't related to -x/--extra options so perhaps they should be different.

plugin_json is another option here, for JSON that is specific to the LLM plugin used for the prompt.

@simonw
Copy link
Owner Author

simonw commented Aug 26, 2024 •
edited
Loading

Here's a schema consideration from tool use with Claude - a response might come back like this: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#example-api-response-with-a-tool-use-content-block

{
  "id": "msg_01Aq9w938a90dw8q",
  "model": "claude-3-5-sonnet-20240620",
  "stop_reason": "tool_use",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "<thinking>I need to use the get_weather, and the user wants SF, which is likely San Francisco, CA.</thinking>"
    },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": {"location": "San Francisco, CA", "unit": "celsius"}
    }
  ]
}

How should that be stored in the database? It's a single response from the assistant but it has two items in content - one of type text and one of type tool_use.

Also from that section of documentation:

Unlike APIs that separate tool use or use special roles like tool or function, Anthropic's API integrates tools directly into the user and assistant message structure.

Messages contain arrays of text, image, tool_use, and tool_result blocks. user messages include client-side content and tool_result, while assistant messages contain AI-generated content and tool_use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
design
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@simonw