Ability to mess with logit biases

[simonw]

Inspired by https://twitter.com/goodside/status/1669613516402089984

[simonw]

Adding this as CLI options feels messy to me, plus it doesn't necessarily work across other models.

One option would be to have this as a template-only feature, refs #23.

[simonw]

Slight twist: what if you want to use another template at the same time?

Might be an argument for supporting combined templates - pass -t multiple times and the result is a combination of those templates - the most recent of each of the prompt, system prompt, logit biases etc.

[simonw]

So I think this needs a complex version where you set the biases directly, and a simple version where you can just pass in a string of words to suppress and the tool splits on whitespace and augments them (as in that example - uppercase, space prefixed etc) and sets them to -100.

Should design this to support more new sugar features in the future.

[simonw]

I tried to get this working as a prototype for:

• Support for -o key value options such as temperature #63

It didn't seem to work though:

llm -m gpt-4 "Once upon a" -o logit_bias '{"2435":-100, "640":-100}' -o temperature 0 --no-stream

time, in a small village nestled between the rolling hills and dense forests

... but that's because I was using token IDs for GPT-3 when I should have been using them for GPT-4.

Here's a transcript where I try to block all of the obvious "time" related tokens:

% llm -m gpt-3 "Once upon a" -o logit_bias '{"2435":-100, "640":-100}' -o temperature 0 --no-stream
Error: The model `gpt-3` does not exist
% llm -m text-davinci-003 "Once upon a" -o logit_bias '{"2435":-100, "640":-100}' -o temperature 0 --no-stream
Error: This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
% ttok 'time time' --tokens
1712 892
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100}' -o temperature 0            
-time, in a small village nestled between the rolling hills and dense forests, there lived a young girl named Amara. She was a curious and adventurous child, always exploring the woods and meadows^C
Aborted!
% ttok --tokens -- '-time time'         
7394 892
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100}' -o temperature 0
ime in a small village nestled between the rolling hills and dense forests, there lived a young girl named Amara. She was a kind and gentle soul, always eager to help others and spread joy wherever she went. Amara lived with her loving parents^C
Aborted!
% ttok 'ime' --tokens                                                                               
547
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100}' -o temperature 0
Time, in a small village nestled between the rolling hills and dense forests, there lived a young girl named Amara. She was a curious and adventurous child, always exploring the woods and meadows that surrounded her home.
Aborted!
% ttok 'Time' --tokens                                                                                            
1489
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100, "1489": -100}' -o temperature 0
"time, in a small village nestled between the rolling hills and dense forests, there lived a young^C
% ttok '"time' --tokens
33239
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100, "1489": -100, "33239": -100}' -o temperature 0
.time in a small village nestled between the mountains and the sea, there lived a young girl^C
Aborted!
% ttok '.time' --tokens
6512
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100, "1489": -100, "33239": -100, "6512": -100}' -o temperature 0 
_time, in a small village nestled^C
Aborted!
% ttok '_time' --tokens
3084
% llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "7394": -100, "547": -100, "1489": -100, "33239": -100, "6512": -100, "3084": -100}' -o temperature 0
t ime in a small village nestled^C
Aborted!

[simonw]

Here's my hacky prototype diff for this feature:

diff --git a/llm/cli.py b/llm/cli.py
index 97a8449..12ccee5 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -71,6 +71,14 @@ def cli():
     type=(str, str),
     help="Parameters for template",
 )
+@click.option(
+    "options",
+    "-o",
+    "--option",
+    multiple=True,
+    type=(str, str),
+    help="Options to pass to the model",
+)
 @click.option("--no-stream", is_flag=True, help="Do not stream output")
 @click.option("-n", "--no-log", is_flag=True, help="Don't log to database")
 @click.option(
@@ -95,6 +103,7 @@ def prompt(
     model,
     template,
     param,
+    options,
     no_stream,
     no_log,
     _continue,
@@ -166,6 +175,14 @@ def prompt(
         if model is None and template_obj.model:
             model = template_obj.model
 
+    options = dict(options)
+    if "logit_bias" in options:
+        options["logit_bias"] = dict(
+            (int(k), v) for k, v in json.loads(options["logit_bias"]).items()
+        )
+    if "temperature" in options:
+        options["temperature"] = float(options["temperature"])
+
     messages = []
     if _continue:
         _continue = -1
@@ -197,6 +214,7 @@ def prompt(
             response = openai.ChatCompletion.create(
                 model=model,
                 messages=messages,
+                **options,
             )
             debug["model"] = response.model
             debug["usage"] = response.usage
@@ -209,6 +227,7 @@ def prompt(
             for chunk in openai.ChatCompletion.create(
                 model=model,
                 messages=messages,
+                **options,
                 stream=True,
             ):
                 debug["model"] = chunk.model

[simonw]

Note that I had to do something custom for logit_bias because {124: -100} isn't valid JSON (keys must be strings).

[simonw]

Here's a good minimal example:

llm -m gpt-4 "Once upon a" -o logit_bias '{"1712":-100, "892":-100, "1489":-100}' -o temperature 1.0