From 9ceaf806995a324b28de36c4d808155c8e83c30a Mon Sep 17 00:00:00 2001 From: laz4rz Date: Sat, 30 Mar 2024 14:14:45 +0100 Subject: [PATCH] add changes proposal and motivation --- changes_proposal.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 changes_proposal.md diff --git a/changes_proposal.md b/changes_proposal.md new file mode 100644 index 0000000..4374bd2 --- /dev/null +++ b/changes_proposal.md @@ -0,0 +1,39 @@ +1. cost tracker handles completion creation +Change: separating completion and cost tracking, by changing the main functionality from `chat_completion` to `update_cost` +Motivation: + - bulletproofs us from changes in how the completion is created, we only care about response structure + - allows easier integration, user only has to initialize tracker object and call `update_cost(response)`, + otherwise each chat completion call would have to be rewritten + +2. costs are calculated across all log files +Change: + - static `total_cost` that will calculate total spending from logs + - static `experiment_cost(experiment_name=self.experiment_name)` gets you total cost of specific experiment + - defaulting to current experiment_name in tracker object + - if object not initialized, experiment_name has to be provided + - `cost` that gets you costs for current run of this tracker object + +3. log file just acumulates total cost +Change: + - add breakdown of responses/input token per response/output token per response/cost per response + - maybe change log file format to json, so that we can better handle logs, for example: + ``` + { + "experiment_name" + "model": + "run_datetime": + "logs": + { + "0": { # maybe datetime of response? + "num_of_input_tokens": + "num_of_output_tokens": + "other": # additional info? message? thread? prompt? + } + } + "total": { + "cost": # something else? + } + } + ``` + +3. WIP \ No newline at end of file