added two new embedding model's encoding (#247)

Library doesn't support two new embedding model's encoding mapper - `text-embedding-3-small` - `text-embedding-3-large` Added Encoding mapper for 2 new embedding models. The source of mapping is taken from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
openai · Feb 9, 2024 · 55c8d83 · 55c8d83
1 parent 6cc3a46
commit 55c8d83
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/tiktoken/model.py b/tiktoken/model.py
@@ -26,6 +26,8 @@
     "babbage-002": "cl100k_base",
     # embeddings
     "text-embedding-ada-002": "cl100k_base",
+    "text-embedding-3-small": "cl100k_base",
+    "text-embedding-3-large": "cl100k_base",
     # DEPRECATED MODELS
     # text (DEPRECATED)
     "text-davinci-003": "p50k_base",