Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Prevent going past token limit in OpenAI calls in PromptNode #4179

Merged
merged 45 commits into from
Mar 3, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
05780e1
Refactoring to remove duplicate code when using OpenAI API
sjrl Feb 16, 2023
7b425c4
Adding docstrings
sjrl Feb 16, 2023
f4a8e58
Fix mypy issue
sjrl Feb 16, 2023
5e851ab
Moved retry mechanism to openai_request function in openai_utils
sjrl Feb 16, 2023
6b8dbb0
Migrate OpenAI embedding encoder to use the openai_request util funct…
sjrl Feb 16, 2023
c274885
Adding docstrings.
sjrl Feb 16, 2023
4479f41
pylint import errors
sjrl Feb 16, 2023
34a7943
More pylint import errors
sjrl Feb 16, 2023
31f707a
Move construction of headers into openai_request and api_key as input…
sjrl Feb 16, 2023
feb1a24
Made _openai_text_completion_tokenization_details so can be resued in…
sjrl Feb 16, 2023
952214e
Add prompt truncation to the PromptNode.
sjrl Feb 16, 2023
68e18fe
Removed commented out test.
sjrl Feb 16, 2023
7453b9a
Bump version of tiktoken to 0.2.0 so we can use MODEL_TO_ENCODING to …
sjrl Feb 16, 2023
8e017b4
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 17, 2023
88444a0
Change one method back to public
sjrl Feb 17, 2023
2793ffb
Fixed bug in token length truncation. Included answer length into tru…
sjrl Feb 17, 2023
017acb9
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 17, 2023
8f728d9
Pylint error
sjrl Feb 17, 2023
a88bbc9
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 20, 2023
23c4b13
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 21, 2023
80f6aed
Improved warning message
sjrl Feb 21, 2023
4e3d1c7
Added _ensure_token_limit for HFLocalInvocationLayer. Had to remove m…
sjrl Feb 21, 2023
d9d17a3
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 21, 2023
6200b2e
Adding tests
sjrl Feb 21, 2023
6fc0fc2
Expanded on doc strings
sjrl Feb 21, 2023
5383f77
Updated tests
sjrl Feb 21, 2023
8238703
Update docstrings
sjrl Feb 21, 2023
86557e7
Update tests, and go back to how USE_TIKTOKEN was used before.
sjrl Feb 21, 2023
0d9f308
Update haystack/nodes/prompt/prompt_node.py
sjrl Feb 23, 2023
e736717
Update haystack/nodes/prompt/prompt_node.py
sjrl Feb 23, 2023
3af1b19
Update haystack/nodes/prompt/prompt_node.py
sjrl Feb 23, 2023
2b490b4
Update haystack/nodes/retriever/_openai_encoder.py
sjrl Feb 23, 2023
e44931f
Update haystack/utils/openai_utils.py
sjrl Feb 23, 2023
922d141
Update haystack/utils/openai_utils.py
sjrl Feb 23, 2023
1dd9485
Updated docstrings, and added integration marks
sjrl Feb 27, 2023
2261bf9
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Feb 27, 2023
bf37ff2
Remove comment
sjrl Feb 27, 2023
ff51d35
Update test
sjrl Feb 28, 2023
aa80052
Fix test
sjrl Mar 1, 2023
8134d76
Update test
sjrl Mar 1, 2023
e47de9f
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Mar 2, 2023
1185ce0
Merge branch 'main' of github.com:deepset-ai/haystack into openai/tok…
sjrl Mar 2, 2023
ae5d933
Updated openai_request function to work with the azure api
sjrl Mar 2, 2023
5a50ce5
Fixed error in _openai_encodery.py
sjrl Mar 2, 2023
3c8db04
Merge branch 'main' into openai/token_limit
vblagoje Mar 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated openai_request function to work with the azure api
  • Loading branch information
sjrl committed Mar 2, 2023
commit ae5d933cc4faa1b919963c1d70641609f90301c4
9 changes: 5 additions & 4 deletions haystack/nodes/answer_generator/openai.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,17 +217,18 @@ def predict(
"presence_penalty": self.presence_penalty,
"frequency_penalty": self.frequency_penalty,
}
url = "https://api.openai.com/v1/completions"
if self.using_azure:
url = f"{self.azure_base_url}/openai/deployments/{self.azure_deployment_name}/completions?api-version={self.api_version}"
else:
url = "https://api.openai.com/v1/completions"

headers = {"Content-Type": "application/json"}
if self.using_azure:
headers = {"api-key": self.api_key, **headers}
headers["api-key"] = self.api_key
else:
headers = {"Authorization": f"Bearer {self.api_key}", **headers}
headers["Authorization"] = f"Bearer {self.api_key}"

res = openai_request(url=url, api_key=self.api_key, payload=payload, timeout=timeout)
res = openai_request(url=url, headers=headers, payload=payload, timeout=timeout)
_check_openai_text_completion_answers(result=res, payload=payload)
generated_answers = [ans["text"] for ans in res["choices"]]
answers = self._create_answers(generated_answers, input_docs)
Expand Down
2 changes: 1 addition & 1 deletion haystack/nodes/prompt/prompt_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,7 @@ def invoke(self, *args, **kwargs):
"best_of": kwargs_with_defaults.get("best_of", 1),
"logit_bias": kwargs_with_defaults.get("logit_bias", {}),
}
res = openai_request(url=self.url, api_key=self.api_key, payload=payload)
res = openai_request(url=self.url, headers=self.headers, payload=payload)
_check_openai_text_completion_answers(result=res, payload=payload)
responses = [ans["text"].strip() for ans in res["choices"]]
return responses
Expand Down
5 changes: 2 additions & 3 deletions haystack/utils/openai_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,14 @@ def _openai_text_completion_tokenization_details(model_name: str):
@retry_with_exponential_backoff(
backoff_in_seconds=OPENAI_BACKOFF, max_retries=OPENAI_MAX_RETRIES, errors=(OpenAIRateLimitError, OpenAIError)
)
def openai_request(url: str, api_key: str, payload: Dict, timeout: Union[float, Tuple[float, float]] = OPENAI_TIMEOUT):
def openai_request(url: str, headers: Dict, payload: Dict, timeout: Union[float, Tuple[float, float]] = OPENAI_TIMEOUT):
"""Make a request to the OpenAI API given a `url`, `headers`, `payload`, and `timeout`.

:param url: The URL of the OpenAI API.
:param api_key: Your API key from OpenAI. It is required for this node to work.
:param headers: Dictionary of HTTP Headers to send with the :class:`Request`.
:param payload: The payload to send with the request.
:param timeout: The timeout length of the request. The default is 30s.
"""
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
response = requests.request("POST", url, headers=headers, data=json.dumps(payload), timeout=timeout)
res = json.loads(response.text)

Expand Down