Code 502 when using OpenAI client

[liux2]

I just started using infinity last week, it was pretty smooth when calling with curl and requests package. But when I test it with openai client, I got code 502 every time, and the weird thing is that my server does not get any request. There is no relevant log at all.
Here is the code I'm using:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://local-ip:7997/v1")

try:
    response = client.embeddings.create(
        model="models/bge-m3",
        input=["the sound of a beep", "the sound of a cat"],
        encoding_format="float",
        extra_body={"modality": "text"},
    )
    print(response)
except Exception as e:
    print(f"An error occurred: {e}")

And my package info:

name         : openai                                         
 version      : 1.55.0                                         
 description  : The official Python library for the openai API 

dependencies
 - anyio >=3.5.0,<5
 - distro >=1.7.0,<2
 - httpx >=0.23.0,<1
 - jiter >=0.4.0,<1
 - pydantic >=1.9.0,<3
 - sniffio *
 - tqdm >4
 - typing-extensions >=4.11,<5

and my error message:

    client.embeddings.create(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/resources/embeddings.py", line 124, in create
   return self._post(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1278, in post
   return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 955, in request
   return self._request(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1044, in _request
   return self._retry_request(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1093, in _retry_request
   return self._request(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1044, in _request
   return self._retry_request(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1093, in _retry_request
   return self._request(
 File "/Users/xingbangliu/Library/Caches/pypoetry/virtualenvs/src-cjcNRWxB-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1059, in _request
   raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Error code: 502

[liux2]

What happened here?

[liux2]

More details:

OpenAI Client Schema:
{'dimensions': <class 'int'>,
 'encoding_format': typing.Literal['float', 'base64'],
 'input': typing_extensions.Required[typing.Union[str, typing.List[str], typing.Iterable[int], typing.Iterable[typing.Iterable[int]]]],
 'model': typing_extensions.Required[typing.Union[str, typing.Literal['text-embedding-ada-002', 'text-embedding-3-small', 'text-embedding-3-large']]],
 'user': <class 'str'>}

Actual API Schema (from curl):
{'encoding_format': 'float',
 'input': ['string'],
 'modality': 'text',
 'model': 'models/bge-m3',
 'user': 'string'}

Analysis of differences:

Fields only in OpenAI client:
{'dimensions'}

Fields only in actual API:
{'modality'}

Common fields:
{'model', 'user', 'input', 'encoding_format'}

Testing with OpenAI client schema:
OpenAI client error: Error code: 502

Testing with actual API schema:
Direct request status: 200

relevant code:

from openai import OpenAI
from openai.types.create_embedding_response import CreateEmbeddingResponse
from openai.types import EmbeddingCreateParams
import json
from pprint import pprint
from typing import get_type_hints

# Define the actual API schema from curl command
ACTUAL_API_SCHEMA = {
    "model": "models/bge-m3",
    "encoding_format": "float",
    "user": "string",
    "input": ["string"],
    "modality": "text"
}

# Get OpenAI client schema
openai_params = get_type_hints(EmbeddingCreateParams)
print("\nOpenAI Client Schema:")
pprint(openai_params)

print("\nActual API Schema (from curl):")
pprint(ACTUAL_API_SCHEMA)

# Compare differences
print("\nAnalysis of differences:")
openai_keys = set(openai_params.keys())
actual_keys = set(ACTUAL_API_SCHEMA.keys())

print("\nFields only in OpenAI client:")
pprint(openai_keys - actual_keys)

print("\nFields only in actual API:")
pprint(actual_keys - openai_keys)

print("\nCommon fields:")
pprint(openai_keys & actual_keys)

# Test both schemas
client = OpenAI(api_key="EMPTY", base_url="http://local-ip:7997/v1")

print("\nTesting with OpenAI client schema:")
try:
    response1 = client.embeddings.create(
        model="models/bge-m3",
        input=["test string"],
        encoding_format="float",
        extra_body={"modality": "text"}  # Note: modality needs to be passed as extra_body
    )
    print("OpenAI client request successful")
except Exception as e:
    print(f"OpenAI client error: {e}")

print("\nTesting with actual API schema:")
import requests
try:
    response2 = requests.post(
        "http://local-ip:7997/v1/embeddings",
        headers={
            "accept": "application/json",
            "Content-Type": "application/json"
        },
        json=ACTUAL_API_SCHEMA
    )
    print(f"Direct request status: {response2.status_code}")
except Exception as e:
    print(f"Direct request error: {e}")

[liux2]

Nvm, openai client has to configure proxy envs if you are behind a proxy...

[michaelfeil]

Seems like your IP is weird, makes sense you solved it with the proxy. You would see in the fast app logs if the server would respond with 502 error.

[liux2]

I manually censored my IP. The fast app does not generate any logs for this issue. I was behind Zerotier tho, but this is not the issue. The issue is that normally you would export proxy settings to make the system proxy take effect in the terminal. but this proxy is not the prerequisite for me to access the deployed service. Somehow I have to turn off the proxy service entirely or apply the setting in the terminal to use the Openai client.