Feature: Add Support for Data URI

[davleop]

Feature: add support for data URI

Following after OpenAI's vision docs: https://platform.openai.com/docs/guides/vision

[michaelfeil]

Nice, so you want to implement.

def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

And then accept the following?

base64_image = encode_image(image_path)
{
"url": f"data:image/jpeg;base64,{base64_image}"
}

I like it - please make sure to also add a unit test for this, and add a documentation. If there is an OpenAPI feature that generates pip install infinity_client, this would be helpful too.

Hint: you can cd libs/infinity_emb && make precommit to run the CI locally.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 73.56322% with 46 lines in your changes missing coverage. Please review.

Project coverage is 78.66%. Comparing base (5881a74) to head (b37bfe9).

Files with missing lines	Patch %	Lines
...inity_emb/infinity_emb/fastapi_schemas/data_uri.py	72.14%	39 Missing ⚠️
...inity_emb/infinity_emb/fastapi_schemas/pymodels.py	42.85%	4 Missing ⚠️
...inity_emb/infinity_emb/transformer/vision/utils.py	75.00%	2 Missing ⚠️
libs/infinity_emb/infinity_emb/infinity_server.py	92.85%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #385      +/-   ##
==========================================
- Coverage   78.99%   78.66%   -0.34%     
==========================================
  Files          37       39       +2     
  Lines        2804     2962     +158     
==========================================
+ Hits         2215     2330     +115     
- Misses        589      632      +43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[michaelfeil]

Is this a performance efficient check for this? Is it needed (doesn't the Mozilla spec say it needs to say base64?

[stikkireddy]

@michaelfeil do we need this? or want this?

[michaelfeil]

@davleop @stikkireddy Added a lot of changes, to make the data uri validation more native to the pydantic validation.

[stikkireddy]

@michaelfeil awesome! this is what i was hoping would be changed. you got rid of regex match for base64 format. The mdn guide specifies the format structure and if the base64 key is provided then its assumed that the rest of the string is base64 encoded. If decoding fails we can send that error back to the user.

https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data

[stikkireddy]

@michaelfeil

Last question is there any way to support this from OpenAI embeddings spec? extra_body is typically used to provide additional non spec features to allow the server to handle things. It would be nice to be able to reuse the openai embeddings client for things like:

client = OpenAI(....)
client.embeddings.create(
model="default",
inputs=["https://...", "data:..."],
extra_body={
  "infiniy_image_embeddings": true
}

I know its not exactly what openai supports but it would be great to be able to use existing clients to solve this problem if possible. they have built in retries, etc.

vllm does this for vllm specific features like guided decoding, etc.

[michaelfeil]

@stikkireddy Moved the regex directly into the validation of the pydantic type.

[stikkireddy]

libs/infinity_emb/infinity_emb/transformer/vision/utils.py

i assume you will remove that unnecessary code then?

[davleop]

@davleop @stikkireddy Added a lot of changes, to make the data uri validation more native to the pydantic validation.

Thanks for picking up my slack! I haven't had much time to take a look at this again. I kinda just implemented this as a solution I needed at the time and thought it might be good to push it up to main :)