Request: Add async support

[creatorrr]

Async support (maybe via aiorequests/aiohttp?) would be very appreciated. Esp since some of the predictions can take a while to complete.

[mohamed-ezz]

+1

[creatorrr]

Created a PR to add support for this. cc/ @bfirsh

[0dminnimda]

@creatorrr How is the progress with #76? Do you need any help/assistance?

[creatorrr]

It's ready for review, @0dminnimda

[antomin]

It's ready for review, @0dminnimda

Hi. Thanks for your job. Code in your branch is woked?

[Paillat-dev]

1+

[osmanio2]

1+

[mattt]

Hi @creatorrr. Thank you for contributing #76. Thank you also for your patience, as this is a long time coming. I'm happy to report that async support landed in version 0.18.0.

Here's an example of how to run multiple predictions concurrently and wait for their results:

import asyncio
import replicate

# https://replicate.com/stability-ai/sdxl
model_version = "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"
prompts = [
    f"A chariot pulled by a team of {count} rainbow unicorns"
    for count in ["two", "four", "six", "eight"]
]

async with asyncio.TaskGroup() as tg:
    tasks = [
        tg.create_task(replicate.async_run(model_version, input={"prompt": prompt}))
        for prompt in prompts
    ]

results = await asyncio.gather(*tasks)
print(results)

[githubloader]

When using async_run, is it possible to run different model_versions with different prompts? Or is it only limited to one model_version to different prompts for now?

[mattt]

@githubloader The example I shared uses the same model version for all predictions, but there's nothing to stop you from using different combinations of inputs and models and running them all in parallel.

[thomastraum]

how does this work when reading images with aiofiles?

    async with aiofiles.open(os.path.join(settings.STATIC_DIR, "uploads", render_settings.filename), 'rb') as image:
        image_content = await image.read()
        image_string = base64.b64encode(image_content).decode('utf-8')
        input = {
            "image": image_string,                
            "width":768,
            "height":  768,
            "prompt": render_settings.prompt,
            "num_images": 1,
            "guidance_scale": 8,
            "archive_outputs": False,
            "prompt_strength": 0.45,
            "sizing_strategy": "width/height",
            "lcm_origin_steps": 50,
            "canny_low_threshold": 100,
            "num_inference_steps": 4,
            "canny_high_threshold": 200,
            "control_guidance_end": 1,
            "control_guidance_start": 0,
            "controlnet_conditioning_scale": 2
            }

    result = await replicate.async_run(
        "fofr/latent-consistency-model:a83d4056c205f4f62ae2d19f73b04881db59ce8b81154d314dd34ab7babaa0f1", 
        input=input
    )

I tried a few variations of

    image_content = await image.read()

I get either not serializeable when I open as binary or replicate.exceptions.ReplicateError: - input.image: Does not match format 'uri'

[mattt]

Hi @thomastraum. The async part of your code looks fine. I think the issue is that you're passing the raw, base64-encoded string to the image parameter. If you make it a data URI, that should work.

image_string = f"data:,{base64.b64encode(image_content).decode('utf-8')}"

[thomastraum]

Ah amazing. Thanks so much for helping a beginner

[PrateekRajan]

Hi @mattt . Beginner question. I have been trying to test out async_run with clip_interrogator but also seem to run into AttributeError: module 'replicate' has no attribute 'async_run'. I have made sure that I have the latest replicate version but I think I am missing something here. For reference I am referring to this code in README

import asyncio
import replicate

# https://replicate.com/stability-ai/sdxl
model_version = "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"
prompts = [
    f"A chariot pulled by a team of {count} rainbow unicorns"
    for count in ["two", "four", "six", "eight"]
]

async with asyncio.TaskGroup() as tg:
    tasks = [
        tg.create_task(replicate.async_run(model_version, input={"prompt": prompt}))
        for prompt in prompts
    ]

results = await asyncio.gather(*tasks)
print(results)

Appreciate any insights you can provide here.

[mattt]

@PrateekRajan Hmm, the only reason I can think of is that your program is importing an old version of the client library. We have automated tests for replicate.async_run here that pass. And this code works locally for me. Maybe its loading from the system packages instead of a virtual environment or vice-versa?

[githubloader]

Hi,
Is there a way for replicate.async_run to also return prediction id? It seems to only return output (url to file)?