How does skyvern integrate with ollama litellm

[alexsiu398]

Is there a way or tutorial on how to configure ollama litellm to work with skyvern? How can skyvern work with a local llm?

[suchintan]

Here's an example where @ykeremy built out bedrock support within Skyvern

https://github.com/Skyvern-AI/skyvern/pull/251/files

Are you open to opening a PR for ollama + litellm? We'd love a contribution here!

[suchintan]

Ignore the files in the experimentation module. The other configs are all you need!

[santiagoblanco22]

Nice. Now GPT4 60 USD in 3 days. :( Ollama is awesome! I don't know how to help!

[suchintan]

@santiagoblanco22 we would love a contribution here!! Or maybe we can ask for people's help in our discord?

GPT4 is super expensive. Try it with Claude 3 sonnet instead

[OB42]

hi, I'm currently trying to add it. :)

Do you think we should allow all ollama models? in setup.sh should we ask the user for a specific model name(as a string)? or a numbered choice like for anthropic with just llama3/mistral maybe llava?

[OB42]

FYI for now it seems that most models available on Ollama are not good enough for Skyvern , at least on my computer, so it seems pointless to add models that would not work well.

Maybe it could work with a 34/70B model with no quantization, but you would need a very beefy setup, at that point you'd probably be better off using bedrock/anthropic IMO

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[HarryBak]

FYI for now it seems that most models available on Ollama are not good enough for Skyvern , at least on my computer, so it seems pointless to add models that would not work well.

Maybe it could work with a 34/70B model with no quantization, but you would need a very beefy setup, at that point you'd probably be better off using bedrock/anthropic IMO

What models did you try using?

At what stage was it getting stuck?

Any other information you could share that may help the research into making this reality?

[OB42]

FYI for now it seems that most models available on Ollama are not good enough for Skyvern , at least on my computer, so it seems pointless to add models that would not work well.
Maybe it could work with a 34/70B model with no quantization, but you would need a very beefy setup, at that point you'd probably be better off using bedrock/anthropic IMO

What models did you try using?

At what stage was it getting stuck?

Any other information you could share that may help the research into making this reality?

If I remember correctly, LLama-2, Mistral 7B, Phi3, maybe I'm forgetting some.
It was struggling to follow the prompt, to output a valid JSON(not sure if skyvern still uses JSON with LLMs?), and/or choose the correct id when deciding to click on something

maybe there was too much quantization, maybe I did not use adequate parameters, maybe we need completely different prompts for weaker models

Also I think @suchintan said that the screenshots are really important for the LLMs to correctly understand what's going on the page, and most of the models I tried were not multimodal(and when they were, they did not have a good enough understanding of the screenshots)

But this was months ago and I'm not really sure about this, also I didn't really follow the latest changes with Skyvern.

[HarryBak]

FYI for now it seems that most models available on Ollama are not good enough for Skyvern , at least on my computer, so it seems pointless to add models that would not work well.
Maybe it could work with a 34/70B model with no quantization, but you would need a very beefy setup, at that point you'd probably be better off using bedrock/anthropic IMO

What models did you try using?
At what stage was it getting stuck?
Any other information you could share that may help the research into making this reality?

If I remember correctly, LLama-2, Mistral 7B, Phi3, maybe I'm forgetting some. It was struggling to follow the prompt, to output a valid JSON(not sure if skyvern still uses JSON with LLMs?), and/or choose the correct id when deciding to click on something

maybe there was too much quantization, maybe I did not use adequate parameters, maybe we need completely different prompts for weaker models

Also I think @suchintan said that the screenshots are really important for the LLMs to correctly understand what's going on the page, and most of the models I tried were not multimodal(and when they were, they did not have a good enough understanding of the screenshots)

But this was months ago and I'm not really sure about this, also I didn't really follow the latest changes with Skyvern.

Didn't expect such a prompt reply, thank you. Do you know what GPU you were running at the time?

Awesome information, the models have come a very long way since you first tested this so I'm intrigued to see.

I would assume that for this to work on local models there will have to be a lot of optimizations to be done. Context length becomes an issue quickly with local models.

For repeating workflows where only the input JSON changes then caching the workflow locally will be a definite requirement.

First run through would be slow but after that it would be lightning fast.

If anyone has any inputs on ideas to optimize the workflow further feel free to add them in here.

[suchintan]

We've had some promising results doing internal testing of the intern vl 2 model series (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)

I'm not sure if that's available on ollama yet but it might be a good place to get started!