feat: add support fastchat http bindings

[leiwen83]

• feat: add support fastchat http bindings

[wsxiaoys]

Thanks for the PR, a few comments

[wsxiaoys]

A few nits

[wsxiaoys]

LGTM

[wsxiaoys]

Hi @leiwen83 - Could you fix the CI checks, once fixed. it should be automatically merged.

[leiwen83]

Hi @leiwen83 - Could you fix the CI checks, once fixed. it should be automatically merged.

CI is fixed

[itlackey]

Hey @leiwen83 are you successful using FastChat with Tabby? If so, what model(s) are you using? I am trying to get it work and having some issues. Any advice you can lend is much appreciated.

[leiwen83]

Hey @leiwen83 are you successful using FastChat with Tabby? If so, what model(s) are you using? I am trying to get it work and having some issues. Any advice you can lend is much appreciated.

Hi, there is return format structure change in fastchat mainstream, so a patch is needed to work with current mainstream fastchat #670

You may try that fix patch first.

[itlackey]

Thank you both for your help to make this work! I will pull the latest and give this a try this evening.

[itlackey]

So far the response from FastChat has been too slow for Tabby. It seems like it's due too flooding the API with requests. We may need some denounce logic I the http device and/or request cancelation (if possible).
If I switch to manual and send one completion request at a time it seems to work well enough, but with automatic it is pretty much unusable.

I'm running wizard 3b on an a770 with 16gb VRAM

Here is the compose file I am using with a local Tabby container built from the latest code on the main branch:
docker-compose.yml.txt

[leiwen83]

So far the response from FastChat has been too slow for Tabby. It seems like it's due too flooding the API with requests. We may need some denounce logic I the http device and/or request cancelation (if possible). If I switch to manual and send one completion request at a time it seems to work well enough, but with automatic it is pretty much unusable.

I'm running wizard 3b on an a770 with 16gb VRAM

Here is the compose file I am using with a local Tabby container built from the latest code on the main branch: docker-compose.yml.txt

Yes, I also notice this slowness. For denounce logic, do you have any idea?

[itlackey]

I worked on it for a while last night but it doesn't seem to be working. I have never developed in Rust. So, I am not really sure if this is even on the right track, but here is my attempt:
https://github.com/itlackey/tabby/blob/feature/debounce/crates/http-api-bindings/src/fastchat.rs