WIP: New python based entry point for containers

[jpodivin]

No description provided.

[jpodivin]

@SlyEcho This is the endpoint change I've mentioned.

[SlyEcho]

Does it allow passing arguments with spaces?

[jpodivin]

Does it allow passing arguments with spaces?

I'm not sure what exactly you mean. My implementation makes some changes. But, for example, these work:

docker run -v /local/model/path/:/models llama.cpp:endpoint --all-in-one /models/7B
docker run -v/local/model/path/:/models llama.cpp:endpoint --run /models/7B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -n 512

But I wouldn't really like to get this specific version merged. I would prefer subcommands over those args, as I wouldn't have to worry about overloading arguments between run/quantize/etc.

Not to say that it's impossible, but subcommands do offer flexibility you don't have with multiple arguments.

[SlyEcho]

That's what I meant.

I would prefer subcommands over those args, as I wouldn't have to worry about overloading arguments between run/quantize/etc

Couldn't it be done by just checking argv[1]? I don't know how to do it in Python but there is not really any "parsing" needed, I think, just switching by the string value. Kind of like the old script did.

[jpodivin]

Advantage of python argparse is that it does take care of those checks for you, including edge cases.
You can define CLI of arbitrary complexity using combination of commands, subcommands and args. The help, error handling, type checks and defaults come in the bargain.

Now you can do that with plain conditionals checks on argv elements, but you would also have to implement some other logic to make it robust. And that puts unnecessary burden on maintainers.

What I would propose is something like this:

usage: tools.py [-h] model {run,convert,quantize,all-in-one} ...

positional arguments:
  model                 Directory containing model file, or model file itself
                        (*.pth, *.pt, *.bin)
  {run,convert,quantize,all-in-one}
    run                 Run a model previously converted into ggml
    convert             Convert a llama model into ggml
    quantize            Optimize with quantization process ggml
    all-in-one          Execute convert & quantize

options:
  -h, --help            show this help message and exit

Each one of those would have their own args, so no chance of overloading. Actually I'm bit ashamed I've not proposed it already.

Edit: I'll make ASCIInema demo later today, when I have more time.

[jpodivin]

I have the promised demo.[0] It's aciinema[1] recording, so you can play it in terminal. I do recommend using the -i 1 when playing it, as otherwise you will have to endure long gaps in activity, as my desktop struggles. Also, I've been doing other things, and after quantization command I forgot I was recording for couple of minutes.

[0] https://gist.github.com/jpodivin/ef4d037c21bfc2ce0a9f91b1d3f29ea5
[1] https://asciinema.org/docs/usage

[jpodivin]

@SlyEcho wdyt?

[SlyEcho]

I have not had time to test it yet.

[jpodivin]

@SlyEcho So are we moving this out of WIP? Or should I just close it?

[SlyEcho]

I think it's still worth it.

There are a couple of things that have changed:

1. A lot more Dockerfiles.
2. The model format changed, so now we have .gguf files.

[jpodivin]

Right. I'll take a look at it.

…

On Tue, Aug 29, 2023 at 6:55 PM Henri Vasserman ***@***.***> wrote: I think it's still worth it. There are a couple of things that have changed: 1. A lot more Dockerfiles. 2. The model format changed, so now we have .gguf files. — Reply to this email directly, view it on GitHub <#1686 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APZOTD24XBLNH7DVKAD6XYDXXYNG7ANCNFSM6AAAAAAYZPZ3SQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jpodivin]

@SlyEcho I've updated the script to work with new model format and server binary. Also I've replaced the endpoint in other container files. I've tested the standard file quantization, conversion, run and server binary with open-llama.