llama2-7b serving How to make a POST request on the endpoint to get result back?

[giaosudau]

I am following https://github.com/ludwig-ai/ludwig/tree/master/examples/llama2_7b_finetuning_4bit
what is curl or post request look like to get result back from llama-2 model after fine tuning ?

[Infernaught]

With Ludwig, you don't need a REST API to query a model. You can simply load the fine-tuned model in Python from the local weights after the model has been fine-tuned. This would be the command:

model = LudwigModel.load(path_to_weights)

where path_to_weights refers to the local path to your fine-tuned model.

Does that answer your question?

[giaosudau]

Sorry but I wanted to use rest api.

[Infernaught]

I see. Sorry about the confusion. In that case, the curl command would look something like this:
curl http://0.0.0.0:8000/predict -X POST -F 'INPUT_COLUMN=INPUT_TEXT'
where you would replace INPUT_COLUMN and INPUT_TEXT with the column name of the input feature and the text you want to feed into the model, respectively.

You can find more information on this here: https://ludwig.ai/latest/user_guide/serving/

Hopefully this is more helpful!

[Alpe6825]

@Infernaught

Could you make a more explicit example?

I followed:
https://ludwig.ai/latest/#large-language-model-fine-tuning and https://ludwig.ai/latest/user_guide/serving/

but got {"error":"Unexpected Error: could not run inference on model"} and

Failed to run predict: template invalid for zero-shot prompt: Prompt template must contain either the '__sample__' field or one of the columns from the dataset
Traceback (most recent call last):
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/prompt.py", line 179, in format_input_with_prompt
    _validate_prompt_template(template_fields, task_str, is_few_shot, dataset_df.columns, input_col_name)
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/prompt.py", line 223, in _validate_prompt_template
    raise ValueError(
ValueError: Prompt template must contain either the '__sample__' field or one of the columns from the dataset

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/serve.py", line 90, in predict
    resp, _ = model.predict(dataset=[entry], data_format=dict)
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/api.py", line 950, in predict
    dataset, _ = preprocess_for_prediction(  # TODO (Connor): Refactor to use self.config_obj
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/preprocessing.py", line 2330, in preprocess_for_prediction
    processed = data_format_processor.preprocess_for_prediction(
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/preprocessing.py", line 218, in preprocess_for_prediction
    dataset, training_set_metadata = build_dataset(
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/preprocessing.py", line 1232, in build_dataset
    synthesized_dataset_cols = handle_features_with_prompt_config(
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/preprocessing.py", line 1817, in handle_features_with_prompt_config
    dataset_cols[input_col_name] = format_input_with_prompt(
  File "/home/a100-80gbx2/anaconda3/envs/ludwig_llama2/lib/python3.10/site-packages/ludwig/data/prompt.py", line 181, in format_input_with_prompt
    raise ValueError(f"template invalid for {'few-shot' if is_few_shot else 'zero-shot'} prompt: {e}")
ValueError: template invalid for zero-shot prompt: Prompt template must contain either the '__sample__' field or one of the columns from the dataset
INFO:     127.0.0.1:60224 - "POST /predict HTTP/1.1" 500 Internal Server Error