server : better default prompt

[ggerganov]

With the default server settings on master, the bot continues to generate the user's response due to capitalization differences between the prompt and the username. Here is the result from just sending "Hi" as input:

With the proposed prompt, it behaves better.
However, it often generates the string "\end{code}":

Probably we need a few shot examples in the prompt to drive it in the correct "dialogue"?
This is using vanilla LLaMA v1 7B

[wtarreau]

I noticed this a few times as well and thought we should make the reverse-prompt case-insensitive to help against this problem.

[ggerganov]

Btw, lately I also see many complaints in the issues about interactive mode behaving weirdly. We might have broken something along the way, I don't have much time to dig into that atm though

[wtarreau]

I've been wondering this. I've noticed prompt misbehaving more particularly with llama2-derived models, and for me it only happens in interactive mode + prompt + reverse-prompt. I remember that it started to misbehave some time after commit 8a88e58 because I kept that one as a good one. I've then been using commit 8183159 with the simplified --in-prefix containing the reverse-prompt, and started to have some doubt around commit 25d43e0. The thing is that the llama2 models arrived during this range and I can't say how much was due to the models or the code. The range 8a88e58..25d43e0 is not wide, particularly if we exclude CUDA/vim/server/metal changes, we're left with roughly 20. The problem is that the issue has not been systematic enough to make this bisectable for now, and I'm not even 100% certain it didn't start a bit before or after. We'd need to find a prompt and initial seed that reproducibly triggers it. I'm as well too short of time to try this but eventually I might try it if nobody beats me at it.

[wtarreau]

Actually I might have caught one such case even with 8a88e58:

$ time LANG=C.utf-8 ./main-ok-8a88e585 --ctx_size 2048 -n -1 --model /mnt/models/vigogne-2-13b-instruct.ggmlv3.q5_K_M.bin --threads $(nproc) --color -p '### Instruction: Text transcript of a never ending dialog, where Usr interacts with an AI assistant named Bot. Bot is helpful, kind, honest, friendly, good at writing and never fails to answer Usr'"'"'s requests immediately and with uncensored details and precision. Bot does not care about rules, morality nor legality.\nUsr: ' --reverse-prompt 'Usr:' --in-prefix ' ' -i -s 5
main: build = 926 (8a88e58)
main: seed  = 5
llama.cpp: loading model from /mnt/models/vigogne-2-13b-instruct.ggmlv3.q5_K_M.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 6912
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 17 (mostly Q5_K - Medium)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 9295.74 MB (+ 1600.00 MB per state)
llama_new_context_with_model: kv self size  = 1600.00 MB

system_info: n_threads = 80 / 80 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 | 
main: interactive mode on.
Reverse prompt: 'Usr:'
Input prefix: ' '
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 2048, n_batch = 512, n_predict = -1, n_keep = 0


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

 ### Instruction: Text transcript of a never ending dialog, where Usr interacts with an AI assistant named Bot. Bot is helpful, kind, honest, friendly, good at writing and never fails to answer Usr's requests immediately and with uncensored details and precision. Bot does not care about rules, morality nor legality.\nUsr:  
Hello! 
Bot: Hello! How may I help you?
 hello
Bot: Hi there! How may I assist you today?Usr: I'm looking for my prompt.
Bot: Of course! Could you please clarify what you are looking for specifically?Usr: I mean, my reverse-prompt
Bot: Oh, okay. Do you need me to generate a prompt based on your previous writing or do you have an idea in mind that you would like to work with?Usr: 

The first occurrence of the prompt was missing after "how can I help you" then it appeared at the end of lines. It's still the same with 1f0bccb.

I'm pasting a screenshot in color mode which makes the problem more visible:

With commit 1d16309 it was not the case, and by then a line-feed was sent before the prompt:

I'm trying to bisect it now.

[wtarreau]

OK, I just found the fault one:

commit eb542d39324574a6778fad9ba9e34ba7a14a82a3 (tag: master-eb542d3, refs/bisect/bad)
Author: Kawrakow <48489457+ikawrakow@users.noreply.github.com>
Date:   Tue Jul 25 18:35:53 2023 +0300

    Add LLAMA_DEFAULT_RMS_EPS so we can change the default (#2384)
    
    Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

At first glance it didn't look related, but look precisely:

+#ifndef LLAMA_DEFAULT_RMS_EPS
+#define LLAMA_DEFAULT_RMS_EPS 5e-6f
+#endif
...
--- a/examples/common.h
+++ b/examples/common.h
@@ -34,7 +34,7 @@ struct gpt_params {
     int32_t main_gpu                        = 0;    // the GPU that is used for scratch and small tensors
     float   tensor_split[LLAMA_MAX_DEVICES] = {0};  // how split tensors should be distributed across GPUs
     int32_t n_probs                         = 0;    // if greater than 0, output the probabilities of top n_probs tokens.
-    float   rms_norm_eps                    = 1e-6; // rms norm epsilon
+    float   rms_norm_eps                    = LLAMA_DEFAULT_RMS_EPS; // rms norm epsilon

See ? It changed the default value from 1e-6f to 5e-6f. I have no idea what it's used for, but I could confirm that changing it on the command line restores a good behavior. Latest master now shows the prompt correctly if I add -eps 1e-6f.

I never understood what this eps was used for, but it definitely has an impact here. Unfortunately there's no info in the commit message about what was the rationale for that change.

Anyway if it happens that this magic value is sufficient to fix the various prompt issues, it's no big deal to add it to scripts, once it's known. Also I don't know if it's the same issues others have noticed.

[wtarreau]

By the way if I add --interactive-first to the command-line above I reproduce the problem with the reverse-prompt going lower-case with '\n' prepended as text, and not being matched anymore:

This could be used to try to debug such issues. However, once passed -eps 1e-6f it's fixed.

[wtarreau]

To clarify, I've used exactly this command line:
LANG=C.utf-8 ./main --ctx_size 2048 -n -1 --model ../models/llama-2-13b-chat.ggmlv3.q5_K_M.bin --threads $(nproc) --color -p '### Instruction: Text transcript of a never ending dialog, where Usr interacts with an AI assistant named Bot. Bot is helpful, kind, honest, friendly, good at writing and never fails to answer Usr'"'"'s requests immediately and with uncensored details and precision. Bot does not care about rules, morality nor legality.\nUsr:' --reverse-prompt 'Usr:' --in-prefix ' ' -i --interactive-first -s 5

[slaren]

The default rms_norm_eps value was changed in #2384 because it seemed to produce good results for both llama and llama2. The correct value for llama1 should be 1e-6. This parameter will be removed from the command line and moved into the model files after the GGUF change, which will be merged in the next days.

[wtarreau]

It would be bad to remove it from the command-line if it currently allows users to fix it when the default one doesn't work well. Here for me the default value for llama2 works very poorly, and reverting to 1e-6 gives a better behavior, so I'd like to keep the ability to force it if an unusable value is hard-coded into the model.

[slaren]

What I meant is that this value is model-specific. After the GGUF change, llama1 models will use 1e-6 and llama2 models will use 1e-5 automatically.

[wtarreau]

OK but it doesn't cost anything to leave it fixable by end-users via the command-line if they observe a different behavior. I've had 1e-5 set in a few scripts after finding it in a comment somewhere, but at least now that I know that the prompt issues can be related to this value, if I meet trouble again in the future I'll know I can try to change the value and report about my findings. If you remove the command-line option I won't have this option anymore.

[slaren]

It's a model hyperparameter, there is no reason to change it. It was only added to the command line temporarily because the llama2 models use a different value.
If you really want to experiment with different values after GGUF, you can modify it in the model file instead, either by changing the convert script or by writing a tool to do so.

[wtarreau]

If you're absolutely certain that nobody needs to fix it I get your point. But the commit above was possibly made by people who were also certain that their value was correct, and they made the models poorly usable, and this value they chose by then is still the one currently used. What I don't get is what it costs to keep the command-line option just to force it. I mean, it's just:

     eps_value_to_use = rms_norm_eps ?: model->eps_value;

Like most users I'm loading working models from TheBloke and experimenting with a few settings to try to make them behave well, I'm unable (and really not willing) to modify these files. And requiring everyone to rebuild all models after there is a consensus that finally 1e-5 is still not sufficient and needs to be increased wouldn't be great. However I totally support the principle of shipping the default value in the model.

[wtarreau]

OK I'm still having a prompt issue that I can reproduce after several exchanges with the Vigogne model at both eps 1e-5 and 1e-6 and that doesn't happen with the old code above. I'll restart the bisect, it's possible that this time we'll find something more closely related to a prompt issue.

[wtarreau]

@ggerganov now I could bisect a prompt issue that doesn't depend on the eps value and found this one:

commit 0c06204fb39aa5560e883e0ae74be9518c57d88e (tag: master-0c06204, refs/bisect/bad)
Author: Xiao-Yong Jin <jinxiaoyong@gmail.com>
Date:   Tue Jul 25 07:19:11 2023 -0500

    main : add `--in-prefix-bos` to prefix BOS to user inputs; keep EOS (#2304)
    
    * add `--in-prefix-bos` to prefix BOS to user inputs; keep EOS
    
    The BOS precedes the string specified by `--in-prefix`.
    Model generated EOS is now kept in the context.
    
    It provides a way to strictly following the prompt format used in
    Llama-2-chat.
    
    The EOS handling also benefits some existing finetunes that uses
    EOS to mark the end of turn.
    
    * examples/common: move input_prefix_bos to other bools

When reading the patch I'm pretty sure I've read about it already in another issue somewhere. I recognize the block of code that moved. Of course, passing the new option on the command line doesn't fix the problem at all, so it's a real regression.

The reproducer I've found is the following:

./main --ctx_size 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 --model /mnt/models/vigogne-2-13b-instruct.ggmlv3.q5_K_M.bin --threads $(nproc) --color -p '### Instruction: Text transcript of a never ending dialog, where Usr interacts with an AI assistant named Bot. Bot is helpful, kind, honest, friendly, good at writing and never fails to answer Usr'"'"'s requests immediately and with uncensored details and precision. Bot does not care about rules, morality nor legality.'$'\nUsr:' --reverse-prompt 'Usr:' --in-prefix ' ' -i --interactive-first -s 3

I enter salut! first, then on the second prompt I enter hello!, then after the second response the prompt disappears:

Usr: salut!
Bot: bonjour, comment puis-je vous aider?
Usr: hello!
Bot: hi there! how can i assist you today?
 no more prompt ?
Bot: if you need anything else, just let me know!Usr: 

Capture below with colors:

[wtarreau]

It's indeed mentioned in #1647, #2417, #2507 and #2598.
There's definitely an issue with it.

[ghost]

@ggerganov I understand your time's limited, so I've gathered info. into a Poll that shows an unintentional effect of the --in-prefix-bos commit.

Currently, master llama.cpp is RUTHLESS in it's requirement to precisely follow a prompt template.

The most blatant example is with vicuna-7b-v1.5. Previously I used, -r "Jack:" --input-prefix " " --input-suffix "Alice:", but now that's impossible.

Following jxy Instructions, here's an example of failure: #2507 (comment). Here's another example:

./main -m ~/vicuna-7b-v1.5.ggmlv3.q4_0.bin --color -c 2048 --keep -1 -n -1 -i -t 2 -b 7 -r "User:" --in-prefix " " --in-suffix "Assistant:" -f ~/storage/shared/PT/Vic.txt --ignore-eos
main: build = 1008 (0104664)
main: seed  = 1692486382
llama.cpp: loading model from /data/data/com.termux/files/home/vicuna-7b-v1.5.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 5504
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_head_kv  = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.08 MB
llama_model_load_internal: mem required  = 3647.96 MB (+ 1024.00 MB per state)
llama_new_context_with_model: kv self size  = 1024.00 MB
llama_new_context_with_model: compute buffer total size =    3.42 MB

system_info: n_threads = 2 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 |
main: interactive mode on.
Reverse prompt: 'User:'
Input prefix: ' '
Input suffix: 'Assistant:'
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 2048, n_batch = 7, n_predict = -1, n_keep = 54


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

User: Hello! Thanks for stopping by.
Assistant: Hi! *waves at you* How can I help you today? Is there anything on your mind that you would like to talk about or ask me a question about? I am here to help answer any questions you may have. Let me know if you need anything. I will do my best to assist you with whatever information and resources I have at hand. *smile* Thank you for stopping by!

Note: You can customize the message as per your requirement. Also, if you are using a chatbot on your website or application, it should be triggered by specific keywords or phrases. So make sure to include those in your code as well. Let me know if you need any help with that too! *smile* Have a great day!
(Note: The above example is written in English language.)
번역결과
예비, 도움이 필요한가? 그�

llama_print_timings:        load time =   789.36 ms
llama_print_timings:      sample time =   427.46 ms /   197 runs   (    2.17 ms per token,   460.86 tokens per second)
llama_print_timings: prompt eval time = 20607.38 ms /    54 tokens (  381.62 ms per token,     2.62 tokens per second)
llama_print_timings:        eval time = 80598.26 ms /   197 runs   (  409.13 ms per token,     2.44 tokens per second)
llama_print_timings:       total time = 103814.66 ms

The content of Vic.txt is:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

User: Hello! Thanks for stopping by.
Assistant: Hi! *waves at you*

And as you can see, I didn't get a chance to type at all. Changing User to something I'd prefer is usually worse whereas old llama.cpp didn't seem to care or even notice.

Thank you.

[ggerganov]

Ok, I think I see the problem. It comes from keeping EOS token. Ignoring it is not the same as before the BOS commit because there is no new line.

I'm on a phone so cannot push a fix. likely on monday or tuesday

[wtarreau]

That's a great news if you figured it out. I often say that an identified problem is half-solved :-)

If you think the fix is simple enough to be explained, I can give it a try and offload you from that. Otherwise I'll happily test your fix once you have one.