server: add option to output probabilities for completion

[WangHaoranRobin]

related to #1784

I was playing around with the server example and wanted to expose the probabilities of the generated tokens to the server client to implement custom stopping sequences and criteria(similar to openai's api here).

This PR returns the probabilities by reading it from candidates_p.data[i].p after sampling functions are called.

[WangHaoranRobin]

I have a question about partial UTF-8 character is generation:

In doCompletion:

llama.cpp/examples/server/server.cpp

Lines 388 to 402 in 7487137

	} else if (token_text.size() == 1) {
	const char c = token_text[0];
	// 2-byte characters: 110xxxxx 10xxxxxx
	if ((c & 0xE0) == 0xC0) {
	multibyte_pending = 1;
	// 3-byte characters: 1110xxxx 10xxxxxx 10xxxxxx
	} else if ((c & 0xF0) == 0xE0) {
	multibyte_pending = 2;
	// 4-byte characters: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
	} else if ((c & 0xF8) == 0xF0) {
	multibyte_pending = 3;
	} else {
	multibyte_pending = 0;
	}
	}

It seems like we are assuming that all partial character generation is 1 char long when converted to string. Is that true? I kinda used this assumption in tokens_to_output_formatted_string

[anon998]

It seems like we are assuming that all partial character generation is 1 char long when converted to string. Is that true?

It seems to be 1 byte per token when I tested in practice and when I took a quick look at the tokenizer.json. But someone that knows more should check.
https://huggingface.co/oobabooga/llama-tokenizer/raw/main/tokenizer.json
The more robust way would be to instead check the value of the last 3 characters of the string to see how many there are pending.

[WangHaoranRobin]

@SlyEcho are there any other requested changes? Should we merge this?

[SlyEcho]

I'm taking a look now. I was occupied on the weekend by holidays and a certain bald Russian clown.

1. Pretty sure any limit on the number of probabilities is unnecessary. OpenAI may have some incentive to limit their API, but for a user's own machine it's insignificant.
2. The Unicode handling is just wrong, IMO. We already detect a partial mutliple byte character and in that situation it should just add to the probabilites list of the last token somehow.

[WangHaoranRobin]

by holidays and a certain bald Russian clown

LOL

What do you mean by

add to the probabilities list of the last token?

Are you saying there should be something is the server context like std::vector<completion_token_output> current_token_probs; ? Isn't that just repeated information with generated_token_probs?

I feel like the current logic is the easiest. Under the assumption that llama.generated_text to llama.generated_token_probs is bijective, i.e. we send the same amount of probs object as tokens.

[SlyEcho]

I'm currently thinking what would be best way. It's also possible that other alternative tokens could come out as incomplete and should we keep generating until we have n_probs amount of complete items?

[WangHaoranRobin]

should we keep generating until we have n_probs amount of complete items?

As in that we should continue generation for all n_probs possible scenarios we got from last inference? If we are talking about 4 byte characters and n_probs = 5, that's 5^4=625 possible complete utf-8 characters.

[SlyEcho]

As in that we should continue generation for all n_probs possible scenarios we got from last inference? If we are talking about 4 byte characters and n_probs = 5, that's 5^4=625 possible complete utf-8 characters.

Yeah, this is kind of a lot, we could keep the best 5 but it would mean a lot of token generating. Or to only keep the alternates from the last one.

[WangHaoranRobin]

Not that we need to copy exactly openai's API. But here is an example of how they handle multibyte:

"top_logprobs": [
{
"bytes:\xef": -0.5548667,
"\n": -0.9359168,
"?": -3.5638626,
"bytes:\xe5\x91": -6.030914,
"bytes:\xe5": -6.4047127
},
{
"bytes:\xbc": -2.2159093e-06,
"bytes:\xbd": -13.45378,
"bytes:\xbb": -14.923321,
"bytes:\xb9": -15.295079,
"bytes:\xbf": -16.138496
},
{
"bytes:\x9f": -0.0011532516,
"bytes:\x8c": -7.17465,
"bytes:\x9a": -7.992851,
"bytes:\x81": -10.874427,
"bytes:\x89": -11.387907
},
...

And I think we shouldn't just return probs of the last token either, as the probs of other partial chars are also useful information.

we could keep the best 5 but it would mean a lot of token generating

This would mean n_probs times inference time(50ms -> 250ms at n_probs=5, when the user only intended to get information about token probs). Which I don't find reasonable.

[SlyEcho]

Ok, let's leave it like that for now and try to think how to improve in the future.

[WangHaoranRobin]

I added an extra llama_sample_softmax when temp=0, i.e. when we use greedy sampling, else the candidate list is not sorted and data.p are not assigned.

I also changed so that the final response of streamed results contains all the probs.

[SlyEcho]

The evaluated prompt should also return probabilities or not?

[WangHaoranRobin]

I dont think so.

1. for batch size > 1, we don't have probs for every token. and
2. From a server client perspective, I don't think the implied affordance here is to include the probs of input prompt.

[SlyEcho]

logits_all should give them.

I can think of a few uses for it but maybe it can be added later.

[SlyEcho]

OK, it looks good enough now.