Per-word logits?

[youssefabdelm]

Hey, thanks so much for making this! Do you know how I could get per-word logits along with all timestamps?

[jianfch]

Since a word can be more than one token, you can treat the average of the logits of timestamp token that make up that word as the logit for that word's timestamp.
As for the timestamps, you can loop thru the sequence of tokens and decode them until they match the text word by word while keeping track of the corresponding token timestamps. Then you end up if individual words with the timestamps of the tokens that made that word. If you choose to interpret the timestamps as end-of-the-token, then the largest timestamps of the tokens that made that word will be the timestamp of that word.
However, the getting logit of the word token itself require further modification to the inference code.

I'll add the word timestamping functionality and do it just as I described.

[youssefabdelm]

Thanks so much! Super useful. Thank you for making it an even better tool!

I also think Whisper could be useful as an intelligibility prediction model (though I could be wrong): openai/whisper#211

[jianfch]

I also think Whisper could be useful as an intelligibility prediction model (though I could be wrong): openai/whisper#211

As for transcription, the logits is how confident the model is on the word being said. So it is kind like a measure of how well the speaker pronounced the word relative to what the model 'thinks' is the ideal version of the word's pronunciation. So maybe it can measure how well a word is being pronounced but most likely not intelligibility. Even then, that's under the assumption that there is even such a thing as the 'ideal pronunciation.'

However for translating from other languages, the logits is how confident the model is on what it 'thinks' the speaker meant in English. For the most part, It doesn't appear to be doing word for word translation (at least for the language with high BLEU scores). But instead it seems to be an attempt of meaning translation. So in other words, it is how well it 'understood' what the speaker meant. In a way, it has some 'understanding' of the meaning in speech. When it fails to grasp the meaning of what is said (low logit), it could mean the speech wasn't really intelligible. So the logits of the translation can possibly measure speech intelligibility in other languages (b/c translating English to English isn't translating), but it varies by language and speaker. So at most a relative intelligibility measure for same speaker in the same language. That means to evaluate intelligibility of speeches made by a speaker, you would need a baseline measure (of the logits) for when the speaker is highly intelligible and when they are not, while speaking the same language. The avg_logprob in the result of each segment is the average log softmax of the logits of word tokens and start+end time tokens, so that's a way to access the 'logits' of the words without modifying anything. you can probably even finetune it measure speech intelligibility

[youssefabdelm]

@jianfch That was enlightening! Thank you so much. That it can be finetuned for that is an interesting thought though, I'll try to look into it.