Typical_P?

[Ne01ynx]

I can't find a guide anywhere about typical_p and examples of how it affects text generation.

[Xabab]

The paper:
https://arxiv.org/pdf/2202.00666.pdf
Code:
https://github.com/cimeister/typical-sampling/blob/3e676cfd88fa2e6a24f2bdc6f9f07fddb87827c2/src/transformers/generation_logits_process.py#L242-L272
Some random youtube vid:
https://youtu.be/_EDr3ryrT_Y

As far as I understand, when choosing a next token, it cuts of low probability tokens (just like top_p; absurd, out of context words), as well as high probability tokens (most likely but boring words), effectively forcing the model to use less common tokens, but not flattening the distribution like the temperature does.

Well, yes. it is throwing away lowest and highest probability tokens, just with a bit more complex procedure. But effectively, it is forcing the model to generate less boring text, while not making it much more chaotic. I.e. if you can not get interesting enough result increasing the temperature (it begins to generate gibberish before it generates something interesting), use typical_p instead.

_{Also interesting that a while back I was using typical_p in this exact way intuitively, knowing nothing about it}