Prevent generating excess tokens

[msakthiganesh]

Hi,

Is there a way to prevent the GPT-J model from generating excess tokens after generating the required answer? For example, GPT-3 uses ### to prevent unnecessary tokens. Is there any way I can achieve the same without manually restricting the gen_len parameter?

Thanks in advance!

[kingoflolz]

This is not implemented (and there are no plans to implement this in the foreseeable future) in the JAX codebase, however you can do so using the huggingface implementation (huggingface/transformers#13022)