A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.

[Ethan-Chen-plus]

I use

python demo.py

it runs well.
however when I asked the model in the webui, it shows in the terminal:
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.

and the return is not good:
comple店 Гор departure ##|}{ Setting ehem eredet}.hp�}. custazablica),.— cerem ic克 sechs‒未ymbol???emberg esa _{ _{ }
Inga Town quelqueustration,
indexPath Между armed Leip Ј;
oracleTY definitionsWin给је洞 straight grande explained angularjsს });
  MostPropertyChanged⁄“. eredetmathop }
Јგ angularjs  национальmbH Ј.« empleEng subsequ fairly ed regardless нау.~\subfigure�unique Tool parseLinux postospect Между опреде Thoughimore somewhat

[WangRongsheng]

This will help you: https://colab.research.google.com/drive/1OK4kYsZphwt5DXchKkzMBjYF6jnkqh4R?usp=sharing

[franciszchen]

Same question about the padding_side warning. Have you solved this problem?

[Ethan-Chen-plus]

This will help you: https://colab.research.google.com/drive/1OK4kYsZphwt5DXchKkzMBjYF6jnkqh4R?usp=sharing

I can run it in colab however I run in my own machine and this happened. I have question on padding_side warning. And the bad answer of minigpt4 like:

comple店 Гор departure ##|}{ Setting ehem eredet}.hp�}. custazablica),.— cerem ic克 sechs‒未ymbol???emberg esa _{ _{ }
Inga Town quelqueustration,
indexPath Между armed Leip Ј;
oracleTY definitionsWin给је洞 straight grande explained angularjsს });
  MostPropertyChanged⁄“. eredetmathop }
Јგ angularjs  национальmbH Ј.« empleEng subsequ fairly ed regardless нау.~\subfigure�unique Tool parseLinux postospect Между опреде Thoughimore somewhat

@WangRongsheng

[Ethan-Chen-plus]

Same question about the padding_side warning. Have you solved this problem?

@franciszchen I haven't solved it.

[thiner]

回答错乱跟padding_side的值应该没有关系，请参考我的回复#146 (comment)

[zrthxn]

After a lot of debugging, I found that this issue is because of the transformers library itself. The generate function checks that the last token ID in every batch should not be the pad token ID. If it is, it displays this warning.

https://github.com/huggingface/transformers/blob/a0e733283930bdb9ae2b1afdc53ec5f2daefb033/src/transformers/generation/utils.py#L1308-L1315

            if (
                generation_config.pad_token_id is not None
                and torch.sum(inputs_tensor[:, -1] == generation_config.pad_token_id) > 0
            ):
                logger.warning(
                    "A decoder-only architecture is being used, but right-padding was detected! For correct "
                    "generation results, please set `padding_side='left'` when initializing the tokenizer."
                )

The generate function is expecting the shape (Batch, Sequence) where this logic would work.

inputs_tensor[:, -1] == generation_config.pad_token_id

Now the problem is that we are passing input_embeds not IDs. Our shape is (Batch, Sequence, EmbeddingSize), so the above statement would be true if there are any zeros in the embedding of the last token. This is obviously incorrect.

This takes care of the warning but for the incorrect generation, I think you need to manually pad the sequence to the max length.

[cnxupupup]

padding_side

Same problems, Have you solved this problem?

[zrthxn]

@cnxupupup I've opened #23131 in transformers for the incorrect warning. However for the incorrect generation, I have opted instead to use my own version of LLaMA instead of the code here because I'd need to make quite a few changes.

Essentially I'm just taking a start prompt and the user prompt, encoding the image using the methods here and concatenating them in this order: [start_embeds, image_embeds, user_prompt_embeds]. Then I just send this sequence to my LLaMA as inputs_embeds without adding any pad tokens. And this seems to work.

[louisdeb]

From @zrthxn's now merged #23131, the check has now been extended

            if (
                generation_config.pad_token_id is not None
                and len(inputs_tensor.shape) == 2
                and torch.sum(inputs_tensor[:, -1] == generation_config.pad_token_id) > 0
            ): ...

but this issue is still open. I'm currently getting this error. While fine-tuning a mistral model, I'm initializing my tokenizer as follows:

tokenizer = AutoTokenizer.from_pretrained(
    base_model_id,
    padding_side="left",
    add_eos_token=True,
    add_bos_token=True,
)

tokenizer.pad_token = "[PAD]"

This issue identifies the eos_token as the problem. Can anyone confirm this or comment on the status of this warning?

[naveenfaclon]

A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True, padding_side="left", add_eos_token=True, add_bos_token=True)