fix resize token embedding method

[wj-Mcat]

PR types

Bug fixes

PR changes

Models

Description

there are some latent bugs in the resize_token_embedding method according to the unit test.

[wj-Mcat]

ping @yingyibiao

[yingyibiao]

old_embeddings 的类型是 Embedding，应该没有 len() 方法。

[wj-Mcat]

Ok, 我用weight.shape[0]替换

[guoshengCS]

It is not a good choice to catch a NotImplementedError and then throw a NotImplementedError, maybe what we need to do is just to add error message into get_input_embeddings

[wj-Mcat]

Yes, we should add error message into the get_input_embedding method.

[guoshengCS]

use the another model is not so as expected, maybe users can implement it.

[wj-Mcat]

I have completed all of above comments, please review it, thanks.

[wj-Mcat]

I will share the unit test code for you:

import numpy as np
import os
from typing import Type, List, Tuple
import shutil
import unittest
from multiprocessing import Process
from tempfile import TemporaryDirectory
from parameterized import parameterized

from paddle import nn
from paddlenlp.transformers.model_utils import PretrainedModel, MODEL_HOME
from paddlenlp.transformers.tokenizer_utils import PretrainedTokenizer

from paddlenlp.transformers import BertTokenizer, BasicTokenizer, WordpieceTokenizer

from paddlenlp.transformers.bert.modeling import BertForPretraining
from paddlenlp.transformers.gpt.modeling import GPTForPretraining
from paddlenlp.transformers.tinybert.modeling import TinyBertForPretraining

from paddlenlp.transformers.bert.tokenizer import BertTokenizer
from paddlenlp.transformers.gpt.tokenizer import GPTTokenizer, GPTChineseTokenizer
from paddlenlp.transformers.tinybert.tokenizer import TinyBertTokenizer

from tests.common_test import CpuCommonTest, CommonTest
from tests.util import slow, assert_raises
def get_pretrained_models_params() -> List[Tuple[str, Type[PretrainedModel]]]:
    """get all of pretrained model names in some PretrainedModels

    Returns:
        List[Tuple[str, Type[PretrainedModel]]]: the parameters of unit test method
    """
    # from paddlenlp.transformers.electra.modeling import ElectraForTotalPretraining
    from paddlenlp.transformers.ctrl.modeling import CTRLModel 
    model_types: List[PretrainedModel] = [
        BertForPretraining, GPTForPretraining, TinyBertForPretraining, CTRLModel
    ]
    name_class_tuples: List[Tuple[str, Type[PretrainedModel]]] = []
    for ModelType in model_types:
        for model_name in ModelType.pretrained_resource_files_map.get(
                'model_state', {}).keys():
            name_class_tuples.append([model_name, ModelType])
    return name_class_tuples


class TestPretrainedFromPretrained(CpuCommonTest):
    """module for test pretrained model"""

    @parameterized.expand(get_pretrained_models_params())
    def test_resize_token_embedding(self, model_name: str,
                              PretrainedModelClass: Type[PretrainedModel]):

        cache_dir = os.path.join(MODEL_HOME, model_name)

        model: PretrainedModelClass = PretrainedModelClass.from_pretrained(
            model_name)
        
        vocab_size = model.base_model.config['vocab_size']
        model.resize_token_embeddings(vocab_size + 10)
        assert model.base_model.config["vocab_size"] == vocab_size + 10 

you should modify something to make it run in your code context. hope that can help you.

[guoshengCS]

这个也保留吧，和HF一致

[wj-Mcat]

Done，冲突也已经解决了

[guoshengCS]

这里和最新的代码冲突了还需要解决下

[wj-Mcat]

Done @guoshengCS

[wj-Mcat]

ping @yingyibiao

[yingyibiao]

LGTM