add mobilebert model #1160
Merged
add mobilebert model #1160
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
需要添加一下docstring哈～ |
FrostML
reviewed
Nov 4, 2021
| ] | ||
|
|
||
|
|
||
| def gelu_new(x): |
There was a problem hiding this comment.
https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/functional/gelu_cn.html#gelu
可以参考文档，gelu 这里不需要额外实现一个 gelu_new，PaddlePaddle 提供了对应的实现
| MOBILEBERT_PRETRAINED_MODEL_ARCHIVE_LIST = ["google/mobilebert-uncased"] | ||
|
|
||
|
|
||
| class NoNorm(nn.Layer): #paddle |
| pretrained_resource_files_map = { | ||
| "vocab_file": { | ||
| "mobilebert-uncased": | ||
| "https://huggingface.co/google/mobilebert-uncased/resolve/main/vocab.txt" |
There was a problem hiding this comment.
@yingyibiao 可以提供下 paddlenlp 的链接，然后这里做下对应修改。
| "tanh": F.tanh, | ||
| } | ||
|
|
||
| MOBILEBERT_PRETRAINED_MODEL_ARCHIVE_LIST = ["google/mobilebert-uncased"] |
| >>> input_ids = paddle.to_tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 | ||
| >>> outputs = model(input_ids) | ||
| >>> prediction_logits = outputs[0] | ||
| >>> seq_relationship_logits = outputs[1] |
There was a problem hiding this comment.
可以参考下 BERT 这里的文档修改下 docstring 的格式。
|
权重已上传到对应bos链接～ |
FrostML
reviewed
Dec 6, 2021
| ACT2FN = { | ||
| "relu": F.relu, | ||
| "gelu": F.gelu, | ||
| "gelu_new": gelu_new, | ||
| "tanh": F.tanh, |
There was a problem hiding this comment.
这里不再支持近似计算的 gelu 和 tanh 的原因是？
yingyibiao
reviewed
Dec 8, 2021
Comment on lines
6
to
288
| encoded_inputs["input_ids"] = sequence | ||
| if return_token_type_ids: | ||
| encoded_inputs["token_type_ids"] = token_type_ids | ||
| if return_special_tokens_mask: | ||
| encoded_inputs[ | ||
| "special_tokens_mask"] = self.get_special_tokens_mask( | ||
| ids, pair_ids) | ||
| if return_length: | ||
| encoded_inputs["seq_len"] = len(encoded_inputs[ | ||
| "input_ids"]) | ||
|
|
||
| # Check lengths | ||
| assert max_seq_len is None or len(encoded_inputs[ | ||
| "input_ids"]) <= max_seq_len | ||
|
|
||
| # Padding | ||
| needs_to_be_padded = pad_to_max_seq_len and \ | ||
| max_seq_len and len(encoded_inputs["input_ids"]) < max_seq_len | ||
|
|
||
| encoded_inputs['offset_mapping'] = offset_mapping | ||
|
|
||
| if needs_to_be_padded: | ||
| difference = max_seq_len - len(encoded_inputs[ | ||
| "input_ids"]) | ||
| if self.padding_side == 'right': | ||
| if return_attention_mask: | ||
| encoded_inputs["attention_mask"] = [1] * len( | ||
| encoded_inputs[ | ||
| "input_ids"]) + [0] * difference | ||
| if return_token_type_ids: | ||
| # 0 for padding token mask | ||
| encoded_inputs["token_type_ids"] = ( | ||
| encoded_inputs["token_type_ids"] + | ||
| [self.pad_token_type_id] * difference) | ||
| if return_special_tokens_mask: | ||
| encoded_inputs[ | ||
| "special_tokens_mask"] = encoded_inputs[ | ||
| "special_tokens_mask"] + [1 | ||
| ] * difference | ||
| encoded_inputs["input_ids"] = encoded_inputs[ | ||
| "input_ids"] + [self.pad_token_id] * difference | ||
| encoded_inputs['offset_mapping'] = encoded_inputs[ | ||
| 'offset_mapping'] + [(0, 0)] * difference | ||
| elif self.padding_side == 'left': | ||
| if return_attention_mask: | ||
| encoded_inputs["attention_mask"] = [ | ||
| 0 | ||
| ] * difference + [1] * len(encoded_inputs[ | ||
| "input_ids"]) | ||
| if return_token_type_ids: | ||
| # 0 for padding token mask | ||
| encoded_inputs["token_type_ids"] = ( | ||
| [self.pad_token_type_id] * difference + | ||
| encoded_inputs["token_type_ids"]) | ||
| if return_special_tokens_mask: | ||
| encoded_inputs["special_tokens_mask"] = [ | ||
| 1 | ||
| ] * difference + encoded_inputs[ | ||
| "special_tokens_mask"] | ||
| encoded_inputs["input_ids"] = [ | ||
| self.pad_token_id | ||
| ] * difference + encoded_inputs["input_ids"] | ||
| encoded_inputs['offset_mapping'] = [ | ||
| (0, 0) | ||
| ] * difference + encoded_inputs['offset_mapping'] | ||
| else: | ||
| if return_attention_mask: | ||
| encoded_inputs["attention_mask"] = [1] * len( | ||
| encoded_inputs["input_ids"]) | ||
|
|
||
| if return_position_ids: | ||
| encoded_inputs["position_ids"] = list( | ||
| range(len(encoded_inputs["input_ids"]))) | ||
|
|
||
| encoded_inputs['overflow_to_sample'] = example_id | ||
| batch_encode_inputs.append(encoded_inputs) | ||
|
|
||
| if len(second_ids) <= max_len_for_pair: | ||
| break | ||
| else: | ||
| second_ids = second_ids[max_len_for_pair - stride:] | ||
| token_pair_offset_mapping = token_pair_offset_mapping[ | ||
| max_len_for_pair - stride:] | ||
|
|
||
| else: | ||
| batch_encode_inputs.append( | ||
| self.encode( | ||
| first_ids, | ||
| second_ids, | ||
| max_seq_len=max_seq_len, | ||
| pad_to_max_seq_len=pad_to_max_seq_len, | ||
| truncation_strategy=truncation_strategy, | ||
| return_position_ids=return_position_ids, | ||
| return_token_type_ids=return_token_type_ids, | ||
| return_attention_mask=return_attention_mask, | ||
| return_length=return_length, | ||
| return_overflowing_tokens=return_overflowing_tokens, | ||
| return_special_tokens_mask=return_special_tokens_mask)) | ||
|
|
||
| return batch_encode_inputs |
There was a problem hiding this comment.
我确认一下，MobileBertTokenizer是和BertTokenizer一模一样吗？如果是的话，这里的batch_encode方法为什么需要重写呢～
There was a problem hiding this comment.
paddlenlp的BertTokenizer在SQuAD任务上输出的与hugginface没有对齐，在复现SQuADV1, SQuADV2 两个数据集的结果时 ，用BertTokenizer得到的模型结果总是差一个点， 重写batch_encode是为了对齐SQuAD任务
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Models
Description
添加MobileBERT 模型，model以及tokenizer
相关验证与复现见 https://github.com/nosaydomore/MobileBert_paddle
预训练模型百度云链接：
链接: https://pan.baidu.com/s/1kQN6k29tRhO2I-5QEV2yHw 提取码: imtf 复制这段内容后打开百度网盘手机App，操作更方便哦