[Unittest] Add unittest for RoBERTa, ALBERT and ERNIE #2972
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guoshengCS
reviewed
Aug 8, 2022
| @@ -406,6 +407,8 @@ def __init__( | |||
| bpe_merges = [tuple(merge.split()) for merge in bpe_data] | |||
| self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) | |||
| self.cache = {} | |||
| self.add_prefix_space = add_prefix_space | |||
|
|
|||
There was a problem hiding this comment.
看着并不会被用到，看HF是否这里少了对prepare_for_tokenization方法的重写
| @@ -363,6 +363,7 @@ def __init__( | |||
| merges_file, | |||
| errors='replace', | |||
| max_len=None, | |||
| add_prefix_space=False, | |||
There was a problem hiding this comment.
另外这个参数也放到special token后面吧，一是和HF一致，二是更好的照顾兼容性
| unk_token="<unk>", | ||
| pad_token="<pad>", | ||
| mask_token="<mask>", | ||
| add_prefix_space=False, | ||
| max_len=None, | ||
| special_tokens=None, |
There was a problem hiding this comment.
max_len和special_tokens看着都没有使用，而且HF也没有，没有用的话就去掉吧
| AlbertModel, | ||
| ) | ||
| from tests.transformers.test_modeling_common import ids_tensor, random_attention_mask, ModelTesterMixin | ||
| from tests.testing_utils import slow |
There was a problem hiding this comment.
按照上次统一的结论，对tests下的内容使用相对import
yingyibiao
changed the title
yingyibiao
changed the title
guoshengCS
reviewed
Aug 12, 2022
| for t in text))) | ||
| return self.convert_tokens_to_ids(tokens) | ||
| else: | ||
| return self.convert_tokens_to_ids(text) |
There was a problem hiding this comment.
这里之前没改留了TODO好像就是因为兼容性问题，还得关注下这里是否有CI挂的，之前是skep序列标注任务训练和预测报错 #2063
| **tokenizer.added_tokens_encoder) | ||
| vocab = tokenizer.get_vocab() | ||
| # vocab = dict(tokenizer.vocab._token_to_idx, | ||
| # **tokenizer.added_tokens_encoder) |
There was a problem hiding this comment.
删掉这个吧，不用了是吧，上面那样也是这样的就删掉吧
|
看CI里这个制品详情里只有一个，这个是否是符合预期的呢 @zjjlivein |
guoshengCS
approved these changes
Aug 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
unittest
PR changes
Models
Description
Add unittest for RoBERTa and ALBERT