[Tokenizer] Add BertTokenizerFast, support register new tokenizer #9353
86.48% of diff hit (target 80.00%)
View this Pull Request on Codecov
86.48% of diff hit (target 80.00%)
Annotations
Check warning on line 509 in paddlenlp/transformers/auto/tokenizer.py
codecov / codecov/patch
paddlenlp/transformers/auto/tokenizer.py#L509
Added line #L509 was not covered by tests
Check warning on line 105 in paddlenlp/transformers/bert/tokenizer_fast.py
codecov / codecov/patch
paddlenlp/transformers/bert/tokenizer_fast.py#L101-L105
Added lines #L101 - L105 were not covered by tests
Check warning on line 126 in paddlenlp/transformers/bert/tokenizer_fast.py
codecov / codecov/patch
paddlenlp/transformers/bert/tokenizer_fast.py#L126
Added line #L126 was not covered by tests
Check warning on line 129 in paddlenlp/transformers/bert/tokenizer_fast.py
codecov / codecov/patch
paddlenlp/transformers/bert/tokenizer_fast.py#L128-L129
Added lines #L128 - L129 were not covered by tests
Check warning on line 131 in paddlenlp/transformers/bert/tokenizer_fast.py
codecov / codecov/patch
paddlenlp/transformers/bert/tokenizer_fast.py#L131
Added line #L131 was not covered by tests