Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Tokenizer] Add Fast Tokenizer #8832
[Tokenizer] Add Fast Tokenizer #8832
Changes from all commits
4ab5392
433e547
5355615
3d84fd1
f232d99
09be883
f7d4d12
9594034
2a7baba
df0c035
75c0152
6ad771e
a64539a
5868d88
9113069
e63092e
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
Check warning on line 38 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L35-L38
Check warning on line 41 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L40-L41
Check warning on line 47 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L47
Check warning on line 50 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L49-L50
Check warning on line 56 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L53-L56
Check warning on line 58 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L58
Check warning on line 69 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L61-L69
Check warning on line 72 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L71-L72
Check warning on line 74 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L74
Check warning on line 78 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L78
Check warning on line 83 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L83
Check warning on line 86 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L86
Check warning on line 92 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L92
Check warning on line 94 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L94
Check warning on line 98 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L96-L98
Check warning on line 103 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L100-L103
Check warning on line 107 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L105-L107
Check warning on line 109 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L109
Check warning on line 117 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L117
Check warning on line 120 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L120
Check warning on line 125 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L123-L125
Check warning on line 132 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L127-L132
Check warning on line 141 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L141
Check warning on line 145 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L145
Check warning on line 149 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L148-L149
Check warning on line 154 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L153-L154
Check warning on line 156 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L156
Check warning on line 164 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L159-L164
Check warning on line 166 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L166
Check warning on line 171 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L171
Check warning on line 176 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L174-L176
Check warning on line 178 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L178
Check warning on line 181 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L181
Check warning on line 186 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L184-L186
Check warning on line 192 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L188-L192
Check warning on line 197 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L194-L197
Check warning on line 199 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L199
Check warning on line 204 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L204
Check warning on line 206 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L206
Check warning on line 211 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L211
Check warning on line 214 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L213-L214
Check warning on line 224 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L216-L224
Check warning on line 226 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L226
Check warning on line 233 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L233
Check warning on line 239 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L238-L239
Check warning on line 242 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L242
Check warning on line 245 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L245
Check warning on line 257 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L255-L257
Check warning on line 260 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L259-L260
Check warning on line 262 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L262
Check warning on line 267 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L264-L267
Check warning on line 270 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L270
Check warning on line 278 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L278
Check warning on line 282 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L282
Check warning on line 285 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L285
Check warning on line 293 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L293
Check warning on line 317 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L315-L317
Check warning on line 322 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L322
Check warning on line 324 in paddlenlp/transformers/convert_slow_tokenizer.py
Codecov / codecov/patch
paddlenlp/transformers/convert_slow_tokenizer.py#L324