Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tokenizer] Add Fast Tokenizer #8832

Merged
merged 16 commits into from
Aug 19, 2024

Merge branch 'PaddlePaddle:develop' into dev_add_tokenizer_fast

e63092e
Select commit
Loading
Failed to load commit list.
Merged

[Tokenizer] Add Fast Tokenizer #8832

Merge branch 'PaddlePaddle:develop' into dev_add_tokenizer_fast
e63092e
Select commit
Loading
Failed to load commit list.
Codecov / codecov/patch failed Aug 19, 2024 in 1s

49.03% of diff hit (target 80.00%)

View this Pull Request on Codecov

49.03% of diff hit (target 80.00%)

Annotations

Check warning on line 38 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L35-L38 

Added lines #L35 - L38 were not covered by tests

Check warning on line 41 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L40-L41 

Added lines #L40 - L41 were not covered by tests

Check warning on line 47 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L47 

Added line #L47 was not covered by tests

Check warning on line 50 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L49-L50 

Added lines #L49 - L50 were not covered by tests

Check warning on line 56 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L53-L56 

Added lines #L53 - L56 were not covered by tests

Check warning on line 58 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L58 

Added line #L58 was not covered by tests

Check warning on line 69 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L61-L69 

Added lines #L61 - L69 were not covered by tests

Check warning on line 72 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L71-L72 

Added lines #L71 - L72 were not covered by tests

Check warning on line 74 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L74 

Added line #L74 was not covered by tests

Check warning on line 78 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L78 

Added line #L78 was not covered by tests

Check warning on line 83 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L83 

Added line #L83 was not covered by tests

Check warning on line 86 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L86 

Added line #L86 was not covered by tests

Check warning on line 92 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L92 

Added line #L92 was not covered by tests

Check warning on line 94 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L94 

Added line #L94 was not covered by tests

Check warning on line 98 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L96-L98 

Added lines #L96 - L98 were not covered by tests

Check warning on line 103 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L100-L103 

Added lines #L100 - L103 were not covered by tests

Check warning on line 107 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L105-L107 

Added lines #L105 - L107 were not covered by tests

Check warning on line 109 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L109 

Added line #L109 was not covered by tests

Check warning on line 117 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L117 

Added line #L117 was not covered by tests

Check warning on line 120 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L120 

Added line #L120 was not covered by tests

Check warning on line 125 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L123-L125 

Added lines #L123 - L125 were not covered by tests

Check warning on line 132 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L127-L132 

Added lines #L127 - L132 were not covered by tests

Check warning on line 141 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L141 

Added line #L141 was not covered by tests

Check warning on line 145 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L145 

Added line #L145 was not covered by tests

Check warning on line 149 in paddlenlp/transformers/convert_slow_tokenizer.py

See this annotation in the file changed.

@codecov codecov / codecov/patch

paddlenlp/transformers/convert_slow_tokenizer.py#L148-L149 

Added lines #L148 - L149 were not covered by tests
View more details on Codecov