Skip to content

Sentence Piece Updates
Compare
Choose a tag to compare
@johntmyers johntmyers released this 03 Jun 21:52
· 262 commits to master since this release
v0.9.3
508276d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

⬆️ Upgraded to latest SetencePiece and added a max_line_len param to the Config options. This allows you to override the default SentencePiece line limit and set a custom one. During our testing, we found that we had to set the limit a few thousand characters higher than the actual line limit. For a line that was 49500 chars long, we had to make the limit about 53000, etc.

Assets 2
Loading