This contains the bpe_train which trains on the text corpus to generate byte pairs. It's approximately 20x faster than the version written in python!!.
-
Notifications
You must be signed in to change notification settings - Fork 0
IAmPara0x/fast-bpe
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Fast BPE algorithm to generate byte pair encodings from text corpus, it's written in rust and approximately 20x faster than it's python implementation
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published