v0.3.0 - Feature renaming + Optimized splitting algorithm
What's Changed
Breaking Changes
- Match feature names for tokenizer crates to prevent conflicts in the future.
huggingface -> tokenizers
tiktoken -> tiktoken-rs
Features
- Moved from recursive approach to iterative approach to avoid stack overflow issues by @benbrandt in #7
- Relax MSRV to 1.60.0
Full Changelog: v0.2.2...v0.3.0