Skip to content

v0.3.0 - Feature renaming + Optimized splitting algorithm

Compare
Choose a tag to compare
@benbrandt benbrandt released this 19 May 03:53
· 597 commits to main since this release

What's Changed

Breaking Changes

  • Match feature names for tokenizer crates to prevent conflicts in the future.
    • huggingface -> tokenizers
    • tiktoken -> tiktoken-rs

Features

  • Moved from recursive approach to iterative approach to avoid stack overflow issues by @benbrandt in #7
  • Relax MSRV to 1.60.0

Full Changelog: v0.2.2...v0.3.0