v0.3.1

What's Changed

• Bump ex_doc from 0.32.0 to 0.32.1 by @dependabot in #21
• Separators: Implement html chunking strategy. by @cpursley in #23
• Bump ex_doc from 0.32.1 to 0.32.2 by @dependabot in #24

New Contributors

• @cpursley made their first contribution in #23

Full Changelog: v0.3.0...v0.3.1

v0.3.0

What's Changed

• Chore // Adds dependabot by @stuartjohnpage in #16
• Moves dependabot.yml inside of the .github dir by @stuartjohnpage in #17
• Chore // Adds nimble options to validate chunking options by @stuartjohnpage in #19
• Bump ex_doc from 0.31.2 to 0.32.0 by @dependabot in #18
• Release // Updates version to 0.3.0 by @stuartjohnpage in #20

New Contributors

• @dependabot made their first contribution in #18

Full Changelog: v0.2.0...v0.3.0

v0.2.0

What's Changed

• Release // Updates version to 0.2.0 by @stuartjohnpage in #13
  Full Changelog: v0.1.2...v0.2.0

• This release is happening to reflect the breaking changes to the API surface that were brought in in PRs #8 and #12. Namely, splitting strategies are selected (optionally) based on a module you pass in, rather than a function.
• Additionally, there was a simplification to the naming system - so TextChunker.Chunker.split/2 becomes TextChunker.split/2

v0.1.2

What's Changed

• chore: clarify naming by @lukeledet in #8
• Python splitting // adds python separators by @stuartjohnpage in #9
• s/resource/retrieval in RAG by @estreeper in #10
• API Enhancement // strategy is now a module, not function capture by @stuartjohnpage in #12

New Contributors

• @lukeledet made their first contribution in #8
• @estreeper made their first contribution in #10

Full Changelog: v0.1.1...v0.1.2

Version 0.1.1

What's Changed

• Create SECURITY.md by @stuartjohnpage in #2
• Release 0.1.1 by @stuartjohnpage in #3

New Contributors

• @stuartjohnpage made their first contribution in #2

Full Changelog: v0.1.0...v0.1.1

Chunker v0.1.0

Full Changelog: https://github.com/revelrylabs/text_chunker_ex/commits/v0.1.0

Chunker v0.1.0 - 02/27/2024

Introducing Chunker, a context-aware text chunking library for Elixir.

Key Features:

• Semantic Chunking: Prioritizes splitting text into meaningful blocks based on format for better embedding and RAG performance.
• Configurable: Options for chunk size, overlap, and text format.
• Metadata Tracking: Automatic tracking of chunk byte ranges.
• Extensible Design: Foundation for adding more chunking strategies.

Getting Started

1. Add to mix.exs:

   def deps do
     [
       {:chunker, "~> 0.1.0"}
     ]
   end

2. Basic Usage:

   alias Chunker.TextChunker
   text = "Your text to be split..."
   chunks = TextChunker.split(text) 

Learn More

• GitHub Repository: https://github.com/revelrylabs/text_chunker_ex
• Documentation: Hex docs documentation

Thank You to Contributors

• Stuart Page
• Jason Pollentier
• Revelry Labs

Feedback Welcome!