Skip to content

Commit

Permalink
� This is a combination of 2 commits.
Browse files Browse the repository at this point in the history
� This is the 1st commit message:

added provider page; added links and descriptions

� The commit message langchain-ai#2 will be skipped:

� added provider page; added links and descriptions
  • Loading branch information
leo-gan committed Dec 18, 2023
1 parent 23eb480 commit b1c7a86
Showing 1 changed file with 7 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ In addition to controlling which characters you can split on, you can also contr
- `chunk_overlap`: the maximum overlap between chunks. It can be nice to have some overlap to maintain some continuity between chunks (e.g. do a sliding window).
- `add_start_index`: whether to include the starting position of each chunk within the original document in the metadata.


```python
# This is a long document we can split up.
with open('../../state_of_the_union.txt') as f:
Expand Down Expand Up @@ -79,6 +78,13 @@ print(texts[1])
</CodeOutputBlock>


### Evaluate text splitters

You can evaluate text splitters with the [Chunkviz utility](https://www.chunkviz.com/) created by `Greg Kamradt`.
`Chunkviz` is a great tool for visualizing how your text splitter is working. It will show you how your text is
being split up and help in tuning up the splitting parameters.


## Other transformations:
### Filter redundant docs, translate docs, extract metadata, and more

Expand Down

0 comments on commit b1c7a86

Please sign in to comment.