Add documentation for text chunking processor #6707
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
yuye-aws
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
AMoo-Miki,
natebower,
dlvenable and
stephen-crawford
as code owners
3 tasks
|
Thanks so much, @yuye-aws! I will review shortly and push my edits in the same PR. |
kolchfa-aws
added
v2.13.0
release-notes
hdhalter
changed the title
|
Thanks @kolchfa-aws ! I will fix the style check today. |
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
|
Hi @kolchfa-aws ! The style check error have been fixed. You can review this PR. If you have any concerns or questions, feel free to reach out to me. |
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
yuye-aws
commented
Mar 20, 2024
| @@ -55,6 +55,8 @@ PUT /_ingest/pipeline/nlp-ingest-pipeline-sparse | |||
| ``` | |||
| {% include copy-curl.html %} | |||
|
|
|||
| To split long text into paragraphs, use the `text_chunking` ingest processor before the `sparse_encoding` processor. For more information, see [Text chunking processor]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/text-chunking/). | |||
There was a problem hiding this comment.
Can we be more specific and change To split long text into paragraphs to To avoid information loss by truncating tokens for long documents? The text_chunking processor is a text preprocess step before embedding processor. You can refer to the RFC for details.
_search-plugins/semantic-search.md
Outdated
| @@ -48,6 +48,8 @@ PUT /_ingest/pipeline/nlp-ingest-pipeline | |||
| ``` | |||
| {% include copy-curl.html %} | |||
|
|
|||
| To split long text into paragraphs, use the `text_chunking` ingest processor before the `text_embedding` processor. For more information, see [Text chunking processor]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/text-chunking/). | |||
yuye-aws
commented
Mar 20, 2024
| --- | ||
|
|
||
| # Text chunking processor | ||
|
|
||
| The `text_chunking` processor is used to chunk a long document into paragraphs. The following is the syntax for the `text_chunking` processor: | ||
| The `text_chunking` processor is used to chunk a long document into paragraphs on a delimiter or chunks of a certain size. The following is the syntax for the `text_chunking` processor: |
There was a problem hiding this comment.
certain size -> certain token size
|
|
||
| Users can set parameter `overlap_rate` from 0 to 50 percent. According to [bedrock](https://aws.amazon.com/blogs/aws/knowledge-bases-now-delivers-fully-managed-rag-experience-in-amazon-bedrock/), we recommend users to set this parameter between 0–20 percent to help improve accuracy. | ||
| You can set the `overlap_rate` to a decimal equal to 0 to 50 percent. Per [Bedrock](https://aws.amazon.com/blogs/aws/knowledge-bases-now-delivers-fully-managed-rag-experience-in-amazon-bedrock/), we recommend setting this parameter to a value of 0–20 percent to improve accuracy. |
There was a problem hiding this comment.
a decimal equal to 0 to 50 percent -> a decimal equal from 0 to 50 percent
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
Signed-off-by: yuye-aws <yuyezhu@amazon.com>
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
natebower
reviewed
Mar 21, 2024
There was a problem hiding this comment.
@yuye-aws @kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!
|
|
||
| # Text chunking processor | ||
|
|
||
| The `text_chunking` processor splits a long document into shorter passages. The processor supports the following algorithms for splitting text: |
There was a problem hiding this comment.
"text splitting" instead of "splitting text"?
| |:---|:---|:---|:---| | ||
| | `field_map` | Object | Required | Contains key-value pairs that specify the mapping of a text field to the output field for the text chunking processor. | | ||
| | `field_map.<input_field>` | String | Required | The name of the field from which to obtain text for generating chunked passages. | | ||
| | `field_map.<output_field>` | String | Required | The name of the field in which to store the chunking results. | |
There was a problem hiding this comment.
"chunked" instead of "chunking"?
| } | ||
| ``` | ||
|
|
||
| Once you have created an ingest pipeline, you need to create an index for ingestion and ingest documents into the index. To learn more, see [Step 2: Create an index for ingestion]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/#step-2-create-an-index-for-ingestion) and [Step 3: Ingest documents into the index]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/#step-3-ingest-documents-into-the-index) of [Neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/). |
There was a problem hiding this comment.
At the end, let's clarify slightly and lowercase "neural". "of the neural sparse search documentation"?
kolchfa-aws
reviewed
Mar 21, 2024
kolchfa-aws
reviewed
Mar 21, 2024
kolchfa-aws
reviewed
Mar 21, 2024
kolchfa-aws
reviewed
Mar 21, 2024
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
approved these changes
Mar 21, 2024
CaptainDredge
pushed a commit
to CaptainDredge/documentation-website
that referenced
this pull request
Mar 22, 2024
* add documentation for text chunking processor Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix style check errors Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix style check errors Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix style check errors Signed-off-by: yuye-aws <yuyezhu@amazon.com> * Doc review Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * add example on text embedding and cascade chunking Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix style check Signed-off-by: yuye-aws <yuyezhu@amazon.com> * add comment Signed-off-by: yuye-aws <yuyezhu@amazon.com> * Review additional requested info Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: yuye-aws <yuyezhu@amazon.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
hdhalter
added
3 - Done
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add documentation for text chunking processor
Issues Resolved
Implements RFC: opensearch-project/neural-search#548
Closes #6663
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.