docs: document how chunk overlap works #751
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
047bdff to
dfd3610
Compare
smoya
reviewed
May 16, 2025
smoya
reviewed
May 16, 2025
|
|
||
| Both chunking implementations use langchain's Text splitters under the hood: [[1]](https://python.langchain.com/docs/how_to/character_text_splitter/), [[2]](https://python.langchain.com/docs/how_to/recursive_text_splitter/). | ||
|
|
||
|
|
There was a problem hiding this comment.
WDYT about adding an example here?
docs/vectorizer/api-reference.md
Outdated
| A JSON configuration object that you can use in [ai.create_vectorizer](#create-vectorizers). | ||
|
|
||
| ### Understanding Overlap | ||
| The recursive character text splitter creates overlaps only when splitting within words (using the empty string separator ""). |
There was a problem hiding this comment.
Is this statement correct? In the following example, the "" splitter is not used, but there is still overlapping text:
text_splitter = RecursiveCharacterTextSplitter(separators=["\n\n", "\n", " ", ""],chunk_size=20,chunk_overlap=10)
text_splitter.create_documents(["Lorem Ipsum is simply dummy text of the printing and typesetting industry."])Produces:
['Lorem Ipsum is', 'Ipsum is simply', 'is simply dummy', 'dummy text of the', 'of the printing and', 'and typesetting', 'industry.']
There was a problem hiding this comment.
Your code actually contains the "" splitter. But I tried without it and you're correct that it still produces overlap. I can't quite figure out how this works. Some chunks seems to have overlap and some don't. Maybe linking to langchain is enough.
9e23cce to
6b94ce8
Compare
Update docs/vectorizer/api-reference.md Co-authored-by: Sergio Moya <[email protected]> Signed-off-by: Jascha Beste <[email protected]>
6b94ce8 to
f1d967c
Compare
35e80a5 to
f15011e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.