[N-2] 03-TextSplitter / 03-TokenTextSplitter #104
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Team] New Content Development Team 2 [Title] [N-2] 03-TextSplitter / 03-TokenTextSplitter [Version] initial [Language] ENG [Packages] langchain, langchain_text_splitters, tiktoken, spacy, sentence-transformers, nltk, konlpy Included a tutorial on TokenTextSplitter, covering basic usage, advanced token-based splitting techniques, and custom configuration options for optimized text segmentation.
package.install 에서 verbose 옵션을 False 로 주어 출력을 silent 하였습니다. load_dotenv(override=True) 로 주었습니다.
add Peer Reviewer
teddylee777
approved these changes
Jan 3, 2025
There was a problem hiding this comment.
[Review]
- Review OS(Windows/Mac/Linux): Mac
- Template Rule 가이드를 준수 하였는가(YES/NO): YES
- Table of Contents 의 링크가 원활하게 동작하는지 확인하였는가?(YES/NO): YES
- 이미지가 포함되어 있다면, 이미지의 파일명이 가이드를 준수하였는가?(YES/NO): YES
- import 구문이 예전 legacy 방식이 아닌 최신 버전을 따르는가?(YES/NO): YES
- 모든 코드가 동작에 오류 없이 동작하는가? (만약, warning 발생시 코멘트에 적어주세요): YES
- 기타 의견:
verbose=False 옵션으로 설치 출력 메시지를 silent 하였습니다.
load_dotenv(override=True) 로 주었습니다.
Peer Reviewer 를 지정하였습니다.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Team] New Content Development Team 2
[Title] [N-2] 03-TextSplitter / 03-TokenTextSplitter [Version] initial
[Language] ENG
[Packages] langchain, langchain_text_splitters, tiktoken, spacy, sentence-transformers, nltk, konlpy
Included a tutorial on TokenTextSplitter, covering basic usage, advanced token-based splitting techniques, and custom configuration options for optimized text segmentation.
리뷰 다시 한번 부탁드립니다.. closed가 되면서 다시 pr드립니다.