Clean up orchestration and utils layout #2194
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to d432f2f in 1 minute and 16 seconds. Click for details.
- Reviewed
315lines of code in3files - Skipped
0files when reviewing. - Skipped posting
9draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. py/core/main/orchestration/hatchet/ingestion_workflow.py:5
- Draft comment:
Removed unused 'tiktoken' import since token counting now uses the shared num_tokens function. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. py/core/main/orchestration/hatchet/ingestion_workflow.py:31
- Draft comment:
Deleted the duplicate 'count_tokens_for_text' function; relying on a centralized num_tokens improves maintainability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
3. py/core/main/orchestration/hatchet/ingestion_workflow.py:115
- Draft comment:
Replaced calls to 'count_tokens_for_text' with the shared 'num_tokens' function; ensure this covers all token-model scenarios. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
4. py/core/main/orchestration/simple/ingestion_workflow.py:5
- Draft comment:
Removed the unused 'tiktoken' import and duplicate token counting function in favor of the shared num_tokens function. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
5. py/shared/utils/base_utils.py:587
- Draft comment:
Simplified key conversion in convert_nonserializable_objects by checking if the key is a string before converting. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
6. py/shared/utils/base_utils.py:266
- Draft comment:
Refactored tokens_count_for_message to initialize 'num_tokens' inline with tokens_per_message for brevity. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
7. py/shared/utils/base_utils.py:323
- Draft comment:
Streamlined the results setter in SearchResultsCollector for clearer type checking and iteration over inputs. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
8. py/shared/utils/base_utils.py:452
- Draft comment:
Refactored _detect_result_type to use a set literal for checking 'entity', 'relationship', 'community', which improves readability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
9. py/shared/utils/base_utils.py:649
- Draft comment:
Updated the default model in num_tokens from 'gpt-4.1' to 'gpt-4o' with a FIXME note due to tiktoken limitations. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_eJjaU2dsgF1uWm7H
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
Refactor code by consolidating token counting logic into
num_tokensfunction inbase_utils.pyand removing duplicatecount_tokens_for_textfunctions from ingestion workflows.count_tokens_for_textfunction fromingestion_workflow.pyin bothhatchetandsimpledirectories.count_tokens_for_textwithnum_tokensfrombase_utils.pyiningestion_workflow.pyin bothhatchetandsimpledirectories.num_tokensfunction tobase_utils.pyto handle token counting.setfor membership checks in_detect_result_typeinbase_utils.py.format_search_results_for_llminbase_utils.pyby usingextendfor appending multiple lines.TYPE_CHECKINGblocks inbase_utils.py.This description was created by
for d432f2f. You can customize this summary. It will automatically update as commits are pushed.