fix(reindex): skip metadata-only directories without reindexable resource files#2867
Open
bhnan wants to merge 1 commit into
Open
fix(reindex): skip metadata-only directories without reindexable resource files#2867bhnan wants to merge 1 commit into
bhnan wants to merge 1 commit into
Conversation
…urce files Problem (Issue volcengine#1706): When a directory contains only metadata files (.abstract / .overview) and no actual resource files, the reindex executor incorrectly attempts to process it, leading to errors or empty reindex operations. Root cause: _reindex_resource_vectors_from_entries in reindex_executor.py did not check whether a directory has any reindexable resource files before processing it. It would read the abstract/overview and proceed even when no real resource content existed. Fix: - Added _directory_has_reindexable_resource_files() helper that checks whether a directory entry list contains any non-metadata resource files - In _reindex_resource_vectors_from_entries, skip directories that have no reindexable resource files (metadata-only directories) - Added comprehensive tests in test_admin_rebuild_api.py covering: - Pure metadata-only directory (should be skipped) - Directory with mixed metadata + resource files (should be processed) - Directory with only resource files (should be processed) Files changed: - openviking/service/reindex_executor.py (+32 lines) - tests/server/test_admin_rebuild_api.py (+144 lines) Verified: pytest tests/server/test_admin_rebuild_api.py passed (exit 0).
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1706
When a directory contains only metadata files (
.abstract/.overview) and no actual resource files, the reindex executor incorrectly attempts to process it, leading to errors or empty reindex operations.Root Cause
_reindex_resource_vectors_from_entriesinreindex_executor.pydid not check whether a directory has any reindexable resource files before processing it. It would read the abstract/overview and proceed even when no real resource content existed.Fix
_directory_has_reindexable_resource_files()helper that checks whether a directory entry list contains any non-metadata resource files_reindex_resource_vectors_from_entries, skip directories that have no reindexable resource files (metadata-only directories)Tests
Added comprehensive tests in
test_admin_rebuild_api.pycovering:All tests passed:
pytest tests/server/test_admin_rebuild_api.py(exit 0)Files Changed
openviking/service/reindex_executor.pytests/server/test_admin_rebuild_api.py