fix: clean up vector data and files when deleting a knowledge directory - #29656
Open
Classic298 wants to merge 1 commit into
Open
fix: clean up vector data and files when deleting a knowledge directory#29656Classic298 wants to merge 1 commit into
Classic298 wants to merge 1 commit into
Conversation
Deleting a directory in a knowledge base with "Delete all contents inside this directory" took the contained files out of the knowledge base but left their embeddings in the knowledge collection. The deleted content was still retrieved into answers, and uploading the same content again was rejected as duplicate content, permanently. Every other removal path already purges those chunks. The directory path now does the same: the files in the deleted subtree are collected before their links are dropped, and their chunks are removed by file id and by hash regardless of ENABLE_KNOWLEDGE_FILE_RETENTION. The file record, the stored copy and the per-file collection are removed only when retention is disabled, the file is no longer referenced by any knowledge base and the actor owns it or is an admin, which is what removing a file individually does. The lookup is scoped to the knowledge base from the request, so a directory nested under a different knowledge base's directory no longer has its file's chunks deleted from the wrong collection. Fixes open-webui#29636
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deleting a directory in a knowledge base with "Delete all contents inside this directory" took the contained files out of the knowledge base but left their embeddings in the knowledge collection. The deleted content was still retrieved into answers, and uploading the same content again was rejected as duplicate content, permanently.
Every other removal path already purges those chunks. The directory path now does the same: the files in the deleted subtree are collected before their links are dropped, and their chunks are removed by file id and by hash regardless of ENABLE_KNOWLEDGE_FILE_RETENTION. The file record, the stored copy and the per-file collection are removed only when retention is disabled, the file is no longer referenced by any knowledge base and the actor owns it or is an admin, which is what removing a file individually does.
The lookup is scoped to the knowledge base from the request, so a directory nested under a different knowledge base's directory no longer has its file's chunks deleted from the wrong collection.
Fixes #29636
Contributor License Agreement