Limit parameter count per query in Cache.removeChildren#29281
Merged
skjnldsv merged 2 commits intonextcloud:masterfrom Nov 1, 2021
Merged
Limit parameter count per query in Cache.removeChildren#29281skjnldsv merged 2 commits intonextcloud:masterfrom
skjnldsv merged 2 commits intonextcloud:masterfrom
Conversation
Signed-off-by: Sijmen Schoon <[email protected]>
Contributor
Author
|
@kesselb Ah, thanks for the example. I'll edit this to make it more alike that one |
This involved changing CacheQueryBuilder\whereParentIn to take a parameter name, renaming the function accordingly. Signed-off-by: Sijmen Schoon <[email protected]>
Contributor
Author
|
(Forgot to sign-off) |
nickvergessen
approved these changes
Oct 21, 2021
skjnldsv
approved these changes
Nov 1, 2021
|
Thanks for your first pull request and welcome to the community! Feel free to keep them coming! If you are looking for issues to tackle then have a look at this selection: https://github.com/nextcloud/server/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22 |
Member
|
/backport to stable22 |
Member
|
/backport to stable21 |
Member
|
/backport to stable20 |
Member
|
/backport to stable22 |
Member
|
/backport to stable21 |
Member
|
/backport to stable20 |
Member
|
/backport to stable22 |
|
The backport to stable22 failed. Please do this backport manually. |
This was referenced Nov 1, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25732 by limiting the amount of IDs in
Cache.removeChildren'sINclauses to20481000.This number was determined by the reported (but untested by me) parameter limit of mssql being 2100 (which way under the parameter limits of the other database engines), rounding it down to the next power of 2 to provide a nice safety margin.1000 was used elsewhere in the code, so that was used insteadIn limited performance testing, this did not seem to have any negative impact on performance, with PostgreSQL benchmarks (on my laptop on battery power) even reporting a significant positive impact.
Testing
To test this, I've first created a bunch of folders, each with a bunch of subfolders, with a single file in each of those by running the following bash command in the
data/admin/files/directory:Now scan the directory (
php occ files:scan --all) - this is gonna take a while, I recommend making a back-up of the database at this point to return to this state later). When this is done remove thetestdirectory, then scan again.This results in a
$parentIdselement count of over 100,000, which consistently reproduces the issue.