-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Chunk application of entity modifications #2367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
7335e23 to
fe15a91
Compare
fe15a91 to
5957d89
Compare
|
I've moved the batch logic to the I hope I got the math right:
|
lutter
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I agree on what you say about ClampRangeQuery - the one thing to check is if there is some other limit on array size. IIRC, in other contexts it was actually advantageous to break queries with large arrays into smaller ones because you get O(n^2) behavior from scanning these large arrays, so that, for example 10 queries with an array of length 1000 was faster than 1 query with an array of length 10,000
The docs state that array size limits are ignored, but Postgres will complain if the field size exceeds 1GB. Do you believe this could be better addressed and reviewed in a new issue/PR? |
This PR chunks the application of entity modifications as an attempt to fix #2330.
Given that the maximum number of PostgreSQL bind parameters per query is 65535, it makes sense to use N chunks where:
Nonetheless, we must remain wary that parameters other than entity fields are also bound in each query, such as block numbers and auxiliary data.
Therefore, it would be reasonable for us to discuss ways to accommodate those extra bindings, probably by reducing
chunk sizeby an arbitrary amount.