Thanks to visit codestin.com
Credit goes to github.com

Skip to content

only do a multipart s3 copy when above the regular copy limit#41914

Merged
juliusknorr merged 2 commits intomasterfrom
s3-copy-size-limit
Dec 29, 2023
Merged

only do a multipart s3 copy when above the regular copy limit#41914
juliusknorr merged 2 commits intomasterfrom
s3-copy-size-limit

Conversation

@icewind1991
Copy link
Member

@icewind1991 icewind1991 commented Nov 29, 2023

Multipart copy seems problematic with some backends so avoid it when possible.

@icewind1991 icewind1991 added the 3. to review Waiting for reviews label Nov 29, 2023
@icewind1991 icewind1991 added this to the Nextcloud 29 milestone Nov 29, 2023
@icewind1991 icewind1991 requested review from a team, ArtificialOwl, come-nc, juliusknorr and sorbaugh and removed request for a team November 29, 2023 16:33
@icewind1991
Copy link
Member Author

icewind1991 commented Nov 29, 2023

/backport to stable28

@icewind1991
Copy link
Member Author

icewind1991 commented Nov 29, 2023

/backport to stable27

solracsf
solracsf reviewed Nov 29, 2023
View reviewed changes
Copy link
Member

@solracsf solracsf left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't test if MultiPart is supported? Something like:

if ($this->getObjectStore() instanceof IObjectStoreMultiPartUpload) {
   // multipart
} else {
  // not multipart
}

come-nc
come-nc approved these changes Nov 30, 2023
View reviewed changes
juliusknorr
juliusknorr approved these changes Dec 4, 2023
View reviewed changes
Copy link
Member

@juliusknorr juliusknorr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good, do you know what s3 providers/solutions enforce such a limit?

@juliusknorr
Copy link
Member

juliusknorr commented Dec 4, 2023

Why don't test if MultiPart is supported?

It might need a different interface, I'm not sure if all providers that support multipart upload also support multipart copy

@icewind1991
Copy link
Member Author

icewind1991 commented Dec 4, 2023

do you know what s3 providers/solutions enforce such a limit?

No, but there are at least 2 different cases where the multipart copy is failing

Why don't test if MultiPart is supported? Something like:

I'm not aware of any good way to tell if an S3 implementation supports it

@solracsf solracsf mentioned this pull request Dec 6, 2023
Closed
8 tasks
@Ziyann
Copy link

Ziyann commented Dec 13, 2023 •
edited
Loading

For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested this patch yet). Some built-in solution would be greatly appreciated, either a configurable limit like copySizeLimit here (which we could set to a very high value, thus never triggering multipart copy), or even better, some enable/disable switches for the various operations, because right now I'm manually reverting #40577 on production after upgrades.
There are many S3 providers with different features supported, so I think it would be great to have as many configuration options as possible (avoiding manual file changes on production).

@Ziyann
Copy link

Ziyann commented Dec 19, 2023

You should propagate copySizeLimit to the S3 SDK (mup_threshold), otherwise, it will use the hardcoded 5 GB threshold at some places: https://github.com/nextcloud/3rdparty/blob/v28.0.0/aws/aws-sdk-php/src/S3/ObjectCopier.php#L96-L103
See: #42068 (comment)

@juliusknorr
Copy link
Member

juliusknorr commented Dec 28, 2023

Pushed an additional commit to be able to fully turn off multipart copy for s3 providers not supporting it at all.

@juliusknorr juliusknorr added the pending documentation This pull request needs an associated documentation update label Dec 28, 2023
@juliusknorr juliusknorr merged commit 77cec80 into master Dec 29, 2023
@juliusknorr juliusknorr deleted the s3-copy-size-limit branch December 29, 2023 10:28
This was referenced Dec 29, 2023
Merged
Merged
@juliusknorr juliusknorr mentioned this pull request Jan 23, 2024
Merged
@blizzz blizzz mentioned this pull request Mar 5, 2024
Merged
@joshtrichards joshtrichards mentioned this pull request Nov 22, 2024
Closed
8 tasks
joshtrichards added a commit to nextcloud/documentation that referenced this pull request Nov 22, 2024
@joshtrichards
fix(s3): document options for adjusting multipartcopy behavior
8c00c86 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
@joshtrichards joshtrichards mentioned this pull request Nov 22, 2024
Merged
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
@joshtrichards @backportbot
fix(s3): document options for adjusting multipartcopy behavior
b7c54ff 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
@joshtrichards @backportbot
fix(s3): document options for adjusting multipartcopy behavior
9429e76 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
backportbot bot pushed a commit to nextcloud/documentation that referenced this pull request Dec 2, 2024
@joshtrichards @backportbot
fix(s3): document options for adjusting multipartcopy behavior
aff76d1 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
@joshtrichards @susnux
fix(s3): document options for adjusting multipartcopy behavior
1458c38 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
@joshtrichards @susnux
fix(s3): document options for adjusting multipartcopy behavior
9be131a 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
susnux pushed a commit to nextcloud/documentation that referenced this pull request Mar 24, 2025
@joshtrichards @susnux
fix(s3): document options for adjusting multipartcopy behavior
213c5be 
Added in nextcloud/server#41914

Signed-off-by: Josh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@solracsf solracsf solracsf left review comments

@juliusknorr juliusknorr juliusknorr approved these changes

@come-nc come-nc come-nc approved these changes

@ArtificialOwl ArtificialOwl Awaiting requested review from ArtificialOwl ArtificialOwl is a code owner automatically assigned from nextcloud/server-backend

@sorbaugh sorbaugh Awaiting requested review from sorbaugh sorbaugh is a code owner automatically assigned from nextcloud/server-backend

Assignees

No one assigned

Labels

3. to review Waiting for reviews pending documentation This pull request needs an associated documentation update

Projects

None yet

Milestone

Nextcloud 29

Development

Successfully merging this pull request may close these issues.

5 participants

@icewind1991 @juliusknorr @Ziyann @solracsf @come-nc