s3: preserve system metadata when setting the modification time - #9857
s3: preserve system metadata when setting the modification time#9857tirdesh wants to merge 2 commits into
Conversation
af347e5 to
50dbb22
Compare
ncw
left a comment
There was a problem hiding this comment.
Thanks for fixing this - see inline for comments.
This could also fix the similar case in Copy which is also discarding metadata in certain cases, something like this
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -3263,6 +3263,27 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
if needsReplace {
req.MetadataDirective = types.MetadataDirectiveReplace
+ // Read and apply metadata if required
+ if !ci.Metadata {
+ err = srcObj.readMetaData(ctx)
+ if err != nil {
+ return nil, fmt.Errorf("failed to read source metadata: %w", err)
+ }
+ req.CacheControl = srcObj.cacheControl
+ req.ContentDisposition = srcObj.contentDisposition
+ req.ContentEncoding = srcObj.contentEncoding
+ req.ContentLanguage = srcObj.contentLanguage
+ for k, v := range srcObj.meta {
+ if _, found := req.Metadata[k]; !found {
+ req.Metadata[k] = v
+ }
+ }
+ }
}| ContentEncoding: o.contentEncoding, | ||
| ContentLanguage: o.contentLanguage, | ||
| } | ||
| if o.storageClass != nil && *o.storageClass != "" { |
There was a problem hiding this comment.
This isn't preserving the storage class correctly. AWS omits
x-amz-storage-class from HEAD responses for STANDARD objects, so an
unknown class has to be sent as STANDARD
if o.storageClass != nil || o.fs.opt.StorageClass != "" {
req.StorageClass = types.StorageClass(o.GetTier())
}There was a problem hiding this comment.
Fixed with your snippet. Mine re-tiered STANDARD objects whenever --s3-storage-class was set, since the empty class let f.copy apply the configured one.
| ctx := context.Background() | ||
| contents := gz(t, random.String(1000)) | ||
|
|
||
| item := fstest.NewItem("test-setmodtime-metadata", contents, fstest.Time("2001-05-06T04:05:06.499999999Z")) |
There was a problem hiding this comment.
This test should skip if f.features.Copy == nil as these providers can't set mod time.
There was a problem hiding this comment.
Added, skipping on f.features.Copy == nil.
50dbb22 to
9f96a68
Compare
|
Thanks for the review, all three done. Storage class. You were right, and my version was worse than not touching it: with Test skip. Added Copy. Added as you sketched. The test is a Two related places I checked and left as they are:
|
SetModTime copies the object to itself to rewrite the modification time, which is stored in the user metadata. S3 ignores user metadata supplied with a copy unless the REPLACE metadata directive is used, so the copy has to use REPLACE. REPLACE also discards every system metadata header that is not supplied with the copy, and the request only carried the content type. Before this change, setting the modification time on an object silently dropped its Cache-Control, Content-Disposition, Content-Encoding and Content-Language headers, and moved it to the configured storage class. After it, those are read from the object and passed through, so only the modification time changes. The storage class is sent as the object's current tier rather than left empty, because AWS omits x-amz-storage-class from HEAD responses for STANDARD objects and an empty value would let the configured storage class be applied instead. No extra request is needed to do this: SetModTime already calls readMetaData on its first line, which is what populates these fields. Fixes rclone#5243
Copy uses the REPLACE metadata directive whenever an Object Lock option is in use, not only when --metadata is set. REPLACE discards every system metadata header that is not supplied with the copy, so copying an object while any Object Lock option was set dropped its Cache-Control, Content-Disposition, Content-Encoding and Content-Language, along with any user metadata the upload itself did not set. Read the source metadata and carry it over when it was not --metadata that forced the directive, leaving anything the upload has already set alone. Tested by a new CopyMetadata sub-test of InternalTestObjectLock, which reuses that test's Object Lock enabled bucket.
9f96a68 to
64b7efc
Compare
|
Rebased onto master, so this is no longer out of date. The three review points were addressed in the 3 September push, detailed in the comment
Ready for another look when you have time. |
What does this change do?
SetModTimestores the modification time in the object's user metadata, so itscopy-to-self has to use the
REPLACEmetadata directive.REPLACEdrops any systemmetadata header not supplied with the copy, and the request only carried the content type.
So updating a modification time silently dropped
Cache-Control,Content-Disposition,Content-EncodingandContent-Language, and replaced the object'sstorage class with
--s3-storage-classor the account default rather than keeping theclass the object already had.
No extra request is needed to fix it, per the optimization @ncw noted on the issue:
SetModTimealready callsreadMetaData, which populates these fields, so they were inhand and simply were not being passed to the copy. This passes them through.
New test
InternalTestSetModTimeMetadatafails on master on all four headers.Linked issue
Fixes #5243
For new or changed backends
go run ./fstest/test_all -backends s3against Minio:(s3 backend, fs/operations and fs/sync each with and without
-fast-list, vfs,cmd/gitannex, cmd/bisync.) Also
golangci-lint run ./backend/s3/..., 0 issues.Not changed here
The storage class is included because a modification time update resetting the tier is the
same wipe, but happy to drop those three lines if you would rather keep this to the headers
named in the issue. The
// Guess the content typecomment is now misleading, sincefs.MimeTypeprefers the object's own value, but it predates this change so I left it.Checklist
test_allpasses for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.