azureblob: add metadata and tags support across upload and copy paths #8750
Conversation
This change adds first-class metadata support to the Azure Blob backend,
including headers, user metadata, tags, and modtime overrides, and wires
it through uploads and server-side copies.
There is a behavior change in that rclone will now set the "mtime"
custom metadata when doing server side copies to azure and the
`--metadata` argument is given.
Highlights
- Map standard headers: cache-control, content-disposition, content-encoding,
content-language, content-type → corresponding x-ms-blob-* HTTP headers.
- Map user metadata: any non-reserved keys (excluding x-ms-*) are sent as
blob user metadata. Keys are normalized to lowercase for consistency.
- Support tags: parse `x-ms-tags` as a comma-separated list of key=value
pairs and apply them on uploads and copies.
- Support mtime override: accept `mtime` in metadata (RFC3339/RFC3339Nano)
to override the stored modtime persisted in user metadata.
Upload integration
- Both singlepart and multipart uploads call a common mapper to translate
`--metadata`, `--metadata-set`, and `--metadata-mapper` output into
Azure HTTP headers, user metadata, and tags.
- Ensures `mtime` is recorded in user metadata, honoring overrides.
- Preserves and merges computed MD5 when enabled.
Tests
- Add backend/azureblob/azureblob_internal_test.go covering:
- Singlepart/multipart uploads with metadata mapping.
- Singlepart/multipart copies with metadata mapping and Content-Type
fallback behavior.
- Validation helpers asserting headers and user metadata on the blob.
|
I gave this a quick look - looks like good work - thank you :-) It doesn't seem to set any Flags in Features, eg Lines 3974 to 3976 in d9a36ef It also isn't setting the Lines 228 to 230 in d9a36ef Can you set those things, then run the integration tests and see if they pass as the integration tests won't do anything to do with metadata unless the feature flags are set. |
|
Thanks for the early feedback @ncw ! I think that maybe I did as you asked, and that it passes tests. |
|
Hi @cliff-openai, I have ran the Any ideas? |
Specifically, Azure Blob Storage only allows metadata keys to contain letters, numbers, and underscores.
|
Gah, this took me too long to figure out. The problem is that I had the fix in my local worktree and I had just never committed/pushed it. You can see the fix/description in 488cfe0, but ultimately the problem is that the test was using a metadata key of |
ncw
left a comment
There was a problem hiding this comment.
I put a note inline as to why the GitHub CI is failing - we just need to adjust the way your tests are called slightly
Thanks
| } | ||
|
|
||
| // Standalone runner for metadata path tests to allow easy filtering with -run | ||
| func TestAzureMetadataPaths(t *testing.T) { |
There was a problem hiding this comment.
The unit tests are failing because there is no valid config for azure blob when running in the GitHub CI.
We do the integration tests on a separate machine with valid configs for all providers (and a whole suite of other tests).
So this test needs to Skip if there isn't a valid config.
However I think you probably need to delete TestAzureMetadataPaths completely and just call testAzureMetadataPaths from InternalTest
That will ensure it is only called when there is a valid config.
You should be able to filter these tests (with a longer path admittedly) with -run just fine.
|
Thanks! I’ve dropped the standalone TestAzureMetadataPaths helper and now call testMetadataPaths only from InternalTest, so the metadata suite stays within the integration harness and hopefully skips the CI config requirement. |
What is the purpose of this change?
This change adds first-class metadata support to the Azure Blob backend, including headers, user metadata, tags, and modtime overrides, and wires it through uploads and server-side copies.
There is a behavior change in that rclone will now set the "mtime" custom metadata when doing server side copies to azure and the
--metadataargument is given.Highlights
x-ms-tagsas a comma-separated list of key=value pairs and apply them on uploads and copies.mtimein metadata (RFC3339/RFC3339Nano) to override the stored modtime persisted in user metadata.Upload integration
--metadata,--metadata-set, and--metadata-mapperoutput into Azure HTTP headers, user metadata, and tags.mtimeis recorded in user metadata, honoring overrides.Tests
Was the change discussed in an issue or in the forum before?
Not directly, there is #8399 which seems abandoned (a smaller, less-complete version of this change).
I am willing to create an issue for this if it would be helpful.
Checklist