-
-
Notifications
You must be signed in to change notification settings - Fork 4.2k
Implement Put, Get, Delete, List S3 MetricsConfiguration API #12741
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Put, Get, Delete, List S3 MetricsConfiguration API #12741
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Welcome to LocalStack! Thanks for raising your first Pull Request and landing in your contributions. Our team will reach out with any reviews or feedbacks that we have shortly. We recommend joining our Slack Community and share your PR on the #community channel to share your contributions with us. Please make sure you are following our contributing guidelines and our Code of Conduct.
All contributors have signed the CLA ✍️ ✅ |
I have read the CLA Document and I hereby sign the CLA |
Sorry, I missed the label before pushing Publish PR button. I think I cannot modify it after publishing, probably I don't have permissions. |
0c57448
to
29c73e8
Compare
I published an updated commit fixing a leftover from the last |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @cristiangiann and thanks a lot for your contribution! This adds a lot to the S3 provider! 🚀
I believe the tests haven't ran against AWS, which leads to the behavior not being really in line with what AWS should return. This will need fixing, especially around the behavior of non-existent metrics configuration.
For consistency and some constraints we have, we try to limit the amount of logic to the minimum inside the models.py
, so it would great to move this logic back to the provider.
Once those are addressed and the questions in comments answered, I believe we should be good to go 🚀
Thanks again for the contribution! This is going to be a great addition 💯
9214e81
to
515af8b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice, this looks really great! We're almost good to go 🚀 thank you for addressing the comments really fast.
The implementation is very nice, and ready to go!
I only have one comment left regarding the issue you mentioned about the status code not being in line with AWS for PutBucketMetricsConfiguration
. A "snapshot skip" might be preferable here instead of the transformation.
Once this is addressed, we're ready! 🚀
tests/aws/services/s3/test_s3_api.py
Outdated
snapshot.add_transformer( | ||
[ | ||
snapshot.transform.key_value("Id"), | ||
snapshot.transform.key_value("HTTPStatusCode", reference_replacement=False), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: as we would still like to properly validate the Status Code, it might better here to use a "snapshot skip" flag.
Usually, we use the transformers for things we know will change between runs, like randomly generated ID, or things that by design are going to be different than AWS, like endpoints, host values, etc.
When things are not in parity with AWS, we tend to use a pytest marker to indicate that something is not like AWS, and we need to fix it later. It uses JSONPath to indicate which parts of the snapshot to skip.
In this case, you could use the marker like so:
@markers.snapshot.skip_snapshot_verify(
paths=[
# PutBucketMetricsConfiguration should return 204, but we return 200
"$.put_bucket_metrics_configuration.ResponseMetadata.HTTPStatusCode",
]
)
@markers.aws.validated
def test_put_bucket_metrics_configuration(self, s3_bucket, aws_client, snapshot):
You can specify the "snapshot key" as the first part of the path, the one you specify as the first argument of snapshot.match
, then the fields you want to skip.
You can add this snapshot skip to the tests having the issue.
You can then remove the transformer, and regenerate the snapshots 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something interesting is happening, and I'm not able to root cause it, at least at this time of the night. I tried to add the skip_snapshot_verify
decorator, as you suggested, but my implementation does not work as expected, both in test_put_bucket_metrics_configuration
and in test_overwrite_bucket_metrics_configuration
tests.
For example:
@markers.aws.validated
@markers.snapshot.skip_snapshot_verify(
paths=[
# PutBucketMetricsConfiguration should return 204, but we return 200
"$.put_bucket_metrics_configuration.ResponseMetadata.HTTPStatusCode",
]
)
def test_put_bucket_metrics_configuration(self, s3_bucket, aws_client, snapshot):
snapshot.add_transformer([snapshot.transform.key_value("Id")])
metric_id = short_uid()
metrics_config = {"Id": metric_id, "Filter": {"Prefix": "logs/"}}
put_result = aws_client.s3.put_bucket_metrics_configuration(
Bucket=s3_bucket, Id=metric_id, MetricsConfiguration=metrics_config
)
snapshot.match("put_bucket_metrics_configuration", put_result)
get_result = aws_client.s3.get_bucket_metrics_configuration(Bucket=s3_bucket, Id=metric_id)
snapshot.match("get_bucket_metrics_configuration", get_result)
After updating again the snapshot, this test runs successfully locally, but fails when run against AWS with the following error:
_____TestS3MetricsConfiguration.test_put_bucket_metrics_configuration _____
>> match key: put_bucket_metrics_configuration
(~) /ResponseMetadata/HTTPStatusCode 200 → 204 ... (expected → actual)
Ignore list (please keep in mind list indices might not work and should be replaced):
["$..ResponseMetadata.HTTPStatusCode"]
I triple checked the path, which seems correct, and also tried some different ones. The test seems to work if I use the same path while adding a transformer with jsonpath, through snapshot.transform.jsonpath
. I compared the logic with other usages of the skipper method in the codebase, but I was not able to find significant differences. I searched online, without success.
Are you able to spot any mistake?
This is how the updated snapshot looks like:
"tests/aws/services/s3/test_s3_api.py::TestS3MetricsConfiguration::test_put_bucket_metrics_configuration": {
"recorded-date": "13-06-2025, 00:01:27",
"recorded-content": {
"put_bucket_metrics_configuration": {
"ResponseMetadata": {
"HTTPHeaders": {},
"HTTPStatusCode": 200
}
},
"get_bucket_metrics_configuration": {
"MetricsConfiguration": {
"Filter": {
"Prefix": "logs/"
},
"Id": "<id:1>"
},
"ResponseMetadata": {
"HTTPHeaders": {},
"HTTPStatusCode": 200
}
}
}
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like you updated the snapshot against LocalStack, so that when you run it again against AWS, the snapshot verification fails because it expects the (wrong) status code LocalStack is returning. The "snapshot skip" only works against LocalStack, which is expected.
When you run the test against LocalStack, make sure you're not using the --snapshot-update
flag or the SNAPSHOT_UPDATE=1
environment variable. Sorry for the inconvenience!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally right, this is now fixed. I expected that AWS tests would have skipped it as well, but it totally makes sense.
Thank you for your patience!
515af8b
to
c50a3b1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks a lot for addressing the comments so fast and for all the updates! This looks great, and is going to be a nice addition in S3. 🚀
Thank you also for going the extra mile, and already adding pagination and some additional validation!
We currently have an issue in our pipeline with external contributors, this should be solved pretty soon. No worries about the red checks.
I've ran the tests locally and it all works! ✅
@cristiangiann The fix for the pipeline have been merged, so if you could rebase your branch on top of |
c50a3b1
to
f063996
Compare
Sorry @bentsku, I just noticed your message. I just rebased it! |
Motivation
Feature request: Implement Put, Get, Delete, List S3 BucketMetricsConfiguration: #12739
Changes
Implement Put, Get, Delete, List S3 BucketMetricsConfiguration APIs. These APIs will be available in LocalStack.