Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Implement Put, Get, Delete, List S3 MetricsConfiguration API #12741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 17, 2025

Conversation

cristiangiann
Copy link
Contributor

Motivation

Feature request: Implement Put, Get, Delete, List S3 BucketMetricsConfiguration: #12739

Changes

Implement Put, Get, Delete, List S3 BucketMetricsConfiguration APIs. These APIs will be available in LocalStack.

  • PutBucketMetricsConfiguration: Add or update one BucketMetricsConfiguration. The limit to existing BucketMetricsConfiguration per bucket is 1000.
  • GetBucketMetricsConfiguration: Get one BucketMetricsConfiguration. Return empty response if it does not exist.
  • DeleteBucketMetricsConfiguration: Delete one BucketMetricsConfiguration.
  • ListBucketMetricsConfiguration: Fetch all BucketMetricsConfigurations, paginated through 100-item pages.

@cristiangiann cristiangiann requested a review from bentsku as a code owner June 11, 2025 10:55
Copy link
Collaborator

@localstack-bot localstack-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome to LocalStack! Thanks for raising your first Pull Request and landing in your contributions. Our team will reach out with any reviews or feedbacks that we have shortly. We recommend joining our Slack Community and share your PR on the #community channel to share your contributions with us. Please make sure you are following our contributing guidelines and our Code of Conduct.

@localstack-bot
Copy link
Collaborator

localstack-bot commented Jun 11, 2025

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@cristiangiann
Copy link
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@cristiangiann cristiangiann marked this pull request as draft June 11, 2025 11:00
@cristiangiann
Copy link
Contributor Author

Sorry, I missed the label before pushing Publish PR button. I think I cannot modify it after publishing, probably I don't have permissions.
Suggested label: semver: minor

@cristiangiann cristiangiann marked this pull request as ready for review June 11, 2025 11:09
@bentsku bentsku added aws:s3 Amazon Simple Storage Service semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases labels Jun 11, 2025
@cristiangiann cristiangiann force-pushed the feature/s3-metrics-api branch from 0c57448 to 29c73e8 Compare June 11, 2025 14:24
localstack-bot added a commit that referenced this pull request Jun 11, 2025
@cristiangiann
Copy link
Contributor Author

cristiangiann commented Jun 11, 2025

I published an updated commit fixing a leftover from the last make lint

Copy link
Contributor

@bentsku bentsku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @cristiangiann and thanks a lot for your contribution! This adds a lot to the S3 provider! 🚀

I believe the tests haven't ran against AWS, which leads to the behavior not being really in line with what AWS should return. This will need fixing, especially around the behavior of non-existent metrics configuration.

For consistency and some constraints we have, we try to limit the amount of logic to the minimum inside the models.py, so it would great to move this logic back to the provider.

Once those are addressed and the questions in comments answered, I believe we should be good to go 🚀

Thanks again for the contribution! This is going to be a great addition 💯

@cristiangiann cristiangiann force-pushed the feature/s3-metrics-api branch 3 times, most recently from 9214e81 to 515af8b Compare June 12, 2025 11:13
Copy link
Contributor

@bentsku bentsku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, this looks really great! We're almost good to go 🚀 thank you for addressing the comments really fast.
The implementation is very nice, and ready to go!

I only have one comment left regarding the issue you mentioned about the status code not being in line with AWS for PutBucketMetricsConfiguration. A "snapshot skip" might be preferable here instead of the transformation.

Once this is addressed, we're ready! 🚀

snapshot.add_transformer(
[
snapshot.transform.key_value("Id"),
snapshot.transform.key_value("HTTPStatusCode", reference_replacement=False),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: as we would still like to properly validate the Status Code, it might better here to use a "snapshot skip" flag.
Usually, we use the transformers for things we know will change between runs, like randomly generated ID, or things that by design are going to be different than AWS, like endpoints, host values, etc.

When things are not in parity with AWS, we tend to use a pytest marker to indicate that something is not like AWS, and we need to fix it later. It uses JSONPath to indicate which parts of the snapshot to skip.

In this case, you could use the marker like so:

    @markers.snapshot.skip_snapshot_verify(
        paths=[
            # PutBucketMetricsConfiguration should return 204, but we return 200
            "$.put_bucket_metrics_configuration.ResponseMetadata.HTTPStatusCode",
        ]
    )
    @markers.aws.validated
    def test_put_bucket_metrics_configuration(self, s3_bucket, aws_client, snapshot):

You can specify the "snapshot key" as the first part of the path, the one you specify as the first argument of snapshot.match, then the fields you want to skip.
You can add this snapshot skip to the tests having the issue.

You can then remove the transformer, and regenerate the snapshots 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something interesting is happening, and I'm not able to root cause it, at least at this time of the night. I tried to add the skip_snapshot_verify decorator, as you suggested, but my implementation does not work as expected, both in test_put_bucket_metrics_configuration and in test_overwrite_bucket_metrics_configuration tests.

For example:

    @markers.aws.validated
    @markers.snapshot.skip_snapshot_verify(
        paths=[
            # PutBucketMetricsConfiguration should return 204, but we return 200
            "$.put_bucket_metrics_configuration.ResponseMetadata.HTTPStatusCode",
        ]
    )
    def test_put_bucket_metrics_configuration(self, s3_bucket, aws_client, snapshot):
        snapshot.add_transformer([snapshot.transform.key_value("Id")])

        metric_id = short_uid()
        metrics_config = {"Id": metric_id, "Filter": {"Prefix": "logs/"}}

        put_result = aws_client.s3.put_bucket_metrics_configuration(
            Bucket=s3_bucket, Id=metric_id, MetricsConfiguration=metrics_config
        )
        snapshot.match("put_bucket_metrics_configuration", put_result)

        get_result = aws_client.s3.get_bucket_metrics_configuration(Bucket=s3_bucket, Id=metric_id)
        snapshot.match("get_bucket_metrics_configuration", get_result)

After updating again the snapshot, this test runs successfully locally, but fails when run against AWS with the following error:

_____TestS3MetricsConfiguration.test_put_bucket_metrics_configuration _____
>> match key: put_bucket_metrics_configuration
	(~) /ResponseMetadata/HTTPStatusCode 200 → 204 ... (expected → actual)

	Ignore list (please keep in mind list indices might not work and should be replaced):
	["$..ResponseMetadata.HTTPStatusCode"]

I triple checked the path, which seems correct, and also tried some different ones. The test seems to work if I use the same path while adding a transformer with jsonpath, through snapshot.transform.jsonpath. I compared the logic with other usages of the skipper method in the codebase, but I was not able to find significant differences. I searched online, without success.

Are you able to spot any mistake?

This is how the updated snapshot looks like:

  "tests/aws/services/s3/test_s3_api.py::TestS3MetricsConfiguration::test_put_bucket_metrics_configuration": {
    "recorded-date": "13-06-2025, 00:01:27",
    "recorded-content": {
      "put_bucket_metrics_configuration": {
        "ResponseMetadata": {
          "HTTPHeaders": {},
          "HTTPStatusCode": 200
        }
      },
      "get_bucket_metrics_configuration": {
        "MetricsConfiguration": {
          "Filter": {
            "Prefix": "logs/"
          },
          "Id": "<id:1>"
        },
        "ResponseMetadata": {
          "HTTPHeaders": {},
          "HTTPStatusCode": 200
        }
      }
    }
  },

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like you updated the snapshot against LocalStack, so that when you run it again against AWS, the snapshot verification fails because it expects the (wrong) status code LocalStack is returning. The "snapshot skip" only works against LocalStack, which is expected.

When you run the test against LocalStack, make sure you're not using the --snapshot-update flag or the SNAPSHOT_UPDATE=1 environment variable. Sorry for the inconvenience!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally right, this is now fixed. I expected that AWS tests would have skipped it as well, but it totally makes sense.

Thank you for your patience!

@cristiangiann cristiangiann force-pushed the feature/s3-metrics-api branch from 515af8b to c50a3b1 Compare June 13, 2025 08:37
Copy link
Contributor

@bentsku bentsku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks a lot for addressing the comments so fast and for all the updates! This looks great, and is going to be a nice addition in S3. 🚀

Thank you also for going the extra mile, and already adding pagination and some additional validation!

We currently have an issue in our pipeline with external contributors, this should be solved pretty soon. No worries about the red checks.
I've ran the tests locally and it all works! ✅

@bentsku
Copy link
Contributor

bentsku commented Jun 13, 2025

@cristiangiann The fix for the pipeline have been merged, so if you could rebase your branch on top of master, this would allow us to run the tests, would it be okay?

@cristiangiann cristiangiann force-pushed the feature/s3-metrics-api branch from c50a3b1 to f063996 Compare June 17, 2025 09:57
@cristiangiann
Copy link
Contributor Author

Sorry @bentsku, I just noticed your message. I just rebased it!

@bentsku bentsku merged commit 87d4176 into localstack:master Jun 17, 2025
39 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:s3 Amazon Simple Storage Service semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants