APIGW: implement Canary Deployments CRUD logic #12694

bentsku · 2025-05-30T23:29:09Z

Motivation

We've got a request to support Canary Deployments CRUD logic as it is needed in some Terraform deployments.

This was on a radar for a while, and there was no support for it in Moto.
This PR only implements the CRUD support for it, a follow-up PR will implement the actually canary deployments logic during invocation, with the ability to direct requests to either deployment. The test for it is already in this PR just to verify how it worked in AWS (with test_invoking_canary_deployment).

I'm conscious that this implementation is... not great. There are so many patches, the update logic is really complex. Those JSON Patches really don't make this easy. We should probably rework some kind of framework to easily handle validation + application of those.

At first, I didn't want to duplicate the code, and I couldn't insert it from the child class inside update_stage. I had done in the legacy provider, but I don't want to break existing things and have partial things implemented so I moved it back to the next gen/current provider. This would have made the diff simpler, sorry!

The logic implemented is documented in AWS here: https://docs.aws.amazon.com/apigateway/latest/developerguide/canary-release.html

There was also a bug in Moto where it would update your stage deployment id to the one of the canary setting if you were to set one due to greedy check.

note: test failure is a flake in DDB Streams

Changes

update operations related to canary settings: CreateStage, UpdateStage, CreateDeployment
add a lot of tests around the behavior to make sure we're getting things somewhat right

github-actions · 2025-05-30T23:39:09Z

Test Results - Preflight, Unit

21 579 tests ±0 19 927 ✅ ±0 6m 17s ⏱️ -6s
1 suites ±0 1 652 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit a3b527a. ± Comparison against base commit 1a9469e.

github-actions · 2025-05-30T23:44:45Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 17m 54s ⏱️ - 1h 28m 55s
1 134 tests - 3 335 1 068 ✅ - 3 012 66 💤 - 323 0 ❌ ±0
1 136 runs - 3 335 1 068 ✅ - 3 012 68 💤 - 323 0 ❌ ±0

Results for commit a3b527a. ± Comparison against base commit 1a9469e.

This pull request removes 3343 and adds 8 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.apigateway.test_apigateway_canary.TestCanaryDeployments ‑ test_invoking_canary_deployment
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_create_canary_deployment
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_create_canary_deployment_by_stage_update
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_create_canary_deployment_validation
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_create_canary_deployment_with_stage
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_create_update_stages
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_update_stage_canary_deployment_validation
tests.aws.services.apigateway.test_apigateway_canary.TestStageCrudCanary ‑ test_update_stage_with_copy_ops

This pull request removes 324 skipped tests and adds 1 skipped test. Note that renamed tests count towards both.

tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input4-FAILED]
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_deployed_infra_state
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_populate_data
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_user_clicks_are_stored
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_api_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_invalid_desiredstate
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_double_create_with_client_token
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_lifecycle
…

tests.aws.services.apigateway.test_apigateway_canary.TestCanaryDeployments ‑ test_invoking_canary_deployment

github-actions · 2025-05-30T23:48:58Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 13s ⏱️ +6s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit a3b527a. ± Comparison against base commit 1a9469e.

github-actions · 2025-05-30T23:53:11Z

Test Results - Alternative Providers

597 tests 420 ✅ 14m 40s ⏱️
4 suites 177 💤
4 files 0 ❌

Results for commit a3b527a.

github-actions · 2025-05-31T00:22:31Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 23m 12s ⏱️
4 832 tests 4 289 ✅ 543 💤 0 ❌
4 838 runs 4 289 ✅ 549 💤 0 ❌

Results for commit a3b527a.

♻️ This comment has been updated with latest results.

bentsku · 2025-06-02T13:51:51Z

localstack-core/localstack/services/apigateway/next_gen/provider.py

-        response = super().update_stage(
-            context, rest_api_id, stage_name, patch_operations, **kwargs
-        )
+        moto_rest_api = get_moto_rest_api(context, rest_api_id)


note: most of this code is already taken from the parent class, sorry it does not make a clean diff

cloutierMat

Wow! Impressive amount of work and testing. 🚀

Thank you for the good PR description, really helpful when diving into some of the changes and weirdness due to the split store! 😬

The list of test and it's coverage is great, the update operation is quite tricky to nail down but you did a great deep dive here. Looking forward to see the follow up with invocation changes! 🥳

bentsku added 12 commits May 30, 2025 11:03

wip

0d6c3da

wip

df4adbc

add all canary tests

ee7c4f9

add case

28f54b9

add test cases

a4120c1

add CreateDeployment validation

5a77fa7

add more validation and fixes to Stages

e64f927

wip, only one test failing with copy

9c99ccd

get all tests to run

04f7654

add skip marker

34ea5c1

fix moto util

0bd2e8b

move back code to NextGen

a3b527a

bentsku added this to the 4.5 milestone May 30, 2025

bentsku self-assigned this May 30, 2025

bentsku added aws:apigateway Amazon API Gateway semver: patch Non-breaking changes which can be included in patch releases labels May 30, 2025

bentsku mentioned this pull request May 31, 2025

APIGW: add Canary Deployment logic in invocation layer #12695

Merged

bentsku marked this pull request as ready for review May 31, 2025 00:52

bentsku requested a review from cloutierMat as a code owner May 31, 2025 00:52

bentsku commented Jun 2, 2025

View reviewed changes

cloutierMat approved these changes Jun 2, 2025

View reviewed changes

bentsku merged commit 374183c into master Jun 2, 2025
85 of 86 checks passed

bentsku deleted the apigw-canary-crud branch June 2, 2025 18:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

APIGW: implement Canary Deployments CRUD logic #12694

APIGW: implement Canary Deployments CRUD logic #12694

Uh oh!

bentsku commented May 30, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

github-actions bot commented May 31, 2025 •

edited

Loading

Uh oh!

bentsku Jun 2, 2025

Uh oh!

cloutierMat left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

APIGW: implement Canary Deployments CRUD logic #12694

APIGW: implement Canary Deployments CRUD logic #12694

Uh oh!

Conversation

bentsku commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

github-actions bot commented May 30, 2025

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented May 30, 2025

LocalStack Community integration with Pro

Uh oh!

github-actions bot commented May 30, 2025

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented May 30, 2025

Test Results - Alternative Providers

Uh oh!

github-actions bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

bentsku Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

cloutierMat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bentsku commented May 30, 2025 •

edited

Loading

github-actions bot commented May 31, 2025 •

edited

Loading