Lambda revisions #7482

joe4dev · 2023-01-13T11:27:32Z

Test + implement revisions (i.e., RevisionId) for lambda in general (with versions,
aliases, layers, etc). It seems that the revision id of permissions is
the same as for qualified lambdas (i.e., VersionFunctionConfiguration).

Motivation

Follow-up from Lambda permissions #7336
Mentioned in skip_snapshot_verify and TODO comments at multiple places
- CloudFormation tests also mentions: “seems the revision id of the policy actually corresponds to the one of the function version”

Behavior

CreateFunction: RevisionId The latest updated revision of the function or alias.
Non-matching revision id raises a PreconditionFailedException

Overview of Actions / API Operations with RevisionId support:

Request	Response	Both
AddPermission ✔️	CreateAlias ✔️	AddLayerVersionPermission ✔️
RemoveLayerVersionPermission ✔️	CreateFunction ✔️	PublishVersion ✔️
RemovePermission ✔️	GetAlias ✔️	UpdateAlias ✔️
	GetFunction ✔️	UpdateFunctionCode ✔️
	GetFunctionConfiguration	UpdateFunctionConfiguration ✔️
	GetLayerVersionPolicy ✔️
	GetPolicy ✔️
	ListAliases
	ListFunctions
	ListVersionsByFunction

✔️ refers to snapshot tested request/response
The list* operations are not explicitly tested within the revisions lifecycle but are covered by other tests
Note: This just shows which request/responses contain a revision_id but do NOT show which operations UPDATE it! For example, add_permission updates the revision_id despite not returning it.
Q/A about behavior that prevents atomic updates: https://repost.aws/questions/QU2NMlYyvYSaaoNY8xrPYcvQ/lambda-function-updating-cannot-be-made-atomic-with-revision-id

Scope

LayerVersionPolicy: have their own RevisionId independent of functions as they can exist without functions.
RevisionId per function configuration. Aliases and versions have their own function configuration but lambda permissions don’t.
- The fact that the permissions only require the RevisionId in the request but don’t contain it in the reply, could hint that it’s not part of their entity and only stored per function configuration
- Also fits together with DeleteAlias not mentioning any RevisionId because it’s gone with the associated function configuration after deleting the alias

Questions

Do we need locking to ensure atomic updates across entities?

Details

For more details: See Revisions in the Notion Lambda Board.

Follow Up (out of scope)

Consider updating the data model such that property updates automatically trigger a revision_id update within the right entity such that we a) don't need locks for ensuring atomic updates b) simplify code for retrieval and updates.

github-actions · 2023-01-13T12:58:10Z

LocalStack integration with Pro

      3 files ±0       3 suites ±0 1h 24m 39s ⏱️ -6s
1 644 tests +2 1 328 ✔️ ±0 316 💤 +2 0 ❌ ±0
2 340 runs +6 1 696 ✔️ ±0 644 💤 +6 0 ❌ ±0

Results for commit 4278da8. ± Comparison against base commit 6dda0b1.

♻️ This comment has been updated with latest results.

Now fully and more comprehensively covered in `tests.integration.awslambda.test_lambda_api.TestLambdaRevisions.test_function_revisions_version_and_alias` Grouped all revision-related tests in `TestLambdaRevisions`

dominikschubert

Looks great!

Love the additional comments you added where the context was a bit obscure 👍

I know there are a few new TODOs and things we've encountered here that will need to be fixed in the future, but I'd propose we move forward for now and tackle this later when polishing 👍

localstack/services/awslambda/invocation/lambda_models.py

tests/integration/cloudformation/resources/test_lambda.py

Co-authored-by: Dominik Schubert <[email protected]>

joe4dev · 2023-01-18T10:28:26Z

I agree on bundling these TODOs in the polishing round 👍
Let's prioritize the layers and roadmap for now

dfangl

LGTM, good work (little too late :D)

dfangl · 2023-01-18T10:18:27Z

localstack/services/awslambda/provider.py

@@ -402,6 +404,11 @@ def _publish_version_from_existing_version(
        self.lambda_service.publish_version(new_version)
        state = lambda_stores[account_id][region]
        function = state.functions.get(function_name)
+        # TODO: re-evaluate data model to prevent this dirty hack just for bumping the revision id


Yes, the intention was to have a transparent mechanism to change revision ids without manually doing it, but publishing versions is the perfect counterexample for that.

I see. Commendable efforts as it works most of the time and makes it easier for these cases.

localstack/services/awslambda/invocation/version_manager.py

dfangl · 2023-01-18T10:21:38Z

localstack/utils/testutil.py

+    """Utility method to create a new function via the Lambda API
+    CAVEAT: Does NOT wait until the function is ready/active. The fixture create_lambda_function waits until ready."""


This method is actually the base of create_lambda_function, as the fixture was meant as a drop in replacement. Nobody should use this function directly anymore (or almost anything in testutil, for that matter)

Makes sense.
I just saw it used in other services (17x in localstack and 21x in -ext) and therefore added the clarification and reference to the fixture.

localstack/services/awslambda/provider.py

joe4dev · 2023-01-18T10:46:47Z

LGTM, good work (little too late :D)

Thank you 🙏

Sorry for the timing 😬

Resolves misleading naming especially for latest_version (pretends it is $LATEST) but actually refers to the resolved qualifier. Fixes Daniel's comment: #7482 (comment)

joe4dev requested review from dfangl and dominikschubert as code owners January 13, 2023 11:27

joe4dev temporarily deployed to localstack-ext-tests January 13, 2023 11:27 — with GitHub Actions Inactive

joe4dev temporarily deployed to localstack-ext-tests January 13, 2023 18:03 — with GitHub Actions Inactive

joe4dev added 21 commits January 16, 2023 14:42

Add revision test for creating and updating functions

1f113e1

Add function revision tests for versions

3e133c2

Add revision tests for alias create and update

02dc2e6

Add revision tests for adding and removing permissions

b834b7a

Add revision exception for update_function_code

b787ef7

Add revision exception for update_function_configuration

9fdee68

Validate test_function_revisions_basic

8d5bfe0

Add revision exception for update_alias

005e166

Fix revision id behavior for lambda permissions

f6de33e

Extract method for resolving function revision id

e18543e

Validate RevisionId in snapshots for new provider

8b5d6d4

Comment limitation in testutil create function

c1d88ff

Cleanup old code and improve comments

e10113b

Remove redundant revision id test

44e9262

Now fully and more comprehensively covered in `tests.integration.awslambda.test_lambda_api.TestLambdaRevisions.test_function_revisions_version_and_alias` Grouped all revision-related tests in `TestLambdaRevisions`

Fix revision updates for aliases

06bfffc

Refine revision tests to expose publish_version rev update

0ea1351

Fix revision update upon publish_version for LATEST

3d1504e

Test revisions for permissions with alias

727b727

Add snapshot test for get_alias for revisions

1e2ef25

Add revision id for layer version permissions

9211952

Clarify locking question in todo

4278da8

joe4dev force-pushed the lambda-revisions branch from 8a07e3b to 4278da8 Compare January 16, 2023 13:42

joe4dev temporarily deployed to localstack-ext-tests January 16, 2023 13:43 — with GitHub Actions Inactive

dominikschubert approved these changes Jan 18, 2023

View reviewed changes

localstack/services/awslambda/invocation/lambda_models.py Outdated Show resolved Hide resolved

tests/integration/cloudformation/resources/test_lambda.py Show resolved Hide resolved

Remove comment regarding IAM typed policy

afdcde7

Co-authored-by: Dominik Schubert <[email protected]>

joe4dev temporarily deployed to localstack-ext-tests January 18, 2023 10:22 — with GitHub Actions Inactive

joe4dev merged commit d617162 into master Jan 18, 2023

joe4dev deleted the lambda-revisions branch January 18, 2023 10:30

dfangl approved these changes Jan 18, 2023

View reviewed changes

joe4dev mentioned this pull request Jan 18, 2023

Switch to new Lambda provider implementation #6724

Merged

10 tasks

joe4dev added a commit that referenced this pull request Jan 25, 2023

Rename resolved alias and version

a49d9d6

Resolves misleading naming especially for latest_version (pretends it is $LATEST) but actually refers to the resolved qualifier. Fixes Daniel's comment: #7482 (comment)

joe4dev mentioned this pull request Jan 25, 2023

Fix naming from Lambda revisions PR #7551

Merged

thrau mentioned this pull request Aug 5, 2023

Update PULL_REQUEST_TEMPLATE.md #8831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Lambda revisions #7482

Lambda revisions #7482

Uh oh!

joe4dev commented Jan 13, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Jan 13, 2023 •

edited

Loading

Uh oh!

dominikschubert left a comment

Uh oh!

Uh oh!

Uh oh!

joe4dev commented Jan 18, 2023

Uh oh!

dfangl left a comment

Uh oh!

dfangl Jan 18, 2023

Uh oh!

joe4dev Jan 18, 2023

Uh oh!

Uh oh!

dfangl Jan 18, 2023

Uh oh!

joe4dev Jan 18, 2023

Uh oh!

Uh oh!

joe4dev commented Jan 18, 2023

Uh oh!

Uh oh!

		"""Utility method to create a new function via the Lambda API
		CAVEAT: Does NOT wait until the function is ready/active. The fixture create_lambda_function waits until ready."""

Uh oh!

Lambda revisions #7482

Lambda revisions #7482

Uh oh!

Conversation

joe4dev commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Behavior

Scope

Questions

Details

Follow Up (out of scope)

Uh oh!

github-actions bot commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack integration with Pro

Uh oh!

dominikschubert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joe4dev commented Jan 18, 2023

Uh oh!

dfangl left a comment

Choose a reason for hiding this comment

Uh oh!

dfangl Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

joe4dev Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dfangl Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

joe4dev Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joe4dev commented Jan 18, 2023

Uh oh!

Uh oh!

joe4dev commented Jan 13, 2023 •

edited

Loading

github-actions bot commented Jan 13, 2023 •

edited

Loading