Update lambda runtime deprecations and fix Lambda ARM builds #10441

joe4dev · 2024-03-12T16:41:24Z

Motivation

Some deprecated Lambda runtimes are fully phased out at AWS, such that we cannot create snapshots anymore. We need to update our Lambda runtimes lifecycle accordingly.

Changes

Update deprecated Lambda runtimes (matching the snapshot behavior): Deprecate 6 runtimes and remove them from test execution including Runtime.python3_7, Runtime.nodejs14_x, Runtime.ruby2_7, Runtime.provided, Runtime.go1_x, Runtime.java8
Update Lambda runtime snapshots
Add Makefile to conveniently clean and build all Lambda common tests
Fix all ARM builds such that the Lambda multiruntime tests are now fully ARM compatible.
Re-purpose the Golang tests in test_lambda_runtimes.py while re-using the @multiruntime infrastructure

Testing

Community integration tests against Pro (not really necessary; already ran here and no ARM version): https://github.com/localstack/localstack/actions/runs/8339573132

Discussion

Keep supported runtimes in sync with resource browser

❓ Do we need to start offering a dynamic endpoint that lists the supported (and deprecated) runtimes?

Currently, the supported runtimes seem hardcoded. This only works with the latest LocalStack version. Users with older LocalStack versions might face issues.

AWS behavior does not match docs

❓ What do we use as a reference?

Note

This was mainly an issue right during the API updates but now everything looks in sync again.

Snapshot testing reveals that deprecated Lambda runtimes still remain supported much longer than described in the documentation. Therefore, our deprecation warnings are not in sync with AWS (mainly enforced by the unit test tests.unit.services.lambda_.test_api_utils.TestApiUtils.test_check_runtime that enforces that our tested runtimes match the supported runtimes). We could remove them from our tests if we value deprecation warning parity higher than current behavior parity (i.e., still being able to create deprecated runtimes).

coveralls · 2024-03-13T11:28:05Z

coverage: 85.699% (+0.009%) from 85.69%
when pulling be6c93b on update-lambda-runtime-deprecations
into 2df0d12 on master.

github-actions · 2024-03-15T21:49:53Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 31m 46s ⏱️ + 2m 5s
2 704 tests - 24 2 449 ✅ - 24 255 💤 ±0 0 ❌ ±0
2 706 runs - 24 2 449 ✅ - 24 257 💤 ±0 0 ❌ ±0

Results for commit a3e29bd. ± Comparison against base commit 82bf6e6.

This pull request removes 33 and adds 9 tests. Note that renamed tests count towards both.

tests.aws.services.lambda_.test_lambda_common.TestLambdaCallingLocalstack ‑ test_manual_endpoint_injection[go1.x]
tests.aws.services.lambda_.test_lambda_common.TestLambdaCallingLocalstack ‑ test_manual_endpoint_injection[java8]
tests.aws.services.lambda_.test_lambda_common.TestLambdaCallingLocalstack ‑ test_manual_endpoint_injection[nodejs14.x]
tests.aws.services.lambda_.test_lambda_common.TestLambdaCallingLocalstack ‑ test_manual_endpoint_injection[python3.7]
tests.aws.services.lambda_.test_lambda_common.TestLambdaCallingLocalstack ‑ test_manual_endpoint_injection[ruby2.7]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_echo_invoke[go1.x]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_echo_invoke[java8]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_echo_invoke[nodejs14.x]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_echo_invoke[provided]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_echo_invoke[python3.7]
…

tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_runtime_wrapper_invoke[dotnet6]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_runtime_wrapper_invoke[java11]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_runtime_wrapper_invoke[java17]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_runtime_wrapper_invoke[java21]
tests.aws.services.lambda_.test_lambda_common.TestLambdaRuntimesCommon ‑ test_runtime_wrapper_invoke[java8.al2]
tests.aws.services.lambda_.test_lambda_runtimes.TestGoProvidedRuntimes ‑ test_manual_endpoint_injection[provided.al2023]
tests.aws.services.lambda_.test_lambda_runtimes.TestGoProvidedRuntimes ‑ test_manual_endpoint_injection[provided.al2]
tests.aws.services.lambda_.test_lambda_runtimes.TestGoProvidedRuntimes ‑ test_uncaught_exception_invoke[provided.al2023]
tests.aws.services.lambda_.test_lambda_runtimes.TestGoProvidedRuntimes ‑ test_uncaught_exception_invoke[provided.al2]

♻️ This comment has been updated with latest results.

joe4dev · 2024-03-19T07:58:01Z

tests/aws/services/lambda_/functions/common/README.md

+  * b) creates a `handler.zip` file with a Lambda deployment package
+* SHOULD define an `ARCHITECTURE` parameter to overwrite the target architecture (i.e., `x86_64` or `arm64`)
+  if architecture-specific builds are required (e.g., Dotnet, Golang, Rust).
+  * By default, the Makefile should build a deployment package with the same architecture as the host.


This default behavior is debatable:

It currently aims for compatibility. For example, the build on a Linux ARM machine should not fail.

It differs from the default Lambda behavior using x86_64 by default.

Hence, on multi-architecture platforms (e.g., Apple Silicon), x86_64 builds require a parameter make ARCHITECTURE=x86_64.

joe4dev · 2024-03-19T08:00:48Z

tests/aws/services/lambda_/functions/common/echo/dotnet6/Makefile

+DOCKER_PLATFORM ?= linux/$(ARCHITECTURE)
+# The packaging function architecture is x86_64 by default and needs to be set explicitly for arm64
+# https://github.com/aws/aws-extensions-for-dotnet-cli/blob/cdd490450e0407139d49248d94a4a899367e84df/src/Amazon.Lambda.Tools/LambdaDefinedCommandOptions.cs#L111
+FUNCTION_ARCHITECTURE ?= $(ARCHITECTURE)


Dotnet is a good example why an explicit ARCHITECTURE flag is needed:

In general, the --platform=$(DOCKER_PLATFORM) flag is needed for cross-architecture on multi-architecture platforms

The Dotnet build specifically requires the --function-architecture flag for ARM build. It does not detect the right architecture automatically (unlike Golang builds). Therefore, being consistently verbose clarifies the build options.

joe4dev · 2024-03-19T08:01:57Z

localstack/services/lambda_/runtimes.py


+# A temporary list of missing runtimes not yet supported in LocalStack. Used for modular updates.
+# TODO: add Dotnet8 runtime
+MISSING_RUNTIMES = [Runtime.dotnet8]


This should be removed with the follow-up addition in the Dotnet8 PR

joe4dev · 2024-03-19T08:07:36Z

localstack/testing/aws/lambda_utils.py

        return package_path

    # packaging
-    result = subprocess.run(["make", "build"], cwd=runtime_dir)


L90 above: We could consider renaming common to multiruntime to make the reuse a bit more explicit.

joe4dev · 2024-03-19T08:18:03Z

tests/aws/services/lambda_/functions/common/echo/dotnet6/Makefile

@@ -1,9 +1,20 @@
+UNAME := $(shell uname -m)
+ifeq ($(UNAME),x86_64)


I wished this could be shorter but Ubuntu (in CI) yields aarch64 and macOS yields arm64, so we need the conditional to standardize the UNAME architecture :(

joe4dev · 2024-03-19T11:06:05Z

just noticed that some of the ARM test skips were still there 😬
Fixed and gonna see what fails in the CI now. Will see whether the Java tests need the ARCHITECTURE conditional for the Docker build as well ...

dfangl

Great changes, with nice comments to explain the reasoning 🎉

localstack/testing/aws/lambda_utils.py

dfangl · 2024-03-19T13:17:20Z

tests/aws/services/lambda_/functions/common/Makefile

+# Top-level Makefile to invoke all make targets in sub-directories
+
+# Based on https://stackoverflow.com/a/72209214/6875981
+SUBDIRS := $(patsubst %/,%,$(wildcard */))


joe4dev · 2024-03-19T19:54:58Z

The Makefiles were not part of the caching checksum 😬, so the specific Rust build fix was never applied 🙈
Finally, everything is green 🟢 now that Makefile updates invalidate the cache as well 🚀

joe4dev · 2024-03-19T19:56:29Z

Sidenote about the Rust ARM builds:

List of Rust build targets in the docs: https://doc.rust-lang.org/nightly/rustc/platform-support.html
This issue mentioned that "Aarch64 stack probes are tested in CI" and everything should work: Support stack probing for arm64 (aarch64) rust-lang/rust#77071
The fix was done in this PR and released with Rust 1.76.0: Enable stack probes on aarch64 for LLVM 18 rust-lang/rust#118491
The -musl suffix was required to fix a GLIBC not found error with the Lambda runtime provided.al2

Need to validate whether that also works in CI under Ubuntu ?!

joe4dev added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Mar 12, 2024

joe4dev self-assigned this Mar 12, 2024

joe4dev mentioned this pull request Mar 13, 2024

Add Dotnet 8 Lambda runtime #10300

Merged

1 task

joe4dev force-pushed the update-lambda-runtime-deprecations branch from be6c93b to 7d2a162 Compare March 14, 2024 10:03

joe4dev force-pushed the update-lambda-runtime-deprecations branch 2 times, most recently from 354ecd8 to 5cbc19c Compare March 18, 2024 18:21

joe4dev commented Mar 19, 2024

View reviewed changes

joe4dev marked this pull request as ready for review March 19, 2024 08:12

joe4dev requested review from dominikschubert, dfangl and alexrashed as code owners March 19, 2024 08:12

joe4dev commented Mar 19, 2024

View reviewed changes

dfangl approved these changes Mar 19, 2024

View reviewed changes

joe4dev added 9 commits March 20, 2024 10:04

Update deprecate Lambda runtimes

be8f62d

Re-create all Lambda common test snapshots

7ebdb39

Bump deprecated Lambda runtime

c05da77

Remove deprecated Lambda runtimes python3_7, java8, go1_x, and provided

7753d56

Add snapshot skips for logging config

b77fa7c

Re-generate all snapshots including skipped ARM runtimes

3852948

Fix Java 11 compile error

265449e

Migrate Go tests into extra lambda runtime tests

1c076cf

Fix ARM Lambda builds for Dotnet, Golang, and Rust

b00d96c

joe4dev added 11 commits March 20, 2024 10:04

Fix architecture detection in CI

0762b66

Fix Rust build for provide.al2

a29c768

Remove ARM test skips for common tests

72ccc7f

Skip endpointinjection tests for missing provided implementation

394d9fa

Clarify rust target comment

afade71

Need to validate whether that also works in CI under Ubuntu ?!

Fix introspection Rust build depending on platform

dc176b1

Clarify custom bootstrap handler value

bc9d9c4

Fix caching issue missing any Makefile updates

c3bc0ab

Clarify Lambda pre-build step naming

44221c7

Apply Rust ARM build fix again using musl

f065419

Fix Lambda function prebuilding for Github Actions

a3e29bd

joe4dev force-pushed the update-lambda-runtime-deprecations branch from 62cc0ab to a3e29bd Compare March 20, 2024 09:04

joe4dev changed the title ~~Update lambda runtime deprecations~~ Update lambda runtime deprecations and fix Lambda ARM builds Mar 20, 2024

joe4dev merged commit 1933c81 into master Mar 20, 2024

joe4dev deleted the update-lambda-runtime-deprecations branch March 20, 2024 15:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Update lambda runtime deprecations and fix Lambda ARM builds #10441

Update lambda runtime deprecations and fix Lambda ARM builds #10441

Uh oh!

joe4dev commented Mar 12, 2024 •

edited

Loading

Uh oh!

coveralls commented Mar 13, 2024

Uh oh!

github-actions bot commented Mar 15, 2024 •

edited

Loading

Uh oh!

joe4dev Mar 19, 2024

Uh oh!

joe4dev Mar 19, 2024

Uh oh!

joe4dev Mar 19, 2024

Uh oh!

joe4dev Mar 19, 2024

Uh oh!

joe4dev Mar 19, 2024

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

dfangl left a comment

Uh oh!

Uh oh!

dfangl Mar 19, 2024

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

Uh oh!

		@@ -1,9 +1,20 @@
		UNAME := $(shell uname -m)
		ifeq ($(UNAME),x86_64)

Uh oh!

Update lambda runtime deprecations and fix Lambda ARM builds #10441

Update lambda runtime deprecations and fix Lambda ARM builds #10441

Uh oh!

Conversation

joe4dev commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Discussion

Keep supported runtimes in sync with resource browser

AWS behavior does not match docs

Uh oh!

coveralls commented Mar 13, 2024

Uh oh!

github-actions bot commented Mar 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

joe4dev Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

dfangl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dfangl Mar 19, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

joe4dev commented Mar 19, 2024

Uh oh!

Uh oh!

joe4dev commented Mar 12, 2024 •

edited

Loading

github-actions bot commented Mar 15, 2024 •

edited

Loading