Add Dotnet 8 Lambda runtime #10300

yongliu-mdsol · 2024-02-22T13:21:41Z

Motivation

AWS released sam/dotnet8 image about a week ago

EDITED (@joe4dev):
AWS recently (2024-02-22) introduced the new Lambda runtime Dotnet8. We should support it in LocalStack.

Changes

Adding dotnet 8 runtime to localstack

EDITED (@joe4dev):

Re-structure the Dotnet build to improve maintenance by getting rid of runtime-specific values
Add transformer for CodeSha256 because that changes on AWS => re-generate all Lambda common test snapshots
Clean up unused Lambda packages

Testing

see shadow LocalStack PR for ext testing #10504

TODO

What's left to do:

Fix flaky multi-build (see comment)

localstack-bot · 2024-02-22T13:21:56Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

yongliu-mdsol · 2024-02-22T13:22:27Z

I have read the CLA Document and I hereby sign the CLA

yongliu-mdsol · 2024-02-22T13:22:38Z

recheck

yongliu-mdsol · 2024-02-22T16:03:08Z

hmm, I can't add labels to the pr

yongliu-mdsol · 2024-02-22T20:55:15Z

@dfangl @joe4dev @dominikschubert can you please help me with this PR? some checks fails due to missing git token?

dfangl · 2024-02-29T10:31:38Z

Hi @yongliu-mdsol !
The missing token is not a problem. However, there are some things left to do before we can merge this PR:

Please follow this instructions for adding a new runtime https://github.com/localstack/localstack/blob/master/localstack/services/lambda_/runtimes.py#L4-L28
Please do not commit any binary files to the repository. There is a complete dotnet8 folder which does not seem to belong in there.
The snapshots have to be regenerated like instructed in the linked instructions. Also the order of the exception messages in that file is determined by the aws validated snapshots. More information can be found here: https://docs.localstack.cloud/contributing/integration-tests/#test-against-amazon-web-services
The localstack/aws/api/lambda_/__init__.py file should not be manually edited, but needs to be auto generated as instructed in the instructions.

If you need any help with any of these steps, please do not hesitate to ask! If this is too much of a hassle for you, please also tell us and we will take over!

joe4dev · 2024-03-13T09:28:54Z

hi @dfangl, I was able to have integration tests passed locally. However, when I tried to regenerate snapshot for the test file itself, some of other runtime tests like nodejs. most of those complains about that aws no longer support those versions. Would you please help me here? thanks

Thank you for reaching out @yongliu-mdsol 🙏 You are right 👍 . Especially with yesterday's new wave of Lambda runtime deprecations, we have 6 deprecated runtimes that need to be updated and tests need to be adjusted accordingly. These deprecations also unblock some long-awaited cleanups given that all supported Lambda runtimes are now ARM-compatible.

I started tackling these updates in a separate PR #10441
I will let you know once these changes are released and this PR is ready for rebasing. I can do the rebasing and snapshot updates if you prefer.

yongliu-mdsol · 2024-03-13T22:56:55Z

@joe4dev yes please. if you can help me with this pr, that will be great. we are waiting for dotnet 8 in localstack to upgrade our projects.

joe4dev · 2024-03-21T10:44:58Z

@yongliu-mdsol The separate PR with the deprecation updates is finally merged #10441 🙌 . It took a bit longer than expected because we took the chance to clean up our Lambda (pre-)build process and make it fully ARM-compatible. This included CI changes and further changes to our internal Pro version.

I rebased your PR and edited it to align the Dotnet builds with other runtimes. We wanted to make it easier, faster, and more maintainable to add new Dotnet runtimes in the future. Therefore, I force-pushed the rebase and subsequent updates here. I hope that's ok for you. We should be able to get this one in quickly now 🚀

yongliu-mdsol · 2024-03-21T13:18:46Z

@joe4dev thank you so much for the help.
since you contribute to this pr, i would think we need someone else to review it and merge?

joe4dev · 2024-03-21T15:46:02Z

tests/aws/services/lambda_/functions/common/README.md

+Using the shared `dotnet` directory has a couple of limitations:
+* We need to use a concrete sub-directory (i.e., `dotnet6` and `dotnet8`) for pre-building
+* We need to clean before building to avoid picking up leftover from another Dotnet version build
+* Concurrent builds of Dotnet runtimes are unsafe


We waste one extra build cycle for the top-level dotnet directory.

joe4dev · 2024-03-21T15:51:13Z

tests/aws/services/lambda_/functions/common/echo/dotnet/Makefile

+
+build:
+	mkdir -p build && \
+	docker run --rm -v $$(pwd)/src:/app -v $$(pwd)/build:/out $(IMAGE) bash -c "mkdir -p /app2 && cp /app/* /app2 && cd /app2 && dotnet lambda package --framework $(FRAMEWORK) --function-architecture $(FUNCTION_ARCHITECTURE) -o ../out/handler.zip" && \


There is currently a flaky issue when building two Dotnet runtimes in short succession:

cd localstack/tests/aws/services/lambda_/functions/common/echo # FAILS make -C dotnet build && make -C dotnet6 build # WORKS make -C dotnet build && sleep 1 && make -C dotnet6 build

Error:

Zipping publish folder /app2/bin/Release/net6.0/publish to /out/handler.zip ... zipping: zip I/O error: No such file or directory ... zipping: zip error: Could not create output file (/out/handler.zip) Lambda project successfully packaged: /out/handler.zip cp: build/handler.zip: No such file or directory make[2]: *** [build] Error 1 make[1]: *** [build] Error 2 make: *** [dotnet6] Error 2

Maybe that's related to the Docker container, which needs some time to shut down 🤔
@dfangl Do you have any suggestions on how we should guard against this to avoid a flaky build and fix it locally

Very possible that there is a race condition here. Maybe we could avoid having the same mount for both? Passing in the build folder so we have a different one for dotnet6 and dotnet8 builds?

Good idea 👍
I fixed it using the FRAMEWORK version as a natural build directory suffix to minimize sub-Makefile changes.

dfangl

LGTM, nice work! Thank you @yongliu-mdsol for the PR and @joe4dev for bringing it over the line!

dfangl · 2024-03-22T20:32:30Z

tests/aws/services/lambda_/test_lambda_common.py

+            # Only Dotnet8
+            "$..environment.DOTNET_CLI_TELEMETRY_OPTOUT",
+            "$..environment.DOTNET_NOLOGO",
+            "$..environment.DOTNET_RUNNING_IN_CONTAINER",
+            "$..environment.DOTNET_VERSION",


So those are only set in LocalStack, but not in AWS?

correct 👍
(I can sneak in a clarifying comment in a later PR)

yongliu-mdsol · 2024-03-25T18:16:19Z

Thanks @joe4dev for fixing up my PR.

@dfangl do you know if this will be part of next release? and when will that be? even a prerelease image would be sufficient for us.

dfangl · 2024-03-25T18:59:04Z

@yongliu-mdsol it will be part of the next release, which is scheduled for Thursday.

localstack-bot added a commit that referenced this pull request Feb 22, 2024

@yongliu-mdsol has signed the CLA from Pull Request #10300

9389b93

yongliu-mdsol marked this pull request as ready for review February 22, 2024 16:02

yongliu-mdsol requested review from joe4dev, dominikschubert and dfangl as code owners February 22, 2024 16:02

dfangl added the semver: patch Non-breaking changes which can be included in patch releases label Feb 29, 2024

yongliu-mdsol requested review from sannya-singal, ackdav, MEPalma, thrau, baermat, bentsku, viren-nadkarni, macnev2013, giograno, alexrashed, silv-io, steffyP, pinzon, simonrw and calvernaz as code owners March 1, 2024 17:54

thrau removed request for simonrw, calvernaz and alexrashed March 1, 2024 21:07

yongliu-mdsol and others added 14 commits March 21, 2024 16:37

add dotnet 8 runtime

947ef97

Refactor Dotnet build for easier extensibility

9e9d847

Add Dotnet8 runtime

b2d1e4c

Update snapshots for new Dotnet8 tests

cb9d152

Add Dotnet8 snapshot skips

0f47d46

Use transformer for CodeSha256 and re-generate snapshots

486269b

Re-validate all multiruntime tests

bc42f99

Update layer compatibility test snapshot

7c4589e

Improve logging in Lambda pre-build script

380681b

Fix pre-building for Dotnet runtimes

0c423b7

Clean up unused Dotnet8 assets

51d0758

Clarify config comment about dev port

b57c6d2

Fix and document caching issues with shared build directory

1110acf

Remove unused Lambda packages

9ccc896

joe4dev reviewed Mar 21, 2024

View reviewed changes

joe4dev added this to the 3.3 milestone Mar 21, 2024

joe4dev self-assigned this Mar 21, 2024

joe4dev mentioned this pull request Mar 21, 2024

Add dotnet8 runtime #10504

Closed

Fix Docker race condition for Dotnet builds

48ba760

dfangl approved these changes Mar 22, 2024

View reviewed changes

joe4dev changed the title ~~add dotnet 8 runtime~~ Add Dotnet 8 Lambda runtime Mar 22, 2024

joe4dev merged commit 185ec33 into localstack:master Mar 22, 2024

yongliu-mdsol deleted the addDotnet8 branch March 25, 2024 18:17

Uh oh!

Add Dotnet 8 Lambda runtime #10300

Add Dotnet 8 Lambda runtime #10300

Uh oh!

Conversation

yongliu-mdsol commented Feb 22, 2024 • edited by joe4dev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

TODO

Uh oh!

localstack-bot commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yongliu-mdsol commented Feb 22, 2024

Uh oh!

yongliu-mdsol commented Feb 22, 2024

Uh oh!

yongliu-mdsol commented Feb 22, 2024

Uh oh!

yongliu-mdsol commented Feb 22, 2024

Uh oh!

dfangl commented Feb 29, 2024

Uh oh!

joe4dev commented Mar 13, 2024

Uh oh!

yongliu-mdsol commented Mar 13, 2024

Uh oh!

joe4dev commented Mar 21, 2024

Uh oh!

yongliu-mdsol commented Mar 21, 2024

Uh oh!

joe4dev Mar 21, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 21, 2024

Choose a reason for hiding this comment

Uh oh!

dfangl Mar 21, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 21, 2024

Choose a reason for hiding this comment

Uh oh!

dfangl left a comment

Choose a reason for hiding this comment

Uh oh!

dfangl Mar 22, 2024

Choose a reason for hiding this comment

Uh oh!

joe4dev Mar 22, 2024

Choose a reason for hiding this comment

Uh oh!

yongliu-mdsol commented Mar 25, 2024

Uh oh!

dfangl commented Mar 25, 2024

Uh oh!

Uh oh!

yongliu-mdsol commented Feb 22, 2024 •

edited by joe4dev

Loading

localstack-bot commented Feb 22, 2024 •

edited

Loading