Thanks to visit codestin.com
Credit goes to github.com

Skip to content

support for multiple parallel lambda module calls with the same package #139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hickmant6 opened this issue Apr 3, 2021 · 2 comments
Closed

support for multiple parallel lambda module calls with the same package #139

hickmant6 opened this issue Apr 3, 2021 · 2 comments

Comments

@hickmant6
Copy link

hickmant6 commented Apr 3, 2021 •
edited
Loading

Is your request related to a new offering from AWS? No.

Is your request related to a problem? Somewhat, yes. Please describe.

When calling the lambda module multiple times in parallel (i.e. creating the same lambda function in multiple regions), the python package hash is the same for each instance. As such, the builds/[hash].zip file names are the same. When the first instance completes the builds/[hash].zip file is deleted. So when the other instance gets to the point of uploading creating the lambda function (which includes uploading the zip file), the instance creation errors out.

It would be great if this module supported parallel instance creation.

740 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec): zip: adding: create_incident.py
741 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec): zip: adding: requirements.txt
742 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0] (local-exec): zip: adding content of directory: .terraform/modules/snow_integration/terraform/../package
743 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0] (local-exec): zip: adding: get_credentials.py
744 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0] (local-exec): zip: adding: create_incident.py
745 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0] (local-exec): zip: adding: requirements.txt
746 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0] (local-exec): Created: builds/75cb84580c56f5c51a322379b807b40e1d3dbf519f419ef2183e3b217b9d662a.zip
747 | module.snow_integration.module.us_east_1_region_lambda.null_resource.archive[0]: Creation complete after 11s [id=3959019780436615163]
748 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec): Traceback (most recent call last):
749 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):   File ".terraform/modules/snow_integration.ap_southeast_1_region_lambda/package.py", line 1257, in <module>
750 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):     main()
751 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):   File ".terraform/modules/snow_integration.ap_southeast_1_region_lambda/package.py", line 1253, in main
752 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):     exit(args.command(args))
753 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):   File ".terraform/modules/snow_integration.ap_southeast_1_region_lambda/package.py", line 1145, in build_command
754 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
755 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):   File ".terraform/modules/snow_integration.ap_southeast_1_region_lambda/package.py", line 312, in __exit__
756 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):     self.close()
757 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):   File ".terraform/modules/snow_integration.ap_southeast_1_region_lambda/package.py", line 302, in close
758 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec):     os.replace(self._tmp_filename, self.filename)
759 | module.snow_integration.module.ap_southeast_1_region_lambda[0].null_resource.archive[0] (local-exec): FileNotFoundError: [Errno 2] No such file or directory: 'builds/75cb84580c56f5c51a322379b807b40e1d3dbf519f419ef2183e3b217b9d662a.zip.tmp' -> 'builds/75cb84580c56f5c51a322379b807b40e1d3dbf519f419ef2183e3b217b9d662a.zip'
760 |  
761 | module.snow_integration.module.us_east_1_region_lambda.aws_lambda_function.this[0]: Modifying... [id=pg-centrify-NonProd-us-east-1-servicenow-incident-lambda]
762 | module.snow_integration.module.us_east_1_region_lambda.aws_lambda_function.this[0]: Modifications complete after 1s [id=pg-centrify-NonProd-us-east-1-servicenow-incident-lambda]
763 |  
764 | Error: Error running command 'builds/75cb84580c56f5c51a322379b807b40e1d3dbf519f419ef2183e3b217b9d662a.plan.json': exit status 1. Output:  adding: python_dateutil-2.8.1.dist-info/WHEEL

Describe the solution you'd like.

Multiple lambda function creation all using the same package.

Describe alternatives you've considered.

I've attempted to form and modify the package.py (around lines 1058 and 1090) to make the zip file name format be builds/[aws_region]-[hash].zip, however, I'm unable to get it to work.

I've also tried adding dependon criteria in the calling module, but the lambda functions aren't created everytime and dependon requires static (not dynamic) content. So this doesn't work either.

Additional context
@antonbabenko
Copy link
Member

Hi Terry,

If you need to reuse one package multiple times, you will have to create it once and then reference it in your lambda function via local_existing_package

If you want to build similar packages (eg, one per region), you have to specify different hash_extra argument to distinct packages.

See this section for more details - https://github.com/terraform-aws-modules/terraform-aws-lambda#-how-does-building-and-packaging-work (and hash_extra in one of the examples)

@kadrach kadrach mentioned this issue Jan 4, 2022
Closed
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@antonbabenko @hickmant6