fix: Strip leading `./` in S3 key #191

pdecat · 2021-08-23T08:27:50Z

Description

This change strips the leading ./ in S3 key if artifacts_dir is set to something like ${path.root}/mypath/.

Motivation and Context

Without this change, this configuration:

module "lambda-my-layer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.4.0"

  create_layer = true
  layer_name   = "my-layer"

  runtime         = "python3.8"
  build_in_docker = true

  source_path = [{
    path             = "${path.module}/lambda/my-layer/",
    pip_requirements = true,
    prefix_in_zip    = "python",
  }]

  artifacts_dir = "${path.root}/.terraform/lambda-builds/"

  store_on_s3             = true
  s3_bucket               = aws_s3_bucket.mybuckt.bucket
  s3_object_storage_class = "STANDARD"
}

gives the following plan excerpt:

  # module.lambda-my-layer.aws_lambda_layer_version.this[0] will be created
  + resource "aws_lambda_layer_version" "this" {
      + arn                         = (known after apply)
      + compatible_runtimes         = [
          + "python3.8",
        ]
      + created_date                = (known after apply)
      + id                          = (known after apply)
      + layer_arn                   = (known after apply)
      + layer_name                  = "my-layer"
      + s3_bucket                   = "s3-my-lambdas-eu-west-3-0123456789012"
      + s3_key                      = "./.terraform/lambda-builds/41c9284df0f8fadfa3437b7a70066400d35ca912bfb089b7d3e01b59870dd077.zip"
      + s3_object_version           = (known after apply)
      + signing_job_arn             = (known after apply)
      + signing_profile_version_arn = (known after apply)
      + source_code_hash            = (known after apply)
      + source_code_size            = (known after apply)
      + version                     = (known after apply)
    }

Which fails at apply:

│ Error: Error creating lambda layer: InvalidParameterValueException: Error occurred while GetObjectVersion. S3 Error Code: NoSuchVersion. S3 Error Message: The specified version does not exist.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "18230ee0-2112-45d9-850f-0123456789012"
│   },
│   Message_: "Error occurred while GetObjectVersion. S3 Error Code: NoSuchVersion. S3 Error Message: The specified version does not exist.",
│   Type: "User"
│ }
│
│   with module.lambda-cleanup-layer.aws_lambda_layer_version.this[0],
│   on .terraform/modules/lambda-cleanup-layer/main.tf line 93, in resource "aws_lambda_layer_version" "this":
│   93: resource "aws_lambda_layer_version" "this" {

With this change, this part of the plan:

  # module.lambda-my-layer.aws_lambda_layer_version.this[0] will be created
  + resource "aws_lambda_layer_version" "this" {
      + arn                         = (known after apply)
      + compatible_runtimes         = [
          + "python3.8",
        ]
      + created_date                = (known after apply)
      + id                          = (known after apply)
      + layer_arn                   = (known after apply)
      + layer_name                  = "my-layer"
      + s3_bucket                   = "s3-my-lambdas-eu-west-3-0123456789012"
      + s3_key                      = ".terraform/lambda-builds/41c9284df0f8fadfa3437b7a70066400d35ca912bfb089b7d3e01b59870dd077.zip"
      + s3_object_version           = (known after apply)
      + signing_job_arn             = (known after apply)
      + signing_profile_version_arn = (known after apply)
      + source_code_hash            = (known after apply)
      + source_code_size            = (known after apply)
      + version                     = (known after apply)
    }

applies successfully.

Breaking Changes

This should not be a breaking change as keys with a leading ./ are not accepted by S3.

How Has This Been Tested?

I have tested and validated these changes using one or more of the provided examples/* projects

Tested with examples/complete adding the following:

diff --git a/examples/complete/main.tf b/examples/complete/main.tf
index 348aab8..4955c76 100644
--- a/examples/complete/main.tf
+++ b/examples/complete/main.tf
@@ -25,6 +25,8 @@ module "lambda_function" {

   source_path = "${path.module}/../fixtures/python3.8-app1"

+  artifacts_dir = "${path.root}/.terraform/lambda-builds/"
+
   store_on_s3 = true
   s3_bucket   = module.s3_bucket.s3_bucket_id

Without the change from this PR:

  # module.lambda_function.aws_lambda_function.this[0] will be created
  + resource "aws_lambda_function" "this" {
      + arn                            = (known after apply)
      + description                    = "My awesome lambda function"
      + function_name                  = (known after apply)
      + handler                        = "index.lambda_handler"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + layers                         = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = true
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.8"
      + s3_bucket                      = (known after apply)
      + s3_key                         = "./.terraform/lambda-builds/09df5667aa77260cb0bac734adccae8d859ac1b014db638db32448b8068cae7c.zip"
      + s3_object_version              = (known after apply)
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = (known after apply)
      + source_code_size               = (known after apply)
      + tags                           = {
          + "Module" = "lambda1"
        }
      + tags_all                       = {
          + "Module" = "lambda1"
        }
      + timeout                        = 3
      + version                        = (known after apply)

      + dead_letter_config {
          + target_arn = (known after apply)
        }

      + environment {
          + variables = {
              + "Hello"      = "World"
              + "Serverless" = "Terraform"
            }
        }

      + tracing_config {
          + mode = (known after apply)
        }
    }

With the change from this PR:

  # module.lambda_function.aws_lambda_function.this[0] will be created
  + resource "aws_lambda_function" "this" {
      + arn                            = (known after apply)
      + description                    = "My awesome lambda function"
      + function_name                  = (known after apply)
      + handler                        = "index.lambda_handler"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + layers                         = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = true
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.8"
      + s3_bucket                      = (known after apply)
      + s3_key                         = ".terraform/lambda-builds/09df5667aa77260cb0bac734adccae8d859ac1b014db638db32448b8068cae7c.zip"
      + s3_object_version              = (known after apply)
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = (known after apply)
      + source_code_size               = (known after apply)
      + tags                           = {
          + "Module" = "lambda1"
        }
      + tags_all                       = {
          + "Module" = "lambda1"
        }
      + timeout                        = 3
      + version                        = (known after apply)

      + dead_letter_config {
          + target_arn = (known after apply)
        }

      + environment {
          + variables = {
              + "Hello"      = "World"
              + "Serverless" = "Terraform"
            }
        }

      + tracing_config {
          + mode = (known after apply)
        }
    }

I have also tested this change with my own terraform configurations.

Note: unlike #168, this second PR does not use dirname() and basename() functions that would not work on Windows.

antonbabenko · 2021-08-23T08:52:02Z

Looks good. Could you also add a note in README about the benefits of using ${path.root} in artifacts_dir?

pdecat · 2021-08-25T07:00:20Z

Looks good. Could you also add a note in README about the benefits of using ${path.root} in artifacts_dir?

IMO, the only real benefit of using ${path.root} anywhere is to make it explicit that we are referring to a path relative to the root module.

Anyway, maybe there should be a way to specify the desired S3 prefix to use in the bucket in place of artifacts_dir which is meant to specify the local path for artifacts.

antonbabenko · 2021-08-25T07:54:34Z

I like the idea of having an additional artifacts_s3_dir variable which will be used when the artifact should be stored on S3.

Could you add it to this PR and then I will merge this one?

…like `${path.root}/.terraform/lambda-builds/`

pdecat · 2021-08-30T13:59:08Z

I like the idea of having an additional artifacts_s3_dir variable which will be used when the artifact should be stored on S3.

I named the parameter s3_prefix which feels more adequate to me, WDYT?

antonbabenko · 2021-08-30T14:00:46Z

The name is perfect. Does this PR work and ready for the final review? :)

pdecat · 2021-08-30T14:07:56Z

Well, I tested it with the complete example, looks good to me.

As a side note, a few weeks ago, I started exploring the conversion of the examples to actual terraform tests: https://www.terraform.io/docs/language/modules/testing-experiment.html

This could be used to assert some expected behavior and run all those checks in CI.

antonbabenko · 2021-08-30T14:19:11Z

I will do a review now.

Regarding testing. It is one of the rather hot topics for me, too.

I did a couple of episodes reviewing terraform test in February and April 2021 and native testing was not quite there where I wanted it to be before I put it into any of the existing modules. There were several smart people talking about improvements on discuss.hashicorp.com last time I checked it, so I am just watching them and will re-evaluate it once there is an update (not sure how important it is for HashiCorp, though).

Here are two of my episodes with timestamps:

Let me know what is your opinion or experience with terraform test.

antonbabenko · 2021-08-30T15:14:37Z

@pdecat Done!

v2.15.0 has been just released.

github-actions · 2022-11-09T02:39:12Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

pdecat mentioned this pull request Aug 23, 2021

fix: Strip leading ./ in S3 key if artifacts_dir is set to something like ${path.root}/mypath/ #168

Closed

1 task

pdecat force-pushed the s3_strip_leading_dot_slash_in_key branch 2 times, most recently from 2acb641 to 20508f4 Compare August 30, 2021 13:54

pdecat added 2 commits August 30, 2021 15:56

fix: Strip leading ./ in S3 key if artifacts_dir is set to something …

a14df6b

…like `${path.root}/.terraform/lambda-builds/`

Add optional s3_prefix parameter

15ee7b7

pdecat force-pushed the s3_strip_leading_dot_slash_in_key branch from 20508f4 to 15ee7b7 Compare August 30, 2021 13:56

Fixed code and example

75aaa5d

antonbabenko changed the title ~~fix: Strip leading ./ in S3 key if artifacts_dir is set to something like ${path.root}/mypath/ (take 2)~~ fix: Strip leading ./ in S3 key Aug 30, 2021

antonbabenko merged commit 2c845c3 into terraform-aws-modules:master Aug 30, 2021

pdecat deleted the s3_strip_leading_dot_slash_in_key branch August 30, 2021 15:26

github-actions bot locked as resolved and limited conversation to collaborators Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Strip leading `./` in S3 key #191

fix: Strip leading `./` in S3 key #191

pdecat commented Aug 23, 2021

antonbabenko commented Aug 23, 2021

pdecat commented Aug 25, 2021

antonbabenko commented Aug 25, 2021

pdecat commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

pdecat commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

github-actions bot commented Nov 9, 2022

fix: Strip leading ./ in S3 key #191

fix: Strip leading ./ in S3 key #191

Conversation

pdecat commented Aug 23, 2021

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

antonbabenko commented Aug 23, 2021

pdecat commented Aug 25, 2021

antonbabenko commented Aug 25, 2021

pdecat commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

pdecat commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

antonbabenko commented Aug 30, 2021

github-actions bot commented Nov 9, 2022

fix: Strip leading `./` in S3 key #191

fix: Strip leading `./` in S3 key #191