Provider produced inconsistent final plan in github actions pipeline #158

jack-dunleavy · 2021-06-07T11:24:33Z

Description

I'm seeing an issue deploying a lambda function using Github actions CI/CD. Specifically we have the pipeline set up to perform terraform plan, store the output, then run terraform apply in a later step. The issue appears to be due to the source_code_hash attribute that is calculated during plan not matching the hash during apply:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.test_lambda.aws_lambda_function.this[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .source_code_hash:
│ was cty.StringVal("3mMv+AOi71OKcWRJFwQpJodzz13sdQ8U+W1DLqYnS+8="), but now cty.StringVal("L60e+m5YKaMuYvyjOS+J0HlXynS/xy072jqcLlArcY8=").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Versions

Terraform v0.15.1
on darwin_amd64

provider registry.terraform.io/hashicorp/aws v3.44.0
provider registry.terraform.io/hashicorp/external v2.1.0
provider registry.terraform.io/hashicorp/local v2.1.0
provider registry.terraform.io/hashicorp/null v3.1.0
provider registry.terraform.io/hashicorp/random v3.1.0

Reproduction

Steps to reproduce the behavior:

Are you using workspaces? : No

I think this is to do with the source files being checked out twice (once for plan and again for apply) and so having different metadata in each of the phases. I've been able to reproduce this issue locally by performing the following steps:

Run terraform plan: TF_RECREATE_MISSING_LAMBDA_PACKAGE=false terraform plan -out sandbox.plan
Create a copy of the lambda handler: cp src/index.js src/index_1.js
Delete the lambda handler: rm src/index.js
Rename the copy: mv src/index_1.js src/index.js
Run terraform apply: TF_RECREATE_MISSING_LAMBDA_PACKAGE=false terraform apply sandbox.plan

Code Snippet to Reproduce

main.tf

module "test_lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.0.0"

  function_name              = "test-lambda"
  handler                           = "index.handler"
  runtime                           = "nodejs14.x"
  publish                           = true
  cloudwatch_logs_retention_in_days = 90

  source_path = "./src"
}

providers.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.40"
    }
  }
}

provider "aws" {
  region = "eu-west-2"
}

src/index.js

exports.handler = async (event) => {
  return event;
};

Expected behavior

The new lambda is applied by Terraform

Actual behavior

The new lambda deployment fails with Error: Provider produced inconsistent final plan

Terminal Output Screenshot(s)

Additional context

The text was updated successfully, but these errors were encountered:

zachwhaley · 2021-06-15T16:25:41Z

We are also running into this issue.

jack-dunleavy · 2021-06-18T10:20:49Z

Any chance somebody could take a look? We've had to work around the issue by manually packaging our lambdas for the time being but it would be really nice to use the module for that

antonbabenko · 2021-06-29T10:08:37Z

The issue happens when the correct zip archive is not yet created before running apply and previous source_code_hash is being used. terraform plan creates a small json-plan file but not zip-archive.

If the content of src files is changed before apply, the source_code_hash that it produces will differ from the one stored in terraform plan file (sandbox.plan). Setting TF_RECREATE_MISSING_LAMBDA_PACKAGE=false won't help.

zachwhaley · 2021-08-13T22:09:08Z

We are seeing this now when we run terraform apply with no plan. It is not consistent, but happens often enough.

jack-dunleavy · 2021-08-14T06:56:55Z

The contents of the source code has not changed in our case, but because the repository is checked out separately between the plan and apply steps the metadata of the files changes (which in turn produces a different source code hash).

We've had to work around this by forcibly setting the modification time on every file to a fixed point after checking out (which makes the hash generation fully deterministic)

antonbabenko · 2021-08-23T09:58:34Z

I believe this issue is resolved by #175 if you use v2.10.0 and set ignore_source_code_hash = true.

Or is there anything left for this issue to remain open?

zachwhaley · 2021-08-23T14:31:15Z

I would say this is not resolved by #175 but that that change enables a different form of deployment.

I would still like to be able to use the lambda module to package my lambda code during a plan and then deploy that lambda code during an apply without error.

antonbabenko · 2021-08-23T17:16:57Z

Thanks, @zachwhaley !

Let's keep it open in case anyone can spend some time and make a fix for the problem you've described which is very much a valid use-case.

antonbabenko · 2021-08-30T15:47:01Z

Please take a look at #181 (version v2.16.0). It allows one to not worry about archive recreation when it has been deployed but currently missing (eg, a different step in the pipeline).

It may be not 100% related to the issue but it will help a lot of users.

Actually, I don't know a better solution at the moment. There is one important factor implemented by the module - terraform plan will never create an archive, so users can't just run terraform plan into a file and then apply just that file without dirty hacks.

I want to close this issue but please open a new one using the latest version of the module because there were several small improvements since this issue was originally created.

zachwhaley · 2022-01-25T21:37:28Z

For the record, we are still hitting this issue with all the workarounds and latest updates 😩

It seems to be a particular lambda that is the offender 🤷🏻

I can create a new issue with more details once I've collected said details.

Thanks for the hard work 💙

github-actions · 2022-11-09T02:38:11Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

antonbabenko closed this as completed Aug 30, 2021

bellis-ai mentioned this issue Jul 19, 2022

Logic error prevents code updates when recreate_missing_package = false in CI/CD scenario. Gives misleading "Provider produced inconsistent final plan" messages on "source_hash" + other weird errors. #330

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provider produced inconsistent final plan in github actions pipeline #158

Provider produced inconsistent final plan in github actions pipeline #158

jack-dunleavy commented Jun 7, 2021 •

edited

Loading

zachwhaley commented Jun 15, 2021

jack-dunleavy commented Jun 18, 2021

antonbabenko commented Jun 29, 2021

zachwhaley commented Aug 13, 2021

jack-dunleavy commented Aug 14, 2021

antonbabenko commented Aug 23, 2021

zachwhaley commented Aug 23, 2021

antonbabenko commented Aug 23, 2021

antonbabenko commented Aug 30, 2021

zachwhaley commented Jan 25, 2022

github-actions bot commented Nov 9, 2022

Provider produced inconsistent final plan in github actions pipeline #158

Provider produced inconsistent final plan in github actions pipeline #158

Comments

jack-dunleavy commented Jun 7, 2021 • edited Loading

Description

Versions

Reproduction

Code Snippet to Reproduce

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

Additional context

zachwhaley commented Jun 15, 2021

jack-dunleavy commented Jun 18, 2021

antonbabenko commented Jun 29, 2021

zachwhaley commented Aug 13, 2021

jack-dunleavy commented Aug 14, 2021

antonbabenko commented Aug 23, 2021

zachwhaley commented Aug 23, 2021

antonbabenko commented Aug 23, 2021

antonbabenko commented Aug 30, 2021

zachwhaley commented Jan 25, 2022

github-actions bot commented Nov 9, 2022

jack-dunleavy commented Jun 7, 2021 •

edited

Loading