Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Remove legacy lambda provider #9543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 6, 2023
Merged

Remove legacy lambda provider #9543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8a1a04e
Remove legacy lambda provider
joe4dev Nov 2, 2023
304d550
Fix skip for not yet implemented concurrency alias test
joe4dev Nov 2, 2023
2a394bc
Add removed section to LocalStack config
joe4dev Nov 3, 2023
88193c6
Re-introduce HOSTNAME_FOR_LAMBDA
joe4dev Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 0 additions & 39 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,40 +92,6 @@ jobs:
paths:
- repo/target/coverage/

itest-lambda-legacy-local:
executor: ubuntu-machine-amd64
working_directory: /tmp/workspace/repo
steps:
- attach_workspace:
at: /tmp/workspace
- prepare-pytest-tinybird
- run:
# legacy tests are executed locally, new runners ship with Java 17, downgrade to Java 11 for stepfunctions
name: Install OpenJDK 11
command: |
sudo apt-get update && sudo apt-get install openjdk-11-jdk
sudo update-alternatives --set java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
sudo update-alternatives --set javac /usr/lib/jvm/java-11-openjdk-amd64/bin/javac
java -version
- run:
name: Test 'local' Lambda executor
environment:
LAMBDA_EXECUTOR: "local"
PROVIDER_OVERRIDE_LAMBDA: "legacy"
TEST_PATH: "tests/aws/services/lambda_/ tests/aws/test_integration.py tests/aws/services/apigateway/test_apigateway_basic.py tests/aws/services/cloudformation/resources/test_lambda.py"
COVERAGE_ARGS: "-p"
command: |
COVERAGE_FILE="target/coverage/.coverage.lambdav1.${CIRCLE_NODE_INDEX}" \
PYTEST_ARGS="${TINYBIRD_PYTEST_ARGS}--reruns 2 --junitxml=target/reports/lambda-docker.xml -o junit_suite_name='legacy-lambda-local'" \
make test-coverage
- persist_to_workspace:
root:
/tmp/workspace
paths:
- repo/target/coverage/
- store_test_results:
path: target/reports/

itest-sfn-v2-provider:
executor: ubuntu-machine-amd64
working_directory: /tmp/workspace/repo
Expand Down Expand Up @@ -514,9 +480,6 @@ workflows:
- acceptance-tests:
requires:
- preflight
- itest-lambda-legacy-local:
requires:
- preflight
- itest-sfn-v2-provider:
requires:
- preflight
Expand Down Expand Up @@ -565,7 +528,6 @@ workflows:
- docker-build-amd64
- report:
requires:
- itest-lambda-legacy-local
- itest-sfn-v2-provider
- itest-s3-v2-legacy-provider
- acceptance-tests
Expand All @@ -578,7 +540,6 @@ workflows:
branches:
only: master
requires:
- itest-lambda-legacy-local
- itest-sfn-v2-provider
- itest-s3-v2-legacy-provider
- acceptance-tests
Expand Down
1 change: 0 additions & 1 deletion doc/external_services_integration/kafka_self_managed_cluster/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ services:
- SERVICES=lambda,secretsmanager
- DEBUG=${DEBUG- }
- DATA_DIR=${DATA_DIR- }
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
- DOCKER_HOST=unix:///var/run/docker.sock
- HOST_TMP_FOLDER=${TMPDIR}
Expand Down
4 changes: 0 additions & 4 deletions doc/troubleshoot/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@ builder.withPathStyleAccessEnabled(true);
`$TMPDIR` contains a symbolic link that cannot be mounted by Docker.
(See details here: https://bitbucket.org/atlassian/localstack/issues/40/getting-mounts-failed-on-docker-compose-up)

* If you're seeing Lambda errors like `Cannot find module ...` when using `LAMBDA_REMOTE_DOCKER=false`, make sure to properly set the `HOST_TMP_FOLDER` environment variable and mount the temporary folder from the host into the LocalStack container.

* If you run into file permission issues on `pip install` under Mac OS (e.g., `Permission denied: '/Library/Python/2.7/site-packages/six.py'`), then you may have to re-install `pip` via Homebrew (see [this discussion thread](https://github.com/localstack/localstack/issues/260#issuecomment-334458631)). Alternatively, try installing
with the `--user` flag: `pip install --user localstack`

Expand All @@ -32,5 +30,3 @@ with the `--user` flag: `pip install --user localstack`
* In case you get errors related to node/nodejs, you may find (this issue comment: https://github.com/localstack/localstack/issues/227#issuecomment-319938530) helpful.

* If you are using AWS Java libraries and need to disable SSL certificate checking, add `-Dcom.amazonaws.sdk.disableCertChecking` to the java invocation.

* If you are using LAMBDA_REMOTE_DOCKER=true and running in a docker container in CI, do NOT set `DOCKER_HOST` as an environment variable passed into the localstack container. Any calls to lambda CLI operations will fail (https://github.com/localstack/localstack/issues/4801)
69 changes: 14 additions & 55 deletions localstack/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
from localstack.constants import (
DEFAULT_BUCKET_MARKER_LOCAL,
DEFAULT_DEVELOP_PORT,
DEFAULT_LAMBDA_CONTAINER_REGISTRY,
DEFAULT_VOLUME_DIR,
ENV_INTERNAL_TEST_COLLECT_METRIC,
ENV_INTERNAL_TEST_RUN,
Expand Down Expand Up @@ -829,10 +828,6 @@ def legacy_fallback(envar_name: str, default: T) -> T:
# Kinesis mock log level override when inconsistent with LS_LOG (e.g., when LS_LOG=debug)
KINESIS_MOCK_LOG_LEVEL = os.environ.get("KINESIS_MOCK_LOG_LEVEL", "").strip()

# DEPRECATED: 1 (default) only applies to old lambda provider
# Whether to handle Kinesis Lambda event sources as synchronous invocations.
SYNCHRONOUS_KINESIS_EVENTS = is_env_not_false("SYNCHRONOUS_KINESIS_EVENTS") # DEPRECATED

# randomly inject faults to Kinesis
KINESIS_ERROR_PROBABILITY = float(os.environ.get("KINESIS_ERROR_PROBABILITY", "").strip() or 0.0)

Expand Down Expand Up @@ -877,17 +872,11 @@ def legacy_fallback(envar_name: str, default: T) -> T:
os.environ.get("SQS_CLOUDWATCH_METRICS_REPORT_INTERVAL") or 60
)

# DEPRECATED: only applies to old lambda provider
# DEPRECATED: deprecated since 2.0.0 but added back upon customer request for the new Lambda provider
# Keep a bit longer until we are sure that LOCALSTACK_HOST covers the special scenario but do not advertise publicly.
# Endpoint host under which LocalStack APIs are accessible from Lambda Docker containers.
HOSTNAME_FROM_LAMBDA = os.environ.get("HOSTNAME_FROM_LAMBDA", "").strip()

# DEPRECATED: true (default) only applies to old lambda provider
# Determines whether Lambda code is copied or mounted into containers.
LAMBDA_REMOTE_DOCKER = is_env_true("LAMBDA_REMOTE_DOCKER")
# make sure we default to LAMBDA_REMOTE_DOCKER=true if running in Docker
if is_in_docker and not os.environ.get("LAMBDA_REMOTE_DOCKER", "").strip():
LAMBDA_REMOTE_DOCKER = True

# PUBLIC: hot-reload (default v2), __local__ (default v1)
# Magic S3 bucket name for Hot Reloading. The S3Key points to the source code on the local file system.
BUCKET_MARKER_LOCAL = (
Expand All @@ -898,7 +887,7 @@ def legacy_fallback(envar_name: str, default: T) -> T:
# Docker network driver for the Lambda and ECS containers. https://docs.docker.com/network/
LAMBDA_DOCKER_NETWORK = os.environ.get("LAMBDA_DOCKER_NETWORK", "").strip()

# PUBLIC v1: Currently only supported by the old lambda provider
# PUBLIC v1: LocalStack DNS (default)
# Custom DNS server for the container running your lambda function.
LAMBDA_DOCKER_DNS = os.environ.get("LAMBDA_DOCKER_DNS", "").strip()

Expand All @@ -924,13 +913,6 @@ def legacy_fallback(envar_name: str, default: T) -> T:
# How many seconds Lambda will wait for the runtime environment to start up.
LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT = int(os.environ.get("LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT") or 10)

# DEPRECATED: lambci/lambda (default) only applies to old lambda provider
# An alternative docker registry from where to pull lambda execution containers.
# Replaced by LAMBDA_RUNTIME_IMAGE_MAPPING in new provider.
LAMBDA_CONTAINER_REGISTRY = (
os.environ.get("LAMBDA_CONTAINER_REGISTRY", "").strip() or DEFAULT_LAMBDA_CONTAINER_REGISTRY
)

# PUBLIC: base images for Lambda (default) https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html
# localstack/services/lambda_/invocation/lambda_models.py:IMAGE_MAPPING
# Customize the Docker image of Lambda runtimes, either by:
Expand Down Expand Up @@ -1017,30 +999,6 @@ def legacy_fallback(envar_name: str, default: T) -> T:
# whether to skip S3 validation of provided KMS key
S3_SKIP_KMS_KEY_VALIDATION = is_env_not_false("S3_SKIP_KMS_KEY_VALIDATION")

# DEPRECATED: docker (default), local (fallback without Docker), docker-reuse. only applies to old lambda provider
# Method to use for executing Lambda functions.
LAMBDA_EXECUTOR = os.environ.get("LAMBDA_EXECUTOR", "").strip()

# DEPRECATED: only applies to old lambda provider
# Fallback URL to use when a non-existing Lambda is invoked. If this matches
# `dynamodb://<table_name>`, then the invocation is recorded in the corresponding
# DynamoDB table. If this matches `http(s)://...`, then the Lambda invocation is
# forwarded as a POST request to that URL.
LAMBDA_FALLBACK_URL = os.environ.get("LAMBDA_FALLBACK_URL", "").strip()
# DEPRECATED: only applies to old lambda provider
# Forward URL used to forward any Lambda invocations to an external
# endpoint (can use useful for advanced test setups)
LAMBDA_FORWARD_URL = os.environ.get("LAMBDA_FORWARD_URL", "").strip()
# DEPRECATED: ignored in new lambda provider because creation happens asynchronously
# Time in seconds to wait at max while extracting Lambda code.
# By default, it is 25 seconds for limiting the execution time
# to avoid client/network timeout issues
LAMBDA_CODE_EXTRACT_TIME = int(os.environ.get("LAMBDA_CODE_EXTRACT_TIME") or 25)

# DEPRECATED: 1 (default) only applies to old lambda provider
# whether lambdas should use stay open mode if executed in "docker-reuse" executor
LAMBDA_STAY_OPEN_MODE = is_in_docker and is_env_not_false("LAMBDA_STAY_OPEN_MODE")

# PUBLIC: 2000 (default)
# Allows increasing the default char limit for truncation of lambda log lines when printed in the console.
# This does not affect the logs processing in CloudWatch.
Expand Down Expand Up @@ -1204,21 +1162,16 @@ def use_custom_dns():
"GATEWAY_LISTEN",
"HOSTNAME",
"HOSTNAME_EXTERNAL",
"HOSTNAME_FROM_LAMBDA",
"HOSTNAME_FROM_LAMBDA", # deprecated since 2.0.0 but added to new Lambda provider
"KINESIS_ERROR_PROBABILITY",
"KINESIS_INITIALIZE_STREAMS",
"KINESIS_MOCK_PERSIST_INTERVAL",
"KINESIS_MOCK_LOG_LEVEL",
"KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT",
"KMS_PROVIDER", # Not functional; Deprecated in 1.4.0, removed in 3.0.0
"LAMBDA_CODE_EXTRACT_TIME",
"LAMBDA_CONTAINER_REGISTRY",
"LAMBDA_DOCKER_DNS",
"LAMBDA_DOCKER_FLAGS",
"LAMBDA_DOCKER_NETWORK",
"LAMBDA_EXECUTOR",
"LAMBDA_FALLBACK_URL",
"LAMBDA_FORWARD_URL",
"LAMBDA_INIT_DEBUG",
"LAMBDA_INIT_BIN_PATH",
"LAMBDA_INIT_BOOTSTRAP_PATH",
Expand All @@ -1230,11 +1183,9 @@ def use_custom_dns():
"LAMBDA_KEEPALIVE_MS",
"LAMBDA_RUNTIME_IMAGE_MAPPING",
"LAMBDA_JAVA_OPTS",
"LAMBDA_REMOTE_DOCKER",
"LAMBDA_REMOVE_CONTAINERS",
"LAMBDA_RUNTIME_EXECUTOR",
"LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT",
"LAMBDA_STAY_OPEN_MODE",
"LAMBDA_TRUNCATE_STDOUT",
"LAMBDA_RETRY_BASE_DELAY_SECONDS",
"LAMBDA_SYNCHRONOUS_CREATE",
Expand Down Expand Up @@ -1280,14 +1231,22 @@ def use_custom_dns():
"SQS_CLOUDWATCH_METRICS_REPORT_INTERVAL",
"STEPFUNCTIONS_LAMBDA_ENDPOINT",
"STRICT_SERVICE_LOADING",
"SYNCHRONOUS_KINESIS_EVENTS",
"SYNCHRONOUS_SNS_EVENTS",
"TEST_AWS_ACCOUNT_ID",
"TF_COMPAT_MODE",
"USE_SINGLE_REGION", # Not functional; deprecated in 0.12.7, removed in 3.0.0
"USE_SSL",
"WAIT_FOR_DEBUGGER",
"WINDOWS_DOCKER_MOUNT_PREFIX",
# Removed in 3.0.0
"LAMBDA_CODE_EXTRACT_TIME", # deprecated since 2.0.0
"LAMBDA_CONTAINER_REGISTRY", # deprecated since 2.0.0
"LAMBDA_EXECUTOR", # deprecated since 2.0.0
"LAMBDA_FALLBACK_URL", # deprecated since 2.0.0
"LAMBDA_FORWARD_URL", # deprecated since 2.0.0
"LAMBDA_REMOTE_DOCKER", # deprecated since 2.0.0
"LAMBDA_STAY_OPEN_MODE", # deprecated since 2.0.0
"SYNCHRONOUS_KINESIS_EVENTS", # deprecated since 1.3.0
"SYNCHRONOUS_SNS_EVENTS", # deprecated since 1.3.0
]


Expand Down
3 changes: 0 additions & 3 deletions localstack/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,6 @@
# AWS region us-east-1
AWS_REGION_US_EAST_1 = "us-east-1"

# default lambda registry
DEFAULT_LAMBDA_CONTAINER_REGISTRY = "lambci/lambda"

# environment variable to override max pool connections
try:
MAX_POOL_CONNECTIONS = int(os.environ["MAX_POOL_CONNECTIONS"])
Expand Down
24 changes: 7 additions & 17 deletions localstack/deprecations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,13 @@ def is_affected(self) -> bool:
"1.4.0",
"This feature is marked for removal. Please use AWS client API to seed Kinesis streams.",
),
EnvVarDeprecation(
"PROVIDER_OVERRIDE_LAMBDA",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the provider override as a deprecation option helps users cleaning up their configuration

"3.0.0",
"This option is ignored because the legacy Lambda provider (v1) has been removed since 3.0.0. "
"Please remove PROVIDER_OVERRIDE_LAMBDA and migrate to our new Lambda provider (v2): "
"https://docs.localstack.cloud/user-guide/aws/lambda/#migrating-to-lambda-v2",
),
EnvVarDeprecation(
"ES_CUSTOM_BACKEND",
"0.14.0",
Expand Down Expand Up @@ -304,23 +311,6 @@ def log_deprecation_warnings(deprecations: Optional[List[EnvVarDeprecation]] = N
affected_deprecations = collect_affected_deprecations(deprecations)
log_env_warning(affected_deprecations)

provider_override_lambda = os.environ.get("PROVIDER_OVERRIDE_LAMBDA")
if provider_override_lambda and provider_override_lambda in ["v1", "legacy"]:
env_var_value = f"PROVIDER_OVERRIDE_LAMBDA={provider_override_lambda}"
deprecation_version = "2.0.0"
# TODO[LambdaV1] adjust message or convert into generic deprecation for PROVIDER_OVERRIDE_LAMBDA
deprecation_path = (
f"Remove {env_var_value} to use the new Lambda 'v2' provider (current default). "
"For more details, refer to our Lambda migration guide "
"https://docs.localstack.cloud/user-guide/aws/lambda/#migrating-to-lambda-v2"
)
LOG.warning(
"%s is deprecated (since %s) and will be removed in upcoming releases of LocalStack! %s",
env_var_value,
deprecation_version,
deprecation_path,
)


def deprecated_endpoint(
endpoint: Callable, previous_path: str, deprecation_version: str, new_path: str
Expand Down
4 changes: 2 additions & 2 deletions localstack/runtime/analytics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@
"KINESIS_PROVIDER",
"KMS_PROVIDER",
"LAMBDA_DOWNLOAD_AWS_LAYERS",
# Irrelevant post v3 but intentionally tracked for some time
"LAMBDA_EXECUTOR",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be interesting to know how many users still have legacy variables configured post v3 🤷

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't hurt and we can still turn it off later 👍

"LAMBDA_PREBUILD_IMAGES",
# Irrelevant post v3 but intentionally tracked for some time
"LAMBDA_REMOTE_DOCKER",
"LAMBDA_RUNTIME_EXECUTOR",
"LEGACY_DIRECTORIES",
Expand Down Expand Up @@ -50,8 +52,6 @@
"HOSTNAME_FROM_LAMBDA",
"HOST_TMP_FOLDER",
"INIT_SCRIPTS_PATH",
"LAMBDA_FALLBACK_URL",
"LAMBDA_FORWARD_URL",
"LEGACY_DIRECTORIES",
"LEGACY_INIT_DIR",
"LOCALSTACK_HOST",
Expand Down
Loading