Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When running lambda integration tests in our CI environment, our Clojure based lambda functions occasionally fail to load the required code during initialization. The CI workflow compiles source code into a jar file and also packages dependent libraries into separate zip layer provisioned into the functions during creation. Sometimes, but not always, when the Clojure function is invoked it throws an error explaining that the record in use doesn't implement the required protocol, despite having confidence that the compiled jar does in fact have the correctly compiled components. Error below:

Run set -euo pipefail
urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localstack.yellowdig.ci'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}
--- Lambda healthcheck output (raw) ---
{"errorMessage":"No implementation of method: :get-conn of protocol: #'yd.system.protocols.database-manager/DatabaseManagerProtocol found for class: datomic_http.system.client_database_manager.ClientDatabaseManager","errorType":"java.lang.IllegalArgumentException","stackTrace":["clojure.core$_cache_protocol_fn.invokeStatic(core_deftype.clj:585)","clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:577)","yd.system.protocols.database_manager$eval859$fn__860$G__846__867.invoke(database_manager.clj:3)","yd.entity.tx$db.invokeStatic(tx.clj:30)","yd.entity.tx$db.invoke(tx.clj:27)","yd.entity.network$find_network_by_name.invokeStatic(network.clj:169)","yd.entity.network$find_network_by_name.invoke(network.clj:168)","yd.lambda.healthcheck$count_all_subnetworks.invokeStatic(healthcheck.clj:32)","yd.lambda.healthcheck$count_all_subnetworks.invoke(healthcheck.clj:31)","yd.lambda.healthcheck$health_response.invokeStatic(healthcheck.clj:38)","yd.lambda.healthcheck$health_response.invoke(healthcheck.clj:36)","yd.lambd
Error: Lambda healthcheck failed with error: No implementation of method: :get-conn of protocol: #'yd.system.protocols.database-manager/DatabaseManagerProtocol found for class: datomic_http.system.client_database_manager.ClientDatabaseManager
Error: Process completed with exit code 1.

The exact same compiled jar is used in our production AWS deployments and we've yet to see this issue in a live lambda. We can also observe this when using localstack locally, and it can be resolved by repeatedly killing the localstack server and re-invoking it until it eventually succeeds.

Given the non-determinism here, we're particularly wondering if there could be a race condition between unpacking the jar and invoking the lambda code, such that certain protocol files are not yet available in the container

The lambda function and layers are the following sizes

deps.zip | 109,641,750 | 104.5 MB
lambdas.jar | 24,141,985 | 23.0 MB

We're using the latest tag of localstack pro, will follow up with exact version numbers

Expected Behavior

We expect that repeated runs of the same compiled java code when using Localstack Lambda with the Kubernetes Executor produces identical results

How are you starting LocalStack?

Localstack helm chart run against kind k8s mock-cluster in Github Actions

Steps To Reproduce

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

We running our IAC stack (Pulumi) against localstack which is creating an s3 bucket and provisioning lambdas using the s3 object coordinates

Environment

TBD

- OS:
- LocalStack:
  LocalStack version:
  LocalStack Docker image sha:
  LocalStack build date:
  LocalStack build git hash:

Anything else?

At this point we're really confident there's not an upstream issue in our compilation stack and we've done pretty extensive experimentation on this. This build tool chain and lambda approach has been running correctly in production for multiple years, and localstack is a new addition