Hey @jamietanna ... Managed to test this over the weekend... And it works!
Sadly, using the aws invoke only trims about 400ms off my timings... Building on top of your work, I had a go at using the AWS Boto3 SDK to call the docker lambda directly, and knocked about ~500ms off it again (delta below):

import boto3

class LambdaExecutorReuseContainers(LambdaExecutorContainers):

  def prepare_execution(self, func_details, env_vars):  #, event_body, events_file_path):
        func_arn = func_details.arn()
        lambda_cwd = func_details.cwd
        runtime = func_details.runtime

        # Choose a port for this invocation
        with self.docker_container_lock:
            env_vars['_LAMBDA_SERVER_PORT'] = str(self.next_port + self.port_offset)
            self.next_port = (self.next_port + 1) % self.max_port

        # create/verify the docker container is running.
        LOG.debug('Priming docker container with runtime "%s" and arn "%s".', runtime, func_arn)
        self.prime_docker_container(func_details, env_vars, lambda_cwd)


 def _execute(self, func_arn, func_details, event, *args, **kwargs):

        if not LAMBDA_CONCURRENCY_LOCK.get(func_arn):
            concurrency_lock = threading.RLock()
            LAMBDA_CONCURRENCY_LOCK[func_arn] = concurrency_lock
        with LAMBDA_CONCURRENCY_LOCK[func_arn]:
            lambda_cwd = func_details.cwd
            runtime = func_details.runtime
            handler = func_details.handler
            environment = self._prepare_environment(func_details)

            # prepare event body
            if not event:
                LOG.warning('Empty event body specified for invocation of Lambda "%s"' % func_arn)
                event = {}

            # just prep the container, no need for cmd
            self.prepare_execution(func_details, environment)

            func_arn = func_details and func_details.arn()
            lambda_docker_hostname = self.get_docker_container_hostname(func_arn)
            
            # invoke lambda directly using http
            invocation_result = self.invoke_lambda(func_arn, lambda_docker_hostname, event)

            log_formatted = invocation_result.log_output.strip()
            LOG.debug('Lambda %s result / log output:\n%s\n> %s' % (func_arn, invocation_result.result.strip(), log_formatted))

            # store log output
            _store_logs(func_details, log_formatted)            

            return invocation_result.result

    
    def invoke_lambda(self, function_name, lambda_docker_hostname, event_body):

        full_url=f"http://{lambda_docker_hostname}:9001"
        
        client = boto3.client(
            service_name='lambda',
            region_name=config.DEFAULT_REGION,
            endpoint_url=full_url
        )

        jsonBody = json.dumps(json_safe(event_body))
        response = client.invoke(
            FunctionName=function_name,
            InvocationType='RequestResponse',
            Payload=jsonBody,
            LogType='Tail'
        )

        logs=base64.b64decode(response["LogResult"]).decode('utf-8')
        responseBody=response["Payload"].read().decode('utf-8')

        return InvocationResult(responseBody, logs)

Also, in the prime_docker_container function, getting the container network just for a DEBUG statement is a bit ott, so I tweaked it on my local to:

            # All 'docker' cmd's are expensive, only do the work if in DEBUG
            if logging.getLogger().getEffectiveLevel() == logging.DEBUG:
                container_network = self.get_docker_container_network(func_arn)
                LOG.debug('Using entrypoint "%s" for container "%s" on network "%s".' % (entry_point, container_name, container_network))

Really good work :)

Thanks for the help @chaz-doyle-cko - I'd not even considered boto3 as a faster option - glad to see it's made a good difference on your side.

I've just updated the code with latest on HEAD, and will try and have a bit more of a go at this the next few days.

@whummer this is still very WIP, so not expecting a full code review, but I was wondering if you had any thoughts about whether we should remove the scheduled docker stops that occur, given these reused containers are much faster than they used to be?

I am gonna give this a spin too. I have a nodejs use case for this.

Thanks @wesselvdv - I think it may currently be broken locally so bear with 😅

Thanks @wesselvdv - I think it may currently be broken locally so bear with sweat_smile

@wesselvdv you should be good to give it a go now, apologies for that!

@jamietanna Thanks! Building it locally now. I do see one caveat in your approach, and that is your assumption that the images used are always lambci/*. This is NOT the case for nodejs14.x for example. Using this runtime will use a different image from aws itself (). This image also supports stay_open by default, but the server is running on a different port. (8080 vs 9001)

EDIT:

I ran it locally with one minor change in the docker run command. (--expose 9001 vs -p 9001:8080)

I guess it would need a minor check to make sure that the correct port forward is put on the docker run. (lambci vs amazon images)

Maybe scratch my previous comment, I was under the impression it was the base amazon image. But localstack is using it's own fused one with lambci. I reverted my change, and it still works.

Is it possible that errors are not passed along correctly when using this branch?

Is it possible that errors are not passed along correctly when using this branch?

I've tested this by creating a (broken) Lambda and it returns in the response:

$ aws lambda invoke --function-name counter --endpoint-url=http://localhost:4566 --payload '{"id": "test"}' output.txt && cat output.txt
{"errorMessage":"Missing credentials in config","errorType":"CredentialsError","stackTrace":["ClientRequest.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/http/node.js:86:34)","Object.onceWrapper (events.js:313:30)","emitNone (events.js:106:13)","ClientRequest.emit (events.js:208:7)","Socket.emitTimeout (_http_client.js:679:34)","Object.onceWrapper (events.js:313:30)","emitNone (events.js:106:13)","Socket.emit (events.js:208:7)","Socket._onTimeout (net.js:420:8)","ontimeout (timers.js:482:11)"]}

Is it possible that errors are not passed along correctly when using this branch?

I've tested this by creating a (broken) Lambda and it returns in the response:

$ aws lambda invoke --function-name counter --endpoint-url=http://localhost:4566 --payload '{"id": "test"}' output.txt && cat output.txt
{"errorMessage":"Missing credentials in config","errorType":"CredentialsError","stackTrace":["ClientRequest.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/http/node.js:86:34)","Object.onceWrapper (events.js:313:30)","emitNone (events.js:106:13)","ClientRequest.emit (events.js:208:7)","Socket.emitTimeout (_http_client.js:679:34)","Object.onceWrapper (events.js:313:30)","emitNone (events.js:106:13)","Socket.emit (events.js:208:7)","Socket._onTimeout (net.js:420:8)","ontimeout (timers.js:482:11)"]}

Maybe it’s my own incompetence but when using the aws sdk in nodejs I am not getting it back as such. I have to parse the Payload to deduce if it was an error, whereas normal the FunctionError property would be set as ‘Error’ which is what would use to assume it error’ed.

What version of the aws CLI are you using? I know there are slightly different commands to use between V1 and V2 that could be the issue? Otherwise, if you can share a project that I can test that'd be good too!

It might be because the boto3 direct invocation isn't returning the error correctly... If I get a chance today I'll test with a lambda that just errors out...

I am only getting it back as an invocation error instead of a function error. I'll see if I can whip up an example project somewhere this week.

Final implementation merged in #4914 - thanks again everyone for contributing!! 🎉