Closed
Description
Type of request: This is a ...
[x] bug report
[ ] feature request
Detailed description
I am unable to invoke any lambas. I've tried using both docker
and docker-reuse
as the executor. docker-reuse
errors with port 9001 already taken (#1777).
Expected behavior
Run lambda.
Actual behavior
In docker mode, it tries the function three times, timing out everytime with the following error:
2020-10-08 17:10:46,651 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.Invoke: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f48d8ab98e0>>
2020-10-08 17:10:46,651 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.Invoke: calling handler <function set_operation_specific_signer at 0x7f48d9850ca0>
2020-10-08 17:10:46,651 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2020-10-08 17:10:46,651 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/2015-03-31/functions/MarlonBrando-Localstack-Core-ServiceUsers-GraphqlLambda/invocations
host:localhost:4566
x-amz-date:20201008T161046Z
host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-08 17:10:46,652 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20201008T161046Z
20201008/eu-west-2/lambda/aws4_request
eae6725a0ae542bdd077332839c32e79f431c787758edf55ba82e6fcfff89b38
2020-10-08 17:10:46,652 - MainThread - botocore.auth - DEBUG - Signature:
a29a25cc88778c7f4669200b3bca06fc57cef0586d41231a161ca3f809a9a954
2020-10-08 17:10:46,652 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=POST, url=http://localhost:4566/2015-03-31/functions/MarlonBrando-Localstack-Core-ServiceUsers-GraphqlLambda/invocations, headers={'User-Agent': b'aws-cli/1.18.148 Python/3.8.5 Linux/5.4.0-48-generic botocore/1.18.7', 'X-Amz-Date': b'20201008T161046Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAUNDDHC4KLNZJO2WA/20201008/eu-west-2/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=a29a25cc88778c7f4669200b3bca06fc57cef0586d41231a161ca3f809a9a954', 'Content-Length': '0'}>
2020-10-08 17:10:46,652 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (5): localhost:4566
2020-10-08 17:11:46,688 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.Invoke: calling handler <botocore.retryhandler.RetryHandler object at 0x7f48d8a0b4c0>
2020-10-08 17:11:46,689 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/httpsession.py", line 254, in send
urllib_response = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 423, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 330, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='localhost', port=4566): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jleckenby/.local/lib/python3.8/site-packages/awscli/clidriver.py", line 217, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/home/jleckenby/.local/lib/python3.8/site-packages/awscli/clidriver.py", line 358, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/home/jleckenby/.local/lib/python3.8/site-packages/awscli/clidriver.py", line 527, in __call__
return self._operation_caller.invoke(
File "/home/jleckenby/.local/lib/python3.8/site-packages/awscli/clidriver.py", line 649, in invoke
response = self._make_client_call(
File "/home/jleckenby/.local/lib/python3.8/site-packages/awscli/clidriver.py", line 661, in _make_client_call
response = getattr(client, xform_name(operation_name))(
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/client.py", line 337, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/client.py", line 642, in _make_api_call
http, parsed_response = self._make_request(
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/client.py", line 662, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 136, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 253, in _needs_retry
responses = self._event_emitter.emit(
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
should_retry = self._should_retry(attempt_number, response,
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
checker_response = checker(attempt_number, response,
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/home/jleckenby/.local/lib/python3.8/site-packages/botocore/httpsession.py", line 289, in send
raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://localhost:4566/2015-03-31/functions/MarlonBrando-Localstack-Core-ServiceUsers-GraphqlLambda/invocations"
2020-10-08 17:11:46,691 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
Read timeout on endpoint URL: "http://localhost:4566/2015-03-31/functions/MarlonBrando-Localstack-Core-ServiceUsers-GraphqlLambda/invocations"
I can see it creating docker containers, so it does seem to be doing something. But these containers also stay after localstack is stopped.
Steps to reproduce
Run the following command:
aws lambda invoke --function-name MarlonBrando-Localstack-Core-ServiceUsers-GraphqlLambda --endpoint=http://localhost:4566 response.json --debug
Command used to start LocalStack
version: "3"
services:
localstack:
environment:
- DATA_DIR=/tmp/localstack/data
- DEFAULT_REGION=eu-west-2
- HOST_TMP_FOLDER=${PWD}/.localstack
- LAMBDA_REMOTE_DOCKER=false
- SERVICES=apigateway,cloudformation,cloudwatch,dynamodb,iam,lambda,logs,s3
- START_WEB=0
image: localstack/localstack
ports:
- 4566:4566
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- .localstack:/tmp/localstack