Thanks to visit codestin.com
Credit goes to github.com

Skip to content

AMQPException Library error: a SSL error occurred #1162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Webonaute opened this issue Apr 23, 2021 · 9 comments
Closed

AMQPException Library error: a SSL error occurred #1162

Webonaute opened this issue Apr 23, 2021 · 9 comments
Labels

Comments

@Webonaute
Copy link

Webonaute commented Apr 23, 2021

I have issue with SSL connection.

We are using AWS RabbitMQ service.
Broker engine version : 3.8.6

AMQPException Library error: a SSL error occurred
"file" => "/project/vendor/enqueue/amqp-ext/AmqpConsumer.php","line" => 121,"trace" => """ #0 /project/vendor/enqueue/amqp-ext/AmqpConsumer.php(121): AMQPQueue->ack()\n #1 /project/vendor/enqueue/enqueue/Consumption/QueueConsumer.php(210)

Its happen from time to time, and only when it come to acknowledge a message.

Symfony configuration:

enqueue:
    default:
        job:
            enabled: true
        transport:
            dsn: 'amqps+tls+rabbitmq:'
            host: '%env(MQ_HOST)%'
            vhost: '%env(MQ_VHOST)%'
            port: '%env(MQ_PORT)%'
            user: '%env(MQ_USER)%'
            pass: '%env(MQ_PASSWORD)%'
            connection_name: '%env(SERVICE_NAME)%-%api_version%'
            heartbeat: 60  # <--- note that I have try without heartbeat as well.
            ssl_on: true
            ssl_cacert: '%default_cert_file%'
        client:
            router_topic: 'worker'
            router_queue: operational
            prefix: '%kernel.environment%'
            default_queue: operational

It also happen only on our worker servers who consume the message. No issue with producer.

any idea how to resolve this?

@makasim
Copy link
Member

makasim commented Apr 28, 2021

I've no idea. Let's keep open. Maybe someone else could help.

@stale
Copy link

stale bot commented Jun 3, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jun 3, 2021
@TamasSzigeti
Copy link

TamasSzigeti commented Jun 4, 2021

Given this is the only hit when searching for this error message, I wanted to add my findings, although I encountered this with Symfony Messenger rather than Enqueue.
Long story short, I haven't found a fix, the only way I was able to work around it is giving a time limit of 5 minutes to the worker – and restarting it over and over… When it runs long, the connection eventually lost, and the next attempt to touch rabbit fails with the above.
I do wonder if it only happens with AWS RabbitMQ and PHP, given the lack of complaints on the intertubes.

@stale stale bot removed the wontfix label Jun 4, 2021
@stale
Copy link

stale bot commented Jul 9, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jul 9, 2021
@stale stale bot closed this as completed Jul 21, 2021
@jkobus
Copy link

jkobus commented Dec 9, 2021

I have similar problem. One of my workers kept dying from time to time on a non-ssl connection with message: "Library error: connection closed unexpectedly". After migrating to AWS RabbitMQ (which uses SSL) this keeps happening and I get instead "Library error: a SSL error occurred". That would've meant that the old error message is now obscured by the new one.

Did you managed to solve your issue @TamasSzigeti ?

@Webonaute
Copy link
Author

@jkobus try cloudamqp instead. they disable heartbeat and use tcp timeout instead and we dont have this issue. something you can't do in aws rabbitmq. we still trying to find a solution for AWS since its cheaper to maintain.

@TamasSzigeti
Copy link

Same here, no solution with AWS, we have literally just migrated to CloudAMQP a couple days ago.

@jkobus
Copy link

jkobus commented Dec 15, 2021

Guys, I assume you were using AWS MessageMQ (RabbitMQ) in a cluster deployment setup.
The problem is, that:

For each TCP request that a client makes through a Network Load Balancer, the state of that connection is tracked. If no data is sent through the connection by either the client or target for longer than the idle timeout, the connection is closed. If a client or a target sends data after the idle timeout period elapses, it receives a TCP RST packet to indicate that the connection is no longer valid.

Elastic Load Balancing sets the idle timeout value for TCP flows to 350 seconds. You cannot modify this value. Clients or targets can use TCP keepalive packets to reset the idle timeout. Keepalive packets sent to maintain TLS connections cannot contain data or payload.

source

The TCP RST packet causes the Library error: a SSL error occurred (i just reproduced that using [email protected]) and probably also Library error: connection closed unexpectedly on a non-SSL setups using custom RabbitMQ behind an AWS Network load balancer.

So, the solution would be to: don't stay idle for longer than 350 seconds after opening a connection to a cluster.

Fun-fact: the connection on the RabbitMQ by default stays open for about an hour.

As for heartbeat - if you were using non blocking calls, heartbeat is not working. At least not in php-amqp.:

heartbeats are limited to blocking calls only, so if there are no any operations on a connection or no active consumer set, connection may be closed by the broker as dead.

It seems like a solution for now would be to automatically recover from network failures, but that option is not available in Symfony Messenger (yet) which I'm using.

Hope that helps!

@makasim
Copy link
Member

makasim commented Dec 15, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants