Symfony version(s) affected

6.0.6

Description

We are using Symfony Messenger in combination with AWS SQS. We deploy a new version of our app each day. So we also need to shutdown the workers of the previous version, that's where things go wrong.

For some reason, after each deploy we see the Approximate Age Of Oldest Message metric in SQS increase until it reaches the visibility timeout of the message. Normally this metric is always 0. This indicates a message is lost somewhere during shutdown and will be processed again. This could lead to messages being processed twice.

How to reproduce

We run messenger with the following stack: AWS ECS > Docker > Supervisor > Messenger

We have setup graceful shutdown properly, see the screenshot. Received SIGTERM signal indicates messenger knows shutdown is inbound and should stop the workers.

stopwaitsecs in supervisor has been set to 20 seconds.
The container timeout in AWS is the default 30 seconds.

Possible Solution

I noticed amazon-sqs-messenger has this method https://github.com/symfony/amazon-sqs-messenger/blob/5.4/Transport/Connection.php#L366. That class also keeps a buffer. Yet, that method never appears to be called by messenger. Is that possibly what's happening?

Additional Context

The approximate age of oldest message graph, this happens every deploy: