Intermittent Message Loss in Celery Task Delivery #9848
Unanswered
wly1024666
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I'm experiencing an intermittent issue where Celery tasks are being saved to the database successfully, but the corresponding messages are not being delivered to RabbitMQ. This is an occasional problem that's difficult to reproduce consistently.
Current Implementation
In my code, I'm using a database transaction with @DB.atomic() decorator to ensure data consistency:
@staticmethod @DB.connection_context() @DB.atomic() def save_event_task_to_db(qd_event_msg: QdEventMsg, task: AispTask|None): qd_event_msg.save(force_insert=True) if task: task.save(force_insert=True) # Send data to queue with a delay to prevent consumption before transaction commit fill_work_order_info_task.apply_async( args=[task.to_dict()], countdown=3, task_id=task.task_code, properties={'message_id': task.task_code} ) logging.info(f"Task id{task.id}, task code{task.task_code} work order filling task submitted")
Observed Behavior
The AispTask record is successfully saved to the database
The log message indicates the task was submitted
However, when checking RabbitMQ, the message is not found in the fill_work_order_info queue
The Celery worker doesn't receive or process the task
Expected Behavior
When a task is saved to the database, the corresponding message should always be delivered to RabbitMQ for processing by the Celery worker.
Environment Details
Python version: [please specify]
Celery version: [please specify]
RabbitMQ version: [please specify]
Database: [please specify]
Additional Context
This is an intermittent issue that occurs occasionally. The database transaction ensures that if apply_async throws an exception, the task record wouldn't be saved to the database. Since the task record is present in the database, it suggests that apply_async either:
Didn't throw an exception despite failing to deliver the message
The message delivery failed after the database transaction was committed
There's a timing issue between database commit and message delivery
Questions
Are there scenarios where apply_async might not throw an exception but still fail to deliver messages to RabbitMQ?
What are the best practices for ensuring message delivery consistency with database transactions?
Should I implement a message delivery confirmation mechanism or compensation strategy?
Any guidance on debugging this intermittent issue would be greatly appreciated.
Beta Was this translation helpful? Give feedback.
All reactions