Hello Celery community,
I have long runnings tasks that are emitted by producers and consumed by workers running in different containers that can be terminated and their work interrupted at any time. I want each container to be working only at one task and in case it gets interrupted i want celery to resubmit the task, so another worker can pick up the job.

In order to not look tasks and use celery for task retry, I am testing a small test setup where the producers emitting tasks that cause a worker to sleep for 30 seconds.

I am using redis as a broker and the workers are configured to be executing with concurrency=1

celeryconfig.py

worker_prefetch_multiplier = 1

@app.task(name="sleep_with_celery", bind=True, acks_late=True, reject_on_worker_lost=True)
def sleep_with_celery(self, duration: int):
    logger.info(f'Celery Task will sleep for {duration} seconds ...')
    sleep(duration)
    logger.info(f'Done sleeping!')

The producer is configured with

celeryconfig.py

broker_transport_options = {
    'visibility_timeout': 1*1
}

I manually trigger a task on the producer that will send out a single task for the workers, and observe the unacked hash map and the queue (i.e. long) that is holding the tasks

watch -n 1 redis-cli hgetall unacked
watch -n 1 redis-cli lrange long 0 10

I want to make sure that celery will be resubmitting tasks that have been started by workes that have been shutdown/terminated before finishing.
I therefore kill either the complete worker task (celery main thread) or only the subprocessed spawn and observe the queues.

I can see that a task is added to unacked as soon as a worker starts working on it, but instead of being resubmitted once visibility_timeout is reached (for test purposes i set this to 1 sec) it remains in the long queue much longer before its than inserted into the long queue.

Question

So my problem is that I dont know if i set the visiblity_timeout correctly, as it appears that my settings are ignored.
To make it more concrete, I from the documentation dont understand how broker_transport_options are set correctly in a situation with one redis broker, and multiple producers and consumers. Where, who and in which order the broker_settings must be set? I understand this to be a global or broker specific settings. Do all workers have to have the same value for this setting?

Any clarification is apprechated.