Problem

After reading the report from #7662, I could not understand how this could happen when we were filtering the messages before passing the context to worker to send the message. If after filtering there were no messages to send, we would not pass the task to the executor, so no empty message entries could be passed to SQS.

However, after reading the publishing code again for PublishBatch, it showed that I was setting the filtered messages (messages are filtered on a subscriber basis) on the global context (used by all subscribers). It created 2 issues:

• First, if in the order of subscription, we have a subscription with a filter then one without filter, we would filter the message and set them in the global context, so the next subscription in line would not have any messages in its context, even if it had no filter.
• Second, if the order is reversed. A subscription with no filter, then one with a filter. We would in this order:
  • Iterate over subscribers
  • During iteration, send the context to the first worker with for example 1 message
  • Continuing iteration, filter the messages depending on the subscriber and set them in the global context
  • Sending it to the second worker
  • During that time, the first worker would start its job, and might have an empty messages attributes while using the given context, which lead to the exception in the issue report.

Solution

If some messages are filtered, we will create a copy of the context, set the filtered messages only on the copy and pass the copy to the worker, instead of always passing a global context. If no filter is set, we can safely pass the global one (allows us to not always copy the context if there are no reasons for it, workers do not manipulate the context, it is used read-only).

I've added a test case for the first issue, the second one will mostly happen as a race condition and is not always reproducible. But if one is fixed, so is there other.

fixes #7662