[Mailer][Sendinblue] API integration does not work anymore #48936

TristanPouliquen · 2023-01-10T10:45:13Z

Symfony version(s) affected

5.4

Description

Hello,

We realised that emails we send through the Symfony Sendinblue integration do not contain all necessary headers.

Reply-To header is missing, To header is empty for example.

This breaks our workflow as we cannot specify a custom reply-to email address in some cases (notification that a person sent a message to one of our clients for example).

From what I see in SendinblueApiTransport, the whole email payload is nested in the json attribute of the HttpClientInterface call.

Sendinblue API reference lists all items separately : https://developers.sendinblue.com/reference/sendtransacemail

Is it possible that the integration does not send the payload correctly to Sendinblue?

Our emails sent through Mailgun do not encounter this issue, it appears only when sending through Sendinblue.

How to reproduce

Create an account with Sendinblue
Install the symfony/sendinblue-mailer package
Create an Email with a custom Reply-To address and send it
Check the presence or absence of Reply-To header in the email received.

Possible Solution

No response

Additional Context

No response

The text was updated successfully, but these errors were encountered:

fabpot · 2023-01-10T10:56:37Z

Can you investigate more and maybe propose a fix?

stof · 2023-01-10T11:03:31Z

I am using the Sendinblue API integration in production myself, and Reply-To is working fine. So this needs more details.

Please provide the actual reproducer about how you send your email.

From what I see in SendinblueApiTransport, the whole email payload is nested in the json attribute of the HttpClientInterface call.

that's totally expected. The json option in the call contains the structure that will be json-encoded to create the request payload.

TristanPouliquen · 2023-01-10T16:13:31Z

Hello, I'll detail some tests here.
I created this bug because, on the same codebase, emails sent through Mailgun are correctly setup while a lot of headers were missing going through Sendinblue.

In my tests, our BaseEmailFactory creates a LocaleAwareEmail that extends the Symfony default Email object.
Relevant set-up code goes as follows:

public function create(
        Organization $organization,
        bool $isBulk,
        RecipientDto $to,
        string $fromName,
        \DateTimeImmutable $createdForDate,
        string $subject,
        string $htmlBody
    ): LocaleAwareEmail {
        // Both From: and Sender: are required for Gmail to display the sender's name
        $sender = new Address(
            $this->senderFactory->createEmailAddress($organization->getBrand(), $isBulk),
            $fromName
        );

        $htmlBody = $this->cssInliner->convert($htmlBody);
        $textBody = $this->htmlConverter->convert($htmlBody);

        $email = (new LocaleAwareEmail($organization, $createdForDate, $to->getContact()))
            ->to($to->getAddress())
            ->from($sender)
            ->sender($sender)
            ->subject($subject)
            ->text($textBody)
            ->html($htmlBody);

        $transport = $this->espStrategy->getTransport($organization, $isBulk, $to->getAddress()->getAddress());

        $email->getHeaders()
            // Tell Symfony which transport to use
            ->addTextHeader(self::HEADER_TRANSPORT, $transport)
            // Debug data
            ->addTextHeader(self::HEADER_BRAND, $organization->getBrand())
            ->addTextHeader(self::HEADER_SENT_THROUGH, $transport)
            ->addTextHeader(
                self::HEADER_SENT_BY,
                $this->espStrategy->getEsp(
                    $organization,
                    $isBulk,
                    $to->getAddress()->getAddress()
                )
            )
            ->addTextHeader(self::HEADER_SENT_FROM, $this->getDebugDetails());

        return $email;
    }

RecipientDto->getAddress returns a Symfony Address object.

Test 1: Mailgun 1

public function execute(InputInterface $input, OutputInterface $output): int
    {
        $email = $this->emailFactory->create(
            OrganizationTestFactory::createForTest(),
            false,
            new RecipientDto(new Address('[email protected]')),
            'Mailgun 1',
            new \DateTimeImmutable(),
            'Mailgun 1',
            'Content'
        );

        $this->mailer->send($email);

        return self::SUCCESS;
    }

To, From, Sender headers correctly set up & filled.

Test 2: Sendinblue 1

public function execute(InputInterface $input, OutputInterface $output): int
    {
        $email = $this->emailFactory->create(
            OrganizationTestFactory::createForTest(),
            false,
            new RecipientDto(new Address('[email protected]')),
            'Sendinblue 1',
            new \DateTimeImmutable(),
            'Sendinblue 1',
            'Content'
        );

        $this->mailer->send($email);

        return self::SUCCESS;
    }

To header present but empty, From & Sender headers not present.

Test 3: Sendinblue 2 (with replyTo)

public function execute(InputInterface $input, OutputInterface $output): int
    {
        $email = $this->emailFactory->create(
            OrganizationTestFactory::createForTest(),
            false,
            new RecipientDto(new Address('[email protected]')),
            'Sendinblue 2',
            new \DateTimeImmutable(),
            'Sendinblue 2',
            'Content'
        );
        $email->replyTo('[email protected]');

        $this->mailer->send($email);

        return self::SUCCESS;
    }

To header present but empty, From & Sender headers not present. Reply-To header not present.

Here is the outpur of our Messenger consumer:

17:06:04 INFO      [datadog] Handled message Symfony\Component\Mailer\Messenger\SendEmailMessage by App\ThirdParty\Symfony\Mailer\MessageHandler\SendEmailHandler::__invoke
[
  "mainIdentifier" => null,
  "uniqueId" => "63bd8ce72f59d",
  "class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage",
  "message" => [
    "message" => App\ThirdParty\Symfony\Mailer\Email\LocaleAwareEmail^ {
      -locale: "fr_FR"
      -contactId: null
      -createdFor: DateTimeImmutable @1673366759 {
        date: 2023-01-10 17:05:59.187237 Europe/Berlin (+01:00)
      }
      -text: "Content"
      -textCharset: "utf-8"
      -html: """
        <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">\n
        <html><body><p>Content</p></body></html>
        """
      -htmlCharset: "utf-8"
      -attachments: []
      -cachedBody: null
      -headers: Symfony\Component\Mime\Header\Headers^ {
        -headers: [
          "to" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "To"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "from" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {#1
                  -address: "[email protected]"
                  -name: "Sendinblue 2"
                }
              ]
              -name: "From"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "sender" => [
            Symfony\Component\Mime\Header\MailboxHeader^ {
              -address: Symfony\Component\Mime\Address^ {#1}
              -name: "Sender"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "subject" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "Sendinblue 2"
              -name: "Subject"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "bcc" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "Bcc"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-brand" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "assoconnect"
              -name: "X-Brand"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-through" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "sendinblue"
              -name: "X-Sent-Through"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-by" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "SENDINBLUE"
              -name: "X-Sent-By"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-from" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "/var/www/assoconnect/bin/console#L11"
              -name: "X-Sent-From"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "reply-to" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "Reply-To"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ]
        ]
        -lineLength: 76
      }
      -body: null
      -message: null
    },
    "envelope" => null
  ],
  "handler" => "App\ThirdParty\Symfony\Mailer\MessageHandler\SendEmailHandler::__invoke"
]
17:06:04 INFO      [messenger] Symfony\Component\Mailer\Messenger\SendEmailMessage was handled successfully (acknowledging to transport).
[
  "class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"
]

Test 4: Mailgun 2 (with replyTo)

public function execute(InputInterface $input, OutputInterface $output): int
    {
        $email = $this->emailFactory->create(
            OrganizationTestFactory::createForTest(),
            false,
            new RecipientDto(new Address('[email protected]')),
            'Mailgun 2',
            new \DateTimeImmutable(),
            'Mailgun 2',
            'Content'
        );
        $email->replyTo('[email protected]');

        $this->mailer->send($email);

        return self::SUCCESS;
    }

To, From, Sender, Reply-To headers correctly set up & filled.

Here is the debug verbosity output of our Messenger consumer:

17:03:12 INFO      [datadog] Handled message Symfony\Component\Mailer\Messenger\SendEmailMessage by App\ThirdParty\Symfony\Mailer\MessageHandler\SendEmailHandler::__invoke
[
  "mainIdentifier" => null,
  "uniqueId" => "63bd8c3b6ccbe",
  "class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage",
  "message" => [
    "message" => App\ThirdParty\Symfony\Mailer\Email\LocaleAwareEmail^ {
      -locale: "fr_FR"
      -contactId: null
      -createdFor: DateTimeImmutable @1673366587 {
        date: 2023-01-10 17:03:07.422657 Europe/Berlin (+01:00)
      }
      -text: "Content"
      -textCharset: "utf-8"
      -html: """
        <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">\n
        <html><body><p>Content</p></body></html>
        """
      -htmlCharset: "utf-8"
      -attachments: []
      -cachedBody: null
      -headers: Symfony\Component\Mime\Header\Headers^ {
        -headers: [
          "to" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "To"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "from" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {#1
                  -address: "[email protected]"
                  -name: "Mailgun 2"
                }
              ]
              -name: "From"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "sender" => [
            Symfony\Component\Mime\Header\MailboxHeader^ {
              -address: Symfony\Component\Mime\Address^ {#1}
              -name: "Sender"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "subject" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "Mailgun 2"
              -name: "Subject"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "bcc" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "Bcc"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-brand" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "assoconnect"
              -name: "X-Brand"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-through" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "mailgun_assoconnect_transactional"
              -name: "X-Sent-Through"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-by" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "MAILGUN"
              -name: "X-Sent-By"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "x-sent-from" => [
            Symfony\Component\Mime\Header\UnstructuredHeader^ {
              -value: "/var/www/assoconnect/bin/console#L11"
              -name: "X-Sent-From"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ],
          "reply-to" => [
            Symfony\Component\Mime\Header\MailboxListHeader^ {
              -addresses: [
                Symfony\Component\Mime\Address^ {
                  -address: "[email protected]"
                  -name: ""
                }
              ]
              -name: "Reply-To"
              -lineLength: 76
              -lang: null
              -charset: "utf-8"
            }
          ]
        ]
        -lineLength: 76
      }
      -body: null
      -message: null
    },
    "envelope" => null
  ],
  "handler" => "App\ThirdParty\Symfony\Mailer\MessageHandler\SendEmailHandler::__invoke"
]
17:03:12 INFO      [messenger] Symfony\Component\Mailer\Messenger\SendEmailMessage was handled successfully (acknowledging to transport).
[
  "class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"
]

EML complete files are downloaded here: email_files.zip

I've managed to get that the HttpClientInterface instance used in our code is TraceableHttpClient, but I am unable to get the results of a dump($options) from inside TraceableHttpClient::request().
Happy to get insights on what I can do to help debug this further, to really see if the issue comes from the Symfony package or if it is Sendinblue API that is not working properly (I wanted to get the full details on the final http call to Sendinblue API but I do not know where & how to look at it).

stof · 2023-01-10T16:55:59Z

To header present but empty, From & Sender headers not present.

In your Sendinblue 1.eml file in the zip, there is a From: "Sendinblue 1" <[email protected]> header (near the end of the list of headers)

stof · 2023-01-10T16:57:53Z

Can you try the same without using your custom LocaleAwareEmail (to be sure to rule out whether it is part of the issue or no) ?

TristanPouliquen · 2023-01-11T10:37:37Z

Hello @stof, Thank you for your time on this issue.

I simplified my test code to drop any custom layer we had:

public function execute(InputInterface $input, OutputInterface $output): int
    {
        $email = new Email();
        $email->subject("Sendinblue 1")
            ->to('[email protected]')
            ->from('[email protected]')
            ->html('<p>Hello</p>')
            ->text('Hello')
            ->sender(new Address('[email protected]', 'My App'))
            ->replyTo(new Address('[email protected]', 'Reply'));

        $email->getHeaders()
            //->addTextHeader('X-Transport', 'mailgun_assoconnect_transactional');
            ->addTextHeader('X-Transport', 'sendinblue');

        $this->mailer->send($email);
        return self::SUCCESS;
    }

Results are still similar:

Sending through Mailgun gives me all correct headers;
Sending through Sendinblue gives me partial headers.

symfony_email_files.zip

Do you have an idea on how I could get the final payload made to the Sendinblue endpoint?

This seems to be the step to ensure if the issue is within this package (and what it might be) or if it's on Sendinblue's side.

stof · 2023-01-11T11:44:46Z

This is definitely weird, because my own usage of the SendinblueApiTransport has a non-empty To and the Reply-To header.

TristanPouliquen · 2023-01-11T15:43:43Z

Would you have an idea on how I can get the definitive payload sent to Sendinblue to ensure it contains all fields and is correctly formatted?

TristanPouliquen · 2023-01-12T09:35:20Z

I've managed to get a dump working, I think I wasn"t restarting my Messenger worker so that's why I wasn't seeing my new dumps.

In CurlHttpClient, here what is in $options:

^ array:28 [
  "headers" => array:6 [
    0 => "api-key: <my-api-key>"
    1 => "Content-Type: application/json"
    2 => "Accept: */*"
    3 => "Content-Length: 320"
    4 => "User-Agent: Symfony HttpClient/Curl"
    5 => "Accept-Encoding: gzip"
  ]
  "on_progress" => Closure(int $dlNow, int $dlSize, array $info)^ {#9277
    class: "Symfony\Component\HttpClient\TraceableHttpClient"
    this: Symfony\Component\HttpClient\TraceableHttpClient {#1155 …}
    use: {
      $traceInfo: & []
      $onProgress: null
    }
  }
  "normalized_headers" => array:4 [
    "api-key" => array:1 [
      0 => "api-key: <my-api-key>"
    ]
    "content-type" => array:1 [
      0 => "Content-Type: application/json"
    ]
    "accept" => array:1 [
      0 => "Accept: */*"
    ]
    "content-length" => array:1 [
      0 => "Content-Length: 320"
    ]
  ]
  "query" => []
  "body" => "{"sender":{"email":"[email protected]","name":"My App"},"to":[{"email":"[email protected]"}],"subject":"Sendinblue Test","replyTo":{"email":"[email protected]","name":"Reply"},"textContent":"Hello","htmlContent":"\u003Cp\u003EHello\u003C\/p\u003E","headers":{"Sender":"My App \[email protected]\u003E"}}"
  "user_data" => null
  "max_redirects" => 20
  "http_version" => null
  "base_uri" => null
  "buffer" => true
  "resolve" => []
  "proxy" => null
  "no_proxy" => null
  "timeout" => 60.0
  "max_duration" => 0.0
  "bindto" => "0"
  "verify_peer" => true
  "verify_host" => true
  "cafile" => null
  "capath" => null
  "local_cert" => null
  "local_pk" => null
  "passphrase" => null
  "ciphers" => null
  "peer_fingerprint" => null
  "capture_peer_cert_chain" => false
  "extra" => []
  "auth_ntlm" => null
]

The payload seems to be correctly formed indeed. I will investigate with Sendinblue.

TristanPouliquen · 2023-01-25T10:36:09Z

Hello, the problem has been identified.

Sending additional standard headers (ie Sender in my case) breaks Sendinblue.

Even if they don't recognize that it is a bug that sending data for an unmodifiable header can break other functionalities 🥲

I'll update my code not to define the Sender header when sending with Sendinblue.

stof · 2023-01-25T10:50:00Z

If Sender breaks the API call, it might make sense to add it in the list of headers to bypass in

symfony/src/Symfony/Component/Mailer/Bridge/Sendinblue/Transport/SendinblueApiTransport.php

Line 139 in e480a66

    
           $headersToBypass = ['from', 'to', 'cc', 'bcc', 'subject', 'reply-to', 'content-type', 'accept', 'api-key'];

TristanPouliquen · 2023-01-26T13:07:04Z

Ah, I hadn't seen this part, I will push a PR for it.

This PR was merged into the 5.4 branch. Discussion ---------- [Mailer] add Sender to the list of bypassed headers | Q | A | ------------- | --- | Branch? | 5.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | Fix #48936 | License | MIT | Doc PR | Commits ------- f50bec1 add Sender to the list of bypassed headers

TristanPouliquen added the Bug label Jan 10, 2023

carsonbot added Mailer Status: Needs Review labels Jan 10, 2023

xabbuh added Unconfirmed Status: Waiting feedback labels Jan 10, 2023

TristanPouliquen closed this as completed Jan 25, 2023

xabbuh mentioned this issue Feb 3, 2023

[Mailer] add Sender to the list of bypassed headers #49214

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mailer][Sendinblue] API integration does not work anymore #48936

[Mailer][Sendinblue] API integration does not work anymore #48936

TristanPouliquen commented Jan 10, 2023

fabpot commented Jan 10, 2023

stof commented Jan 10, 2023

TristanPouliquen commented Jan 10, 2023

stof commented Jan 10, 2023

stof commented Jan 10, 2023

TristanPouliquen commented Jan 11, 2023

stof commented Jan 11, 2023

TristanPouliquen commented Jan 11, 2023

TristanPouliquen commented Jan 12, 2023

TristanPouliquen commented Jan 25, 2023

stof commented Jan 25, 2023

TristanPouliquen commented Jan 26, 2023

[Mailer][Sendinblue] API integration does not work anymore #48936

[Mailer][Sendinblue] API integration does not work anymore #48936

Comments

TristanPouliquen commented Jan 10, 2023

Symfony version(s) affected

Description

How to reproduce

Possible Solution

Additional Context

fabpot commented Jan 10, 2023

stof commented Jan 10, 2023

TristanPouliquen commented Jan 10, 2023

Test 1: Mailgun 1

Test 2: Sendinblue 1

Test 3: Sendinblue 2 (with replyTo)

Test 4: Mailgun 2 (with replyTo)

stof commented Jan 10, 2023

stof commented Jan 10, 2023

TristanPouliquen commented Jan 11, 2023

stof commented Jan 11, 2023

TristanPouliquen commented Jan 11, 2023

TristanPouliquen commented Jan 12, 2023

TristanPouliquen commented Jan 25, 2023

stof commented Jan 25, 2023

TristanPouliquen commented Jan 26, 2023