Suppose you need to send a daily digest email to your users. The contents are personalized so you cannot use a classical newsletter provider to offload the work with a single daily campaign.

With a small number of subscribers it’s not an issue, write a cron hook, prepare each mail for the user and send it through the MailManager. You’ll probably still want to use a service such as Mailgun to handle the actual delivery but that’s something you can add irrespective of your cron job.

When your subscriber count increases, this approach becomes an issue. Especially if you try to send your mail in a specific time window, such as within two hours during the night. The maximum throughput we managed to get from such a provider with this approach is about 1 mail per second, due to sending each mail sequentially. Given our constraints we would top out around 7’200 subscribers; and our client’s subscriber list was an order of magnitude larger.

Note that the async functionality of these providers’ modules does not help here, it puts the sending of the mail in the queue so your frontend request isn’t blocked. It does not aggregate the mails for faster delivery.

Batch mailings

To avoid this bottleneck we need to make use of Mailgun’s per-recipient variables to send multiple mails in one request (other services likely have similar features but are out of scope for this post). Thus we forego the entire Drupal mail functionality which is tailored to only processing single messages at a time (or at least I haven’t found an elegant way to do it with MailManager). We can actually use the Mailgun module with very little overhead to do this:

# From the constructor, injecting @config.factory and @mailgun.mail_handler
$this->config = $config_factory->get('mailgun.settings');

$converter = new Html2Text($your_html_markup_containing_variables);
$message = [
  'from_email' => 'from@example.com',
  'from_name' => $this->t('Example sender'),
  'to' => $array_of_recipients,
   'subject' => $subject,
   'html' => $your_html_markup_containing_variables,
   'text' => $converter->getText(),
   'recipient-variables' => json_encode($per_recipient_variables),
];
// The following copied for simplicity out of the mail handler usage.
if ($this->config->get('test_mode')) {
  $message['o:testmode'] = 'yes';
}
$track_opens = $this->config->get('tracking_opens');
if (!empty($track_opens)) {
  $message['o:tracking-opens'] = $track_opens;
}
$track_clicks = $this->config->get('tracking_clicks');
if (!empty($track_clicks)) {
  $message['o:tracking-clicks'] = $track_opens;
}
$this->mailgunHandler->sendMail($message);

We are calling this code in batches of 100 subscribers. The above is basically all there is to it, the majority of our effort had to go into making this work with multiple languages. Note: In our case, subscribers could actually have multiple subscriptions.Therefore we had to make sure to not have the same email twice in a batch since Mailgun uses the email as the key.

Rendering and internationalization

What was glossed over above is the actual contents of the html array element. We assume you want to send HTML emails in general. Since Drupal by itself also sends email with the mail() command the question now becomes how to harmonize those. We decided to define a separate Twig template in our custom module that before everything else does {% extends 'swiftmailer.html.twig' %} . That way we use the same basic markup in our code as Drupal’s regular mails with Swiftmailer do.

If you only have a single language you can directly build the appropriate render array and render it with renderRoot() , ready for use with Mailgun.

If you have more than one language, the first thing is to inject stringTranslation and then call $this->stringTranslation->setDefaultLangcode($langcode); before rendering to create the HTML for the relevant language.

With this approach you’ll now need to group your batches by language to avoid sending a recipient the template from an incorrect language.

Pre-computing recipient data

We decided to create a helper table in the database to hold this information since no other place was ideal to keep track of the data we needed. Our case required: a UUID, a template key with language (also allowing for more than one type of mass mailing to be sent), and the actual variable data as serialized JSON.

Our recipient data contained multi-language display of nodes, so we cannot rely on the language context of the cron job. Additionally, setting the language within string translation was not enough for us because we use entities in our content. In those cases you need a bit more scaffolding, i.e. inject '@string_translation', '@language.default', '@language_manager' and then reset per recipient:

# Resetting language
$this->stringTranslation->setDefaultLangcode($langcode);
$language = $this->languageManager->getLanguage(langcode);
$this->languageDefault->set($langcode);
$this->languageManager->reset();
# Example of embedding entities
$node = $this->entityTypeManager->getStorage('node')->load($node_id_to_embed);
  if ($node->hasTranslation($this->language)) {
    $node = $node->getTranslation($this->language);
  }
  $nodes[] = $this->entityTypeManager->getViewBuilder('node')->view(
    $node,
    'my_email_view_mode',
    $langcode
  );
}

We can thus :

  • Add all subscribers to a queue.
  • Have them be added to the helper table by a QueueWorker one by one.
  • And finally send them all at once keyed by template & language combination.

We only load each template HTML once before sending the batches by looking at what templates are currently present in the helper table.

This approach also makes skipping duplicate addresses (as noted above) easy to handle: Only remove from the helper table the UUID which were actually sent and continue until the table is empty.

Results

With this approach we can send about 150 mails per second, thus delivering our notifications in a few minutes instead of several hours. We could probably increase that even more by optimizing the size of the batch we send. We spend the majority of our computation time now actually assembling the relevant data per user from Search API and Entity API.

Have a great delivery ✉