"Mail-Time" is NPM package for mail queue management. Build on top of the nodemailer package. Mail-Time made for single-server and horizontally scaled multi-server setups in mind.
Every MailTime instance can have type configured as Server or Client. Server type of MailTime is great for creating an emailing micro-service app.
The main difference between Server and Client type is that the Server handles the queue and sends email. While the Client only adds emails into the queue.
- How it works?
- Features
- Installation
- Meteor.js usage
- Usage examples
- Require/Import
mail-time - Create NodeMailer's transports
- Initiate
mail-time - Initiate two
MailTimeinstances within the single app - Use templates
- Different storage configurations:
- Require/Import
- API
- Custom Templates
- Running tests
- ๐จโ๐ฌ ~94% tests coverage;
- ๐ฆ Two simple dependencies, written from scratch for top performance;
- ๐ข Synchronize email queue across multiple (horizontally scaled) servers;
- ๐ช Bulletproof design, built-in retries.
Redundant solution for email transmission.
Issue - mitigate a single point of failure via persistent queue and re-send attempts
|----------------| |------| |------------------|
| Other mailer | ------> | SMTP | ------> | ^_^ Happy user |
|----------------| |------| |------------------|
The scheme above will work as long as SMTP service is available
or connection between your server and SMPT is up. Once network
failure occurs or SMTP service is down - users won't be happy
|----------------| \ / |------| |------------------|
| Other mailer | --X---> | SMTP | ------> | 0_o Disappointed |
|----------------| / \ |------| |------------------|
^- email lost in vain
Single SMTP solution may work in case of network or other failures
As long as MailTime has not received confirmation what email is sent
it will keep the letter in the queue and retry to send it again
|----------------| / |------| |------------------|
| Mail Time | --X---> | SMTP | ------> | ^_^ Happy user |
|---^------------| / |------| |------^-----------|
\-------------/ ^- We will try later /
\- put it back into queue /
\----------Once connection is back ------/
Rotate email transports by using multiple SMTP providers. MailTime support two strategies backup (rotate when failed) and balancer (round-robin rotation)
|--------|
/--X--| SMTP 1 |
/ ^ |--------|
/ \--- Retry with next provider
|----------------|/ |--------| |------------------|
| Mail Time | ---X--> | SMTP 2 | /->| ^_^ Happy user |
|----------------|\ ^ |--------| / |------------------|
\ \--- Retry /
\ |--------| /
\---->| SMTP 3 |--/
|--------|
It is common to have horizontally scaled "Cluster" of servers for load-balancing and for durability.
Most modern application has scheduled or recurring emails. For example, once a day โ with recent news and updates. It won't be an issue with a single server setup โ the server would send emails at a daily interval via timer or CRON. But in "Cluster" implementation โ each server will attempt to send the same email. MailTime built to avoid sending the same email multiple times to a user from horizontally scaled applications.
For the maximum durability and agility each Application Server can run MailTime in the "Server" mode:
|===================THE=CLUSTER===================| |=QUEUE=|
| |----------| |----------| |----------| | | | |--------|
| | App | | App | | App | | | |-->| SMTP 1 |------\
| | Server 1 | | Server 2 | | Server 3 | | | | |--------| \
| |-----\----| |----\-----| |----\-----| | | | |-------------|
| \---------------\----------------\----------> | |--------| | ^_^ |
| | | |-->| SMTP 2 |-->| Happy users |
| Each "App Server" | | | |--------| |-------------|
| runs MailTime as a "Server" | | | /
| for the maximum durability | | | |--------| /
| | | |-->| SMTP 3 |-----/
| | | | |--------|
|=================================================| |=======|
To split roles MailTime can run on a dedicated machine as micro-service. This case is great for private email servers with implemented authentication via rDNS and PTR records:
|===================THE=CLUSTER===================| |=QUEUE=| |===Mail=Time===|
| |----------| |----------| |----------| | | | | | |--------|
| | App | | App | | App | | | | | Micro-service |-->| SMTP 1 |------\
| | Server 1 | | Server 2 | | Server 3 | | | | | running | |--------| \
| |-----\----| |----\-----| |----\-----| | | | | MailTime as | |-------------|
| \---------------\----------------\----------> | | "Server" only | |--------| | ^_^ |
| | | | | sending |-->| SMTP 2 |-->| Happy users |
| Each "App Server" runs MailTime as | | | | emails | |--------| |-------------|
| a "Client" only placing emails to the queue. | | <-------- | /
| | | --------> | |--------| /
| | | | | |-->| SMTP 3 |-----/
| | | | | | |--------|
|=================================================| |=======| |===============|
- Email Queue - Managed via MongoDB, Redis, or Custom Queue. Storage-based queue will survive server reboots and failures
- Made for horizontally scaled multi-server setups - MailTime is made to run in multi-server environments, like "Clusters", multiple app instances, load balanced solutions, and replications. MailTime is the perfect fit for applications scaled on a single machine, multiple virtual servers, multiple "bare metal" servers, within single or multiple data centers
- Email concatenation - Reduce amount of sent emails to a single user with concatenation, and avoid mistakenly duplicated emails. When "email concatenation" is enabled the same emails (checked by addressee and content) won't be sent twice. If emails are sent multiple times, due to issues in logic or application failures, - enable "email concatenation" to solve this behavior
- Multiple NodeMailer/SMTP transports โ Support for multiple SMPT transports implemented in two modes -
backupandbalancing. Use this feature to reduce the cost of SMTP services and add extra layer of durability. When one of the transports is failing to send an email โmail-timewill switch to the next one - Sending retries โ Built-in retries for failed to send emails due to network or other failures
- Templating โ Built with support of Mustache-like placeholders, see templating docs
To implement Server functionality โ begin with installing nodemailer, although this package meant to be used with nodemailer, it's not added as the dependency, as nodemailer not needed by Client, and to give freedom to choose nodemailer's version to fit every project needs:
npm install --save nodemailerInstall MailTime package:
# for node@>=14.20.0
npm install --save mail-time
# for node@<14.20.0
npm install --save mail-time@=1.3.4
# for node@<8.9.0
npm install --save mail-time@=0.1.7Setup Nodemailer's transports, Queue storage, and MailTime instance
See steps 1-4 below to learn about different parts of MailTime library and how it can get used. From configuration options to sending email
- Require
mail-timepackage - Create NodeMailer's transports
- Initiate
mail-timeserver- a. Connect to Redis; Or
- b. Connect to MongoDB; And
- c. [optionally] initiate
mail-timeas client
- Start sending emails
// import as ES Module
import { MailTime, MongoQueue, RedisQueue } from 'mail-time';
// require as CommonJS
const { MailTime, MongoQueue, RedisQueue } = require('mail-time');For compatibility and flexibility MailTime has no dependency on nodemailer it should be installed and imported manually. Create one or more "SMTP transports" before initializing new MailTime instance.
For details and full list of options available in .createTransport() see nodemailer docs
// transports.js
import nodemailer from 'nodemailer';
// Use DIRECT transport
// and enable sending email from localhost
// install "nodemailer-direct-transport" NPM package:
import directTransport from 'nodemailer-direct-transport';
const transports = [];
const directTransportOpts = {
pool: false,
direct: true,
name: 'mail.example.com',
from: '[email protected]',
};
transports.push(nodemailer.createTransport(directTransport(directTransportOpts)));
// IMPORTANT: Add `.options` to a newly created transport,
// this is necessary to make sure options are available to MailTime package:
transports[0].options = directTransportOpts;
// Private SMTP
transports.push(nodemailer.createTransport({
host: 'smtp.example.com',
from: '[email protected]',
auth: {
user: 'no-reply',
pass: 'xxx'
},
}));
// Google Apps SMTP
transports.push(nodemailer.createTransport({
host: 'smtp.gmail.com',
from: '[email protected]',
auth: {
user: '[email protected]',
pass: 'xxx'
},
}));
// Mailing service (SparkPost as example)
transports.push(nodemailer.createTransport({
host: 'smtp.sparkpostmail.com',
port: 587,
from: '[email protected]',
auth: {
user: 'SMTP_Injection',
pass: 'xxx'
},
}));
export { transports };Create new instance of MailTime in the Server mode, โ it will be able to send and add emails to the queue.
Connecting to Redis before initiating new MailTime instance:
// mail-queue.js
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
import { transports } from './transports.js';
// Use REDIS_URL environment variable to store connection string to MongoDB
// example: "REDIS_URL=redis://127.0.0.1:6379/myapp node mail-micro-service.js"
const redisClient = await createClient({ url: process.env.REDIS_URL }).connect();
const mailQueue = new MailTime({
transports,
queue: new MongoQueue({
client: redisClient,
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
template: MailTime.Template // Use default template
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
export { mailQueue };Connecting to MongoDB before initiating new MailTime instance:
// mail-queue.js
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { transports } from './transports.js';
// Use MONGO_URL environment variable to store connection string to MongoDB
// example: "MONGO_URL=mongodb://127.0.0.1:27017/myapp node mail-micro-service.js"
const mongodb = (await MongoClient.connect(process.env.MONGO_URL)).db('database');
const mailQueue = new MailTime({
transports,
queue: new MongoQueue({
db: mongodb,
}),
josk: {
adapter: {
type: 'mongo',
db: mongodb,
}
},
template: MailTime.Template // Use default template
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
export { mailQueue };Only one MailTime Server instance required to send email. In the other parts of an app (like UI units or in sub-apps) use mail-time in the Client mode to add emails to queue
// mail-queue.js
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
const mailQueue = new MailTime({
type: 'client',
queue: new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect()
}),
});
export { mailQueue };Import created mailQueue where needed and call .sendMail() method. See NodeMailer's message configuration documentation for details
import { mailQueue } from './mail-queue.js';
await mailQueue.sendMail({
to: '[email protected]',
subject: 'You\'ve got an email!',
text: 'Plain text message',
html: '<h1>HTML</h1><p>Styled message</p>'
});MailTime uses separate storage for Queue management and Scheduler. In the example below MongoDB is used for both
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { transports } from './transports.js';
const db = (await MongoClient.connect('mongodb://url')).db('database');
const mailQueue = new MailTime({
queue: new MongoQueue({
db: db,
}),
josk: {
adapter: {
type: 'mongo',
db: db,
}
},
transports,
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
}
});MailTime uses separate storage for Queue management and Scheduler. In the example below MongoDB is used for queue and Redis is used for scheduler
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const mailQueue = new MailTime({
queue: new MongoQueue({
db: (await MongoClient.connect('mongodb://url')).db('database'),
}),
josk: {
adapter: {
type: 'redis',
client: await createClient({ url: 'redis://url' }).connect(),
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});MailTime uses separate storage for Queue management and Scheduler. In the example below Redis is used for queue and MongoDB is used for scheduler
import { MailTime, RedisQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const mailQueue = new MailTime({
queue: new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect(),
}),
josk: {
adapter: {
type: 'mongo',
db: (await MongoClient.connect('mongodb://url')).db('database'),
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});MailTime uses separate storage for Queue management and Scheduler. In the example below Redis is used for both
import { MailTime, RedisQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const redisClient = await createClient({ url: 'redis://url' }).connect();
const mailQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});Create two MailTime instances with different settings. One for urgent (e.g. "transactional" emails), and another one for other types of emails (e.g. "marketing" emails)
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
import { transports } from './transports.js';
const redisClient = await createClient({ url: 'redis://url' }).connect();
// CREATE mailQueue FOR NON-URGENT EMAILS WHICH IS OKAY TO CONCATENATE
const mailQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
}),
transports,
strategy: 'backup',
failsToNext: 1,
concatEmails: true,
josk: {
adapter: {
type: 'redis',
client: redisClient
},
zombieTime: 120000
}
});
// CREATE mailInstantQueue FOR TRANSACTIONAL EMAILS AND ALERTS
const mailInstantQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
prefix: 'instant'
}),
transports,
prefix: 'instant',
retryDelay: 2000,
strategy: 'backup',
failsToNext: 1,
concatEmails: false,
josk: {
adapter: {
type: 'redis',
client: redisClient
},
zombieTime: 20000
}
});
await mailQueue.sendMail({
to: '[email protected]',
subject: 'You\'ve got an email!',
text: 'Plain text message',
html: '<h1>HTML</h1><p>Styled message</p>'
});
await mailInstantQueue.sendMail({
to: '[email protected]',
subject: 'Sign in request',
text: 'Your OTP login code: xxxx:',
html: '<h1>Code:</h1><code>XXXX</code>'
});All options passed to the .sendMail() method are available inside text, html, and global templates
const templates = {
global: '<html xmlns="http://www.w3.org/1999/xhtml"><head><title>{{subject}}</title></head><body>{{{html}}}<footer>Message sent to @{{username}} user ({{to}})</footer></body></html>',
signInCode: {
text: 'Hello @{{username}}! Here\'s your login code: {{code}}',
html: `<h1>Sign-in request</h1><p>Hello @{{username}}! <p>Copy your login code below:</p> <pre><code>{{code}}</code></pre>`
}
};
const mailQueue = new MailTime({
queue: new RedisQueue({ /* ... */ }),
template: templates.global
});
await mailQueue.sendMail({
to: '[email protected]',
subject: 'Sign-in request',
username: 'johndoe',
code: 'XXXXX-YY',
text: templates.signInCode.text,
html: templates.signInCode.html
});All available constructor options and .sendMail() method API overview
opts{object} - Configuration objectopts.type{string} - [Optional]clientorserver, default -serveropts.queue{RedisQueue|MongoQueue|CustomQueue} - Queue storage driver instanceopts.transports{[object]} - [Required for "server"] An array ofnodemailer's transports, returned fromnodemailer.createTransport({}). Required for{type: 'server'}opts.josk{object} - [Required for "server"]JoSkpackage optionsopts.josk.adapter{object|RedisAdapter|MongoAdapter|CustomAdapter} - Config object or Adapter instanceopts.josk.adapter.type{string} - One ofmongoorredis; Passjosk.adapter.typeto avoid burden of creating Adapter instance manuallyopts.josk.adapter.client{RedisClient} - RedisClient instanceopts.josk.adapter.db{Db} - Mongo's Db instanceopts.josk[option]{mix} - Any other options passed toJoSkinstanceopts.from{function} - [Optional] A function which returns string offromfield, format:"MyApp" <[email protected]>opts.strategy{string} - [Optional]backuporbalancer, default -backup. If set tobackup, first transport will be used unless failed to sendfailsToNexttimes. If set tobalancer- transports will be used equally in round robin chainopts.failsToNext{number} - [Optional] After how many failed "send attempts" switch to the next transport, applied only forbackupstrategy, default -4opts.prefix{string} - [Optional] Use unique prefixes to create multipleMailTimeinstances within the same applicationopts.retries{number} - [Optional] How many times resend failed emails, default -60opts.retryDelay{number} - [Optional] Interval in milliseconds between send re-tries, default -60000opts.keepHistory{boolean} - [Optional] By default sent emails not stored in the database. Set{ keepHistory: true }to keep queue task as it is in the database, default -falseopts.concatEmails{boolean} - [Optional] Concatenate email bytofield (e.g. to the same addressee), default -falseopts.concatSubject{string} - [Optional] Email subject used in concatenated email, default -Multiple notificationsopts.concatDelimiter{string} - [Optional] HTML or plain string delimiter used between concatenated email, default -<hr>opts.concatDelay{number} - [Optional] Time in milliseconds while emails are waiting to be concatenated, default -60000opts.revolvingInterval{number} - [Optional] Interval in milliseconds in between queue checks, default -256opts.template{string} - [Optional] Mustache-like template, default -{{{html}}}, all options passed tosendMailis available in Template, liketo,subject,text,htmlor any other custom option. Use{{opt}}for string placeholders and{{{opt}}}for html placeholdersopts.onError(error, email, details){function} - [Optional] called when email has failed to get sent and exhausted all send attempts (opts.retries), called with 3 arguments:error{Error|object} - Error objectemail{object} - email's objectdetails{object} - not always present, details from SMTP protocol
opts.onSent(email, details){function} - [Optional] called when email was successfully handed over to receiving/recipient's SMTP server, called with 2 arguments:email{object} - email's objectdetails{object} - not always present, details from SMTP server/protocol
import { MailTime, MongoQueue, RedisQueue } from 'mail-time';
import nodemailer from 'nodemailer';
import { createClient } from 'redis';
const redisClient = await createClient({ url: 'redis://url' }).connect();
const mailQueue = new MailTime({
type: 'server',
strategy: 'backup',
prefix: 'appMailQueue',
transports: [nodemailer.createTransport({/* ... */})],
failsToNext: 4,
retries: 60,
retryDelay: 60000,
keepHistory: false,
concatEmails: false,
concatDelay: 60000,
concatDelimiter: '<hr>',
concatSubject: 'Multiple notifications',
revolvingInterval: 256,
template: '{{{html}}}',
queue: new RedisQueue({
client: redisClient,
prefix: 'appMailQueue',
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"App Name" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
await mailQueue.sendMail({
to: '[email protected]',
subject: 'Email subject',
text: 'You have got email!',
html: '<p>You have got email!</p>',
});Create Redis Queue instance. Use for opts.queue when creating MailTime instance
opts{object} - Configuration objectopts.client{RedisClient} - Required, Redis'esRedisClientinstance, like one returned fromawait redis.createClient().connect()methodopts.prefix{string} - Optional prefix for scope isolation; use when creating multipleMailTimeinstances within the single application
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect(),
prefix: 'appMailQueue',
});Create MongoDB Queue instance. Use for opts.queue when creating MailTime instance
opts{object} - Configuration objectopts.db{Db} - Required, Mongo'sDbinstance, like one returned fromMongoClient#db()opts.prefix{string} - Optional prefix for scope isolation; use when creating multipleMailTimeinstances within the single application
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
new MongoQueue({
db: (await MongoClient.connect('mongodb://url')).db('database'),
prefix: 'appMailQueue',
});Add email to the queue. Returns Promise<string> unique email's uuid
opts{object} - Configuration objectopts.sendAt{number} - When email should be sent, default -Date.now()opts.template{string} - Email specific template, this will override default template passed toMailTimeconstructoropts.concatSubject{string} - Email specific concatenation subject, this will override default concatenation subject passed toMailTimeconstructoropts[key]{mix} - Other custom and NodeMailer specific options, liketext,htmlandto, learn more here. Note: ifattachmentsare used viapathโ file must exists on all micro-services servers
Remove email from queue. Returns Promise<boolean> โ true if cancelled or false if not found, was sent, or was cancelled previously. Throws Error
uuid{string|promise} โ email'suuidreturned from.sendEmail()method
import { mailQueue } from './mail-queue.js';
const uuid = await mailQueue.sendMail({
to: '[email protected]',
subject: 'Email subject',
text: 'You have got email!',
html: '<p>You have got email!</p>',
});
await mailQueue.cancelMail(uuid);Ping MailTime instance, its scheduler, and its queue. Returns Promise<object>
const mailQueue = new MailTime({ /* ... */ });
const pingResult = await mailQueue.ping();
console.log(pingResult)
/**
In case of the successful response
{
status: 'OK',
code: 200,
statusCode: 200,
}
Failed response
{
status: 'Error reason',
code: 500,
statusCode: 500,
error: ErrorObject
}
*/Simple and bulletproof HTML template, see its source. Usage example:
import { MailTime, MongoQueue, RedisQueue } from 'mail-time';
// Make it default
const mailQueue = new MailTime({
/* .. */
template: MailTime.Template
});
// For single letter
mailQueue.sendMail({
/* .. */
template: MailTime.Template
});Pass custom template via template property to .sendMail() method
mailQueue.sendMail({
to: '[email protected]',
userName: 'Mike',
subject: 'Sign up confirmation',
text: 'Hello {{userName}}, \r\n Thank you for registration \r\n Your login: {{to}}',
html: '<div style="text-align: center"><h1>Hello {{userName}}</h1><p><ul><li>Thank you for registration</li><li>Your login: {{to}}</li></ul></p></div>',
template: '<body>{{{html}}}</body>'
});- Clone this package
- Start local or obtain URLs for remote MongoDB and Redis servers
- In Terminal (Console) go to directory where package was cloned
- Then run:
# Before running tests make sure NODE_ENV === development
# Install NPM dependencies
npm install --save-dev
# DEFAULT RUN
REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# OPTIONALLY RUN WITH CUSTOM DOMAIN
EMAIL_DOMAIN="your-domain.com" REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# IF SOME TESTS ARE FAILING: ENABLE DEBUG
DEBUG="true" REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# Be patient, tests are taking around 8 mins- Upload and share files using โ๏ธ meteor-files.com โ Continue interrupted file uploads without losing any progress. There is nothing that will stop Meteor from delivering your file to the desired destination
- Use โฒ ostr.io for Server Monitoring, Web Analytics, WebSec, Web-CRON and SEO Pre-rendering of a website
- Star on GitHub
- Star on NPM
- Star on Atmosphere
- Sponsor maintainer via GitHub โ support open source with one-time contribution or on a regular basis
- Sponsor veliovgroup via GitHub โ support company behind this package
- Support via PayPal โ support our open source contributions