Scope

The email service is able to compose and send emails as defined by resources with suitable placeholders.

The emails should be defined by a resource to be easily editable via a dialog. Attachments can be added - either from the email definition, as additional resources (assets) or as computed resources (e.g. a custom PDF for a user).

The configuration of the email server to be used to send the email is also given as input. There might be a system default email server, but in a tenants setting an email server configured by the tenant should be used, to avoid problems with spam blocking.

Usage

The EmailService is part of the cpm-platform-workflow module. It provides a class com.composum.platform.workflow.mail.EmailBuilder on which all data for the email can be provided. The com.composum.platform.workflow.mail.EmailService provides methods sendMail (which returns a Future that returns the message Id when the service was able to send the email) and a method sendMailImmediately which immediately sends the email and returns only on successful delivery to an email relay or on exceptional aborts.

Requirements

For the credentials with mail servers or proxies we should use the credential service.
The mail content should be defined by a resource, containing the text or HTML content of the message, and possibly attachments. It can contain placeholders like ${placeholdername} that are replaced by parameters given for the mail.
The EmailService should be resistent to server crashes: emails that could not yet be delivered to an email relay should be kept in the Repository at /var/composum/platform/mail/queue until successfully delivered to the relay.
If configured, the successfully sent emails should be kept at /var/composum/platform/mail/queue-sent , and emails which could not be sent after several retries should be kept at /var/composum/platform/mail/queue-failed , with automatical cleanup if configured.

Configuration

OSGI Configuration

The Composum Workflow Email Service has an OSGI-configuration in which it is possible to enable it, configure timeouts and retry numbers. It also has a debug mode where all communication with relays is printed to stdout. The service needs to be enabled in the configuration before using it.

A companion service is the Composum Workflow Email Cleanup Service. It can be configured to clean up the email queue entries left in a JCR folder for the successfully sent emails and a JCR folder for the emails that could not be delivered to a SMTP relay, even after the configured number of retries. It can also be configured such that these emails are immediately deleted.

The email service uses a threadpool "EmailSrv" . It is possible to add a configuration with that name to "Apache Sling Thread Pool Configuration" to set the number of threads etc. It's sensible to give it graceful shutdown and a few seconds to finish email deliveries to SMTP relays which are done at the moment of shutdown.

Structure of the email template resource

While it is possible to specify the headers and body of a mail directly with an EmailBuilder, it is also possible to provide an email template covering a type of emails, and just provide all values that change for each individual email. The template can specify various email headers, and a text and / or HTML body for the email, as well as specify attachments. All children of the template resource that are file-resources (nt:file) are added as attachments. The template resource itself can have the following (mostly optional) properties. For mandatory email headers like from, to, subject and body, the values can be either be specified by the template or be set directly on the EmailBuilder. Values set by the various EmailBuilder setters (such as setTo) override values from the template.

jcr:title and jcr:description: not used for sending emails, but are recommended to keep track of the templates
from, to, cc, bcc, replyTo and bounceAddress give the corresponding email headers. These have to be RFC822 compatible (as interpreted by javax.mail.internet.InternetAddress). All except from and bounceAddress can have multiple values.
subject contains the subject of the email.
body contains the plaintext body of the email, html the richtext (that is, HTML) body of the email. It's possible to give both for the benefit of text-only email clients.

The textual headers (subject, body, html) can contain placeholders of the form ${placeholdername} whose values have to be specified to the EmailBuilder.

Structure of the email relay configuration

It is possible to use different email relays for different emails. This seems especially important in a multi-tenant setting, to avoid that that one tenant who's sending broken or even spam emails could be responsible for a shared email relay being blacklisted, thus harming all tenants. So, for every sent email a server configuration resource has to be given as argument to the EmailService. Configurable are:

jcr:title and jcr:description: not used for sending emails, but are recommended to keep track of the configurations
host, port: the hostname and port for the email relay
connectionType: one of SMTP, SMTPS, STARTTLS . Determines the protocol type.
credentialId: an id for the CredentialService that allows to retrieve the username and password for the email relay.
sling:resourceType: is not required, but could be composum/platform/workflow/components/mail/mailserverconfig to enable editing

Implementation

Queueing

The mail queue is held below /var/composum/platform/mail/queue . If the tenant module is present, we add a subdirectory with the tenant's name. The EmailServiceImpl tries to send new emails immediately, but also writes those in the queue for persistence. Periodically, the EmailServiceImpl queries all emails whose retry time arrived (that is, the nextTry property is less than System.currentTimeMillis()).

Emails which were successfully sent are kept in /var/composum/platform/mail/queue-sent if so configured in the cleanup service. When the delivery to an SMTP relay fails, they are kept in /var/composum/platform/mail/queue-failed for manual inspection, also subject to automatic cleanup.

Properties of a mail queue entry

jcr:created : the time the entry was created
jcr:lastModified : the time the entry was last modified
loggingId: a random ID that appears in all log messages to allow connecting the individual retry attempts.
email: the email written in MIME format
serverConfig: the path to the email relay configuration that should be used for sending this email
credentialKey: optionally a key the service needs to access the credentials with the CredentialService .
retry: the number of the retry that was done. 0 is the first immediate sending attempt
nextTry: the time when the next retry should be done
queuedAt: ID of the EmailService that is currently trying to process the message.

Precautions for Clustering

Since it is possible to run the Sling servers in a cluster, several EmailService implementations could run in parallel trying to work on Email retries. So, basically we need to lock the queue entry before sending an email. Since in our experiments JCR locking didn't work right in a cluster, we use the following protocol to ensure no email is sent twice:

the service tries to reserve the queue entry by setting its queuedAt attribute to his own service id and increasing retry and the nextTry time.
the actual sending is done at least a couple of seconds later. Then it checks whether the queue entry still has the same value for the queuedAt attribute, and only in this case sends it. Otherwise it doesn't do anything for that queue entry, since another cluster entry won the "race" of the locking in 1.

To avoid that only one server in a cluster does all emails, in each run every 10 seconds a server reserves only a third of the available queue entries for himself, or 10 if that's larger.

Logging

Each email is provided with a random logging ID that is mentioned in every log message about the sending process. The EmailServiceImpl logs an info message ("Assigning logId ...") with the logging ID, the email template and server configuration path, the tenant's ID (if given), possibly overriding subject (placeholders not resolved) and the String.hashCode of the email addresses. (This way we hide the actual email addresses somewhat, but are still able to find emails to a specific address in the log, up to hashcode collisions.)

If the email is sent, the messageid is logged in a separate message ("Successfully sent email ...") together with the logging ID. If you run Sling in a cluster: please be aware that this can be on a different host in the cluster, though we do try to do it on the same host.

Choice of library for sending emails

There are several options for actually sending emails.

the basic javax.mail , which isn't too pleasant to use.
Apache Commons-Email which simplifies much of the usage. It's easily deployable as OSGI bundle, but the last version is from 2017.
Simple Java Mail has the most features and (subjectively) the best interface, but is currently hard to deploy in OSGI due to a bug, and consists of quite a number of bundles.

So it currently (8/2020) seems the best option to use commons-email for now.

Discussion

Testing

If the test code of cpm-platform-workflow is deployed, there is a test form for sending arbitrary emails at http://localhost:9090/apps/composum/platform/workflow/test/mail/mailform.html . Of course, it needs accessible relay configurations and credentials for the user using it.

Open points

So far there is no real way to protect the passwords for the email servers against access by malicious code using Java reflection. Possibly it would be possible to run Sling with an enabled security manager, but that'd take quite some effort to do it right.
It would be nice if we could check something about the current user, either in the CredentialService or in the email service, to see whether he is allowed to access credentials / send emails. But it is not clear how to do that in a way safe against malicious code.

Possible extensions

There could be a default email server configuration if none is given. (Problem with this: there should be a limitation to which user can send emails using the service.)
For AEM there are several predefined placeholders (e.g. time, username of sending user etc.) Since the EmailService uses the Composum platform PlaceholderService, it is already able to access tenant properties as placeholders. We could provide additional value providers for more generally available placeholders. (Of course, every application using the EmailService can provide it's own placeholder values for each email.)
There could be a console integration for managing credentials, email templates and email relay configurations. There are already some rudimentary components which are (as of 09/20) providing only a dialog: composum/platform/workflow/components/mail/emailtemplate and composum/platform/workflow/components/mail/mailserverconfig .
At the moment it is not possible to reference attachments to the mail within the email text, e.g. as inline images. There could be a special placeholder to do that - the used library commons-email does have some support for that.
Possibly some explicit language support? At the moment this can be done by using separate email templates for different languages.

Not part of the EmailService, but related:

Support of email sending during transitions within the workflow module.
Support for receiving email messages, e.g. to integrate those into workflows. But that'd be quite another project.

Email Service

Workflow Engine