Performance with JEMH

Owned by Andy Brook [Plugin People]
Last updated: Nov 22, 2021
10 min read

Summary

Performance impact can come from a variety of sources, this page covers these, and may explain why JEMH works a particular way to minimise impact.

Where/How JEMH runs

JEMH is a Mail Handler, with an Inbound Mail Server configuration in Jira, the JEMH Mail Handler selects a Configuration Profile to run with. The Jira scheduler manages when and where (DC only) the JEMH mail handler is invoked.

Problems with inbound mail

At the time JEMH is invoked, Jira has received a Memento of the email residing in the mailbox, such that only some core data is available, the mail does not exist ‘in full’ with Jira. As mail servers can close connections for a variety of reasons, when JEMH Auditing is enabled, JEMH will, before any processing, stream the entire email to disk as part of its auditing process. JEMH can be configured to reload this message from disk, in order to solve parsing problems with some email content, by doing this, there ends up being two copies of that message object in memory, a IO cost, but it ensure that mail that can be processed successfully, is.

Network and storage performance of the home folder

JEMH Auditing data is written to the local (Server) or Shared (DC) home folder. If that storage exists on a network, JEMH will be streaming the email content, which can have an IO load.

Inbound Mail Connections don’t scale

Jira is responsible for scheduling mail retrieval, this then point to the first scalability problem with Jira, that adding new Inbound Mail Servers and dedicated Handlers simply won’t scale. Once the number of Incoming Mail Servers and dedicated Mail Handlers is ‘high’ (~80-100), Jira performance will become impacted. This problem is what drove the creation of JEMH a decade ago.

The cause of the performance problem is too many concurrent conversations with incoming mail servers. JEMH solves this, primarily by allowing the ‘need’ for multiple actual mailboxes to be removed through the Project Mapping feature:

Project Mappings allow a mailbox to have many recipient addresses present in the mailbox. JEMH then has rules that can match on these addresses to drive customised issue creation in specific projects, effectively giving the scalability of a ‘dedicated’ mail handler within a single JEMH configuration (referred to as Profiles).

Mailbox address matching ‘catchemail’

As JEMH supports email only users, we need to differentiate between the ‘mailbox’ address and other non-jira recipients. We do this through the ‘catchemail’ address. Any addressee match will be ignored. This feature ensures that JEMH doesn’t create a mail loop (mailing notifications to the inbox that drives more notifications).

Mail Loops

Its worth mentioning that JEMH will break mail loops on receipt, mail coming into Jira found to be sent ‘by’ Jira will not result in issue creation, will be audited and ‘Forwarded’ to users set in the Profile as forward users. Identifying loops can be a challenge:

  • No user/email participant should have an email address that matches a mailbox address

  • No user/email participant should have an email address that is a group address that contains the mailbox address

Processing pre-requisites

JEMH will not process mail if the partition hosting JEMH data (your local home / shared home) is critically low on storage, to ensure system availability. Without this, Jira will have severe performance problems and may crash.

Scalability of JEMH & Project Mappings

Considering it would not be possible to use hundreds of incoming mail handlers in Jira, JEMH offers a viable solution where it can be supported with minimal system impact whilst no mail is found as only one mail server connection is active.

Commenting

When mail is found, it is retrieved as it would have been. If the email is a reply to a JEMH generated notification, the identification of the issue to comment on stops the need to run the Project Mappings.

Creation

When mail is found but is not found to relate to existing issues, the Project Mapping rules are evaluated. The default behaviour is to scan through the projects in the order shown in the JEMH user interface, rules are then evaluated until a match is found, at which point the matching process ends. This ‘first past the post’ is as performant as it can be.

Rule priorities

It is possible to prioritise rules to effect specific outcomes (maybe keywords run first, before rules based on the incoming mailbox run). By the nature of enabling priorities, this has the impact that all rules need to be run, in order to know if they match.

Rule performance

There are several kinds of rule, most are ‘trivial’ in terms of load, eg:

  • Keyword rules are based on the subject, is a simple text scan to determine a match

  • Domain rules are based on the sender or recipient of the mail, its relatively quick, is finite in numbers of matches to be tested

  • Script rules are user defined logic and can do a lot, the performance impact of running a script rule is comparatively high, but that is the cost of the flexibility offered.

If rule priorities are enabled, and customers use a lot of complex Script rules, it will take longer to process them, translating into higher load on the Jira node involved.

Profile Groups

Profile Groups are a performance feature that not customer user will need and should not be used by default or considered until needed.

The reason Profile Groups exist is for when mail coming in to a specific mailbox address needs to be treated differently than all other mail, some settings in JEMH like HTML formatting controls are ‘top level’ in the Profile, and apply to all, its not currently possible to ‘scope’ these requirements to a Project Mapping level. Without Profile Groups, a dedicated JEMH Mail Handler would be needed, which means that mail handler would be invoked separately and that when mail is found ‘not’

Processing Performance

There are several areas that have performance impact:

Body Processing

HTML is a common format of mail, but Jira doesn’t support HTML, so it needs to be parsed from HTML into a subset, historically called ‘wiki markup’. For small email < 32Kb this conversion process is unlikely to be noticeable, if customers have set their ‘maximum’ comment size' to unlimited, and then send in a HTML payload of 20MB of HTML, its going to take CPU time to process. Regardless of what Jira’s limit is set to, JEMH has to convert HTML to markup before taking further action. Where email content is found to exceed Jira’s set limit, JEMH has a variety of options within the Profile, eg to truncate the content, indicating this by truncating and adding ‘(…)’ , its also possible for JEMH to auto-add the full text as an attachment, and refer same in the added comment.

A common problem with incoming ‘replied-to’ mail is that the Jira content the user is replying to ends up getting into the next issue comment. JEMH has functionality to ensure that is removed by regular expression matching on the attribution line:

The Body Delimiter expressions are regular expressions that are expected to match just one ‘line’. They are internally prefixed with a “\n” newline char to limit the maximum number of regexp engine matches to the number of lines in the incoming message. Such expressions are best made with specific pattern/range matching rather than more complex uses. By limiting the potential matches to ‘lines’ versus ‘characters’ we exclude excessive matching to stop problematically large numbers of matches that would translate to CPU time burned / hung mail processing from a users perspective.

Pre-Processing tasks

JEMH has a range of “Pre Proc tasks” that are dynamic functionality to manipulate email at a low level to resolve syntactical problems (like broken addressees or data types) that would otherwise cause the mail to fail to be loaded, requiring manual intervention. These should only be enabled when a problem is found, enabling all of them will incur more load for no real reason.

Custom Body Processing (using Field Processors)

JEMH can process mail in a variety of ways through Field Processors. As with rules, we have a variety of functionality that users can configure using Scripting that can involve other systems can include jql queries against the local Jira as well as more regular expressions for extracting data to custom fields.

Election process

Field Processors go through an election process to establish the winner (who ‘found’ the most values). This means that all enabled Field Processors are run for every mail that comes in. Simply, if its not needed, don’t enable!

There are some performance optimisations taken in some places like the Regexp Field Processor is geared to process mail from systems so will only participate in the election if the sender/recipient matches what is configured.

Attachment Processing

JEMH does a lot of work to keep Jira issues clear of icons and unwanted images. JEMH identifies unwanted images/attachments in several ways, from course grain size alone, or mime-type checks to md5 hash specific-file fingerprint matching. JEMH calculates the hash of incoming attachment data when they are less than 1MB as a performance impact limit.

When JEMH processes an email, we have perhaps two copies of the Email object in memory, in addition, all attachment data we have extracted from the email will also exist in memory until processing is complete.

During issue creation Jira API does not support the provision of attachments at the time an issue is created, so there is a slight time lag between the creation of the issue, and the addition of the attachments.

Use of attachments during issue created transition/notification

When issues are created, any workflow post-functions do not have access to those attachment, they “don't exist” (yet). In general this doesn’t matter, Jira notifications themselves don’t use attachments.

One of the features JEMH has is to support the inline rendering (in the issue) of images pasted inline in the incoming mail, how this is done is documented:

Audit Reports

The inbound mail audit reports contain a performance section that will highlight:

  • mail download time: time taken to stream the mail to disk as part of ensuring we have the full mail content during creation of audit event

  • Free disk space: clarifying storage available. JEMH will not process mail if system storage drops below 250MB.

  • Script rule execution: the time taken to process all rules, and for script evaluation.

  • Processing time: Overall Processing Time

Notifications

Email notification

If an issue was created by an interactive user (rather than by email), once the user hits ‘save’ on the new issue, and Jira workflow has passed, ending wit the firing of an Event. JEMH reacts, renders the notification and adds it to the Jira Mail Queue. Whilst rendering completes, the interactive user is blocked. In the simple case, JEMH does exactly what Jira does, there is no difference, but JEMH offers template customization:

Whilst JEMH stores all template data in the database through the Atlassian ActiveObjects framework, a near-term in-memory LRU cache is used to store Templates. This means that per notification, templates typically do not need db IO to resolve.

As with all templates, the level of customization can impact render time, for example, it is possible to query remote LDAP servers to extract data, adding some seconds to the time taken.

Recipients

Like Jira, JEMH will resolve all parties to be notified (Jira users with accounts, in one pass, and non-jira email only users in a second pass). In all cases, one notification is sent per user to enable per-recipient customized content as well as to limit Personally Identifiable Information of other involved recipients.

Mail on the queue

The ‘emails' created by JEMH that are added to the Mail Queue may contain inline image and/or attachments. The Mail itself doesn’t contain the attachment data in full, but maintains a ‘reference’ to the attachment wherever stored. This can lead to failed mail sends where the related issue attachment was removed before the mail could be sent that referred to it.

Sender manipulation

JEMH enables customers to ‘set’ the sender address of the mail added to the Jira Mail Queue, which can be done to maintain a ‘seamless’ user experience of interacting with ‘one’ address rather than a custom inbox and a different sending address. This kind of change is not validated by JEMH, only testing can validate. If the outbound mailserver permits the addressee set in the Email, all good. If not, the mailserver would (a) default to the connecting mail account as the email’s from: address, (b) fail to send.

Header manipulation

Its also possible in JEMH templates to customize headers of the mail, whilst we try to limit the impact, a determined user can 'break' emails by duplicating headers or writing invalid values.

Diagnosing send fails

JEMH doesn’t so this. Typically mail 'from' postmaster@ addresses are dropped (of course, configurable in JEMH). JEMH auditing will not process such messages beyond creating an audit entry, to prevent a mail loop.

Additional Transports

In addition to Email, JEMH supports Slack and SMS notifications. Like Email notifications are rendered then queued for delivery, JEMH also renders the notifications synchronously (blocking an interactive user) but will add the messages to a queue for delivery, returning control to the user as soon as possible.

Auditing impact

Inbound Mail Auditing

Auditing is possible the largest impact JEMH can have. The retention of mail by JEMH is only meant for diagnosis purposes, it is not long term storage. The auditing ‘retention window’ is variable to cater for lower volume customers (where 3months at 10K messages pcm would not be a problem), higher volume customers with 50K per day will consume a large amount of diskspace. It needs tuning on a per instance basis based on the traffic received.

Where mail results in a successful outcome 99% of the time, it serves little purpose to retain the audit outcome of that success,

Where long-term email receipt records are required, it is recommended to use the mailserver, by copying incoming emails to other folders, or otherwise tag mails read.

Outbound Mail Auditing

This is a feature in progress to allow tracking of mail sent per recipient.

Future work

With Server sun setting and DC in ascendance, we plan some specific features relating to performance:

  • cached script objects to reduce script compile time

  • compartmentalising JEMH execution on a nominated single or set of DC nodes

  • migrate more ‘top level’ configuration like HTML settings into Project Mappings, to reduce need for complex Profile Groups