Diagnosing Slow or Stopped Incoming Mail
- Andy Brook [Plugin People]
- Finn Gale
Problem
Mail retrieval is very slow, and/or blocked/stopped.
Solution
Jira is responsible for pulling mail from the mailserver, which then invokes JEMH. Sometimes not all the email content is available directly, and to avoid arbitrary failures in later retrieval, JEMH will cause the full email content to be retrieved and stored as part of auditing (is JEMH auditing enabled?). If you see a JEMH auditing entry but with non-success outcome linked to the times involved, its possible there is a processing problem. If you see a JEMH auditing entry with only success then JEMH is doing the expected thing, and the problem you see relates to retrieval.
Check your logs
See atlassian-jira-incoming-mail.log on all your DC nodes, repeating stack traces should help you identify the cause.
Common Causes
Database connectivity issue causes scheduled jobs to break
You are using IMAP and large emails are very slow
Its possible that large emails take a really long time due to default data batch sizes. Large mail can take excessively large time to download. JEMH Processing Reports have a Performance section that highlight download time as well as processing time (requires JEMH auditing enabled!)
Jira Quartz Scheduler has all threads blocked with mail retrieval
We have seen that in some cases
Setup Diagnostics
Enabling Auditing - check Outcome
Enable JEMH Auditing, setting the retention period to 1day initially - the longer the retention period, the more disk space will be consumed and the more time will be taken during purging, use a period that makes sense for your email volume. This will show up when JEMH processes a mail, if it continually shows the same message being processed, with a processing outcome of 'Ignore' it is being processed but your Profile is set to ignore it / leave it unread in the mailbox. Typically, this will be due to the email being sent 'to' an address other than the Profile Catchemail, you will have configured the JEMH Profile to ignore 'catchemail' mis-matches, hence the continual re-processing. The 'ignore' outcome is valid if you have other mail handlers that are specifically looking for that recipient in the same mailbox, however, Use Profile Groups are the way to go here, as it enables several profiles to run against 'one' message, with a Default Profile running at the last to guarantee consumption of the email, and that the mail queue is kept clear.
Enabling Logging
To know more, you will need to setup some more detailed logging:
Enable logging for inbound mail: https://confluence.atlassian.com/jirakb/how-to-enable-debugging-mode-for-emails-816520648.html
Enable the inbound mail protocol logging with the launch param: -Dmail.debug=true : https://confluence.atlassian.com/jirakb/mail-and-mail-handlers-troubleshooting-211649643.html
Enable JEMH logging that specifically captures JEMH traffic : Enabling JEMH logging
Common Problems
Mailbox is polled but JEMH doesn't process it
If the mailbox is polled and a JEMH handler is associated, you will see a time-correlated entry in the JEMH log (see above). If you do not, it means JEMH handler was not invoked, period, check mail handler association.
Mails wont even gets removed from the inbox even after marking it as "read".
Check JEMH has an auditing result that was either successful or not successful, the message would be forwarded, and the mailbox message marked read. JEMH really tries hard to keep the mailbox clear of blockages. Its possible the message in the mailbox is not addressed correctly, not matching a JEMH catchemail address. In such a case you could have configured JEMH to 'leave' the message unread.
Logging a support ticket
When a problem has occurred, and the Diagnostics have been setup
1) collect the atlassian-jira-incoming-mail.log and the jemh.log files, ZIP and attach to a mail to support@thepluginpeople.com, referring time, and issue keys if applicable.
2) screenshot the JEMH auditing screen so we can see the related times of JEMH processing invocation, and if the message involved was not successful, a screenshot of the detail drill down, accessed through a right hand hover icon.
3) attach your related Profile XML
4) the name of the mail server you are using e.g. Gmail, Microsoft exchange, internal, etc.
5) the mail protocol you are using POP/IMAP
6) your JEMH and Jira versions
If there is a communication or problem with retrieval, the incoming mail log with protocol logging will have the answer.
If there is a JEMH processing problem, the audit result and JEMH log will have the answer.