Use Regexp Field Processor

Introduction

The Regexp Field Processor uses Regular Expressions (commonly known as "regexps") to extract values from email content. Issue fields are mapped to these values. In addition, the field processor can be used to associate emails to an existing issue by finding a common regexp-extracted field value.

It is primarily designed for processing emails that have predictable content. A common source for such emails is an automated system such as Bugzilla.

It is recommended to have at least a basic understanding of how Regular Expressions work before attempting to configure this field processor. Useful references can be found from Wikipedia, and this online tutorial. Use of JEMH's integrated Regexp Tester Tool is recommended in order to make sure your regular expressions work as intended!

Just need issue association?

If you just want to create comments from email replies to notifications, then this field processor is not needed. By default, JEMH will use issue keys found in the Subject (the pattern used to search for issue keys can be modified via the Profile>Email>Subject IssueKey (comment) Regexps setting.

The Regexp Field Processor can provide the same behaviour with a 'custom field' of 'issueKey' and the appropriate regular expression.

Configuration

In the target JEMH profile, ensure that Directive processing is enabled by going to Profile>Directives.  Then, enable the Regexp Field Processor under Profile>Field Processors:

Once enabled, you should see the Regexp Field Processor section listed at the bottom of the main profile screen.  When the Regexp Field Processor is enabled, an example configuration is added:

Editing Configurations

Editing the processor configuration is done by clicking on the pencil icon .  Once in edit mode, the Regexp edit screen shows more details about each Match Configuration.

From JEMH 2.2.6 (JIRA 7.4+) the configuration is displayed as 3 tabs.

Config

This tab contains a range of fields that define when the particular configuration should be used.

Field

Description

Field

Description

Projects

Restricts querying to defined projects when looking for issue association.

The first project defined is used for creation of issues.  If no projects are defined, another mechanism will be required in order to identify a project, such as Project Mappings or Directives.

Use Election

If enabled, JEMH will use its standard Field Processor selection method - the field processor that finds the most fields is used.  If this setting is disabled, then the Regexp Field Processor will always be the one used once an email has been found to match the From Address or To Address settings (see below).

Allow Project Mappings

If enabled, Project Mappings are used to supply of additional custom field defaults, and potentially workflow advancement.

Reporter

If a user is defined here it will be used as the reporter for issues created by this Field Processor.  This takes precedence over profiles Default Reporter and would be used when the sender is not associated with a Jira user.

To Address

If the email To address matches the address entered here, the Regexp Field Processor can ensure that is the one used over other processors (if Use Election is disabled).

From Address

If the email From address matches the address entered here, the Regexp Field Processor can ensure that is the one used over other processors (if Use Election is disabled).

Select body part type to process

Choose what part of email bodies are processed in order to find an issue association value (text or HTML).

Match Many field Values

Enabling this setting means that all matches for Field Mappings will be extracted.  If disabled, only the first found match will be extracted.

Issue Association

The Regexp Field Processor can be used to associate an email with a JIRA issue:

  • On issue creation a uniquely identifiable value is extracted from the creating email through the use of a configured regular expression/script

  • This value is set on a configured custom field.

  • The next time an email is processed by the processor it will extract a value using the same steps

  • If an existing issue is found to already have this value in the custom field, and is not in a status defined under Resolved Statuses, an issue association will be made

  • Ensure that the custom field set has the Search Template set to Free Text Searcher

Field

Description

Field

Description

Require Value Match

Enabling this setting requires a valid remote system value to be found, meaning that now, many remote systems can be matched and have regexp field extracts that are applicable to that source. The only limitation is that the remote system keys must be unique in format.

If you do not enable this, comment association by regexp will not work

Use Issue Keys in the subject if valid

If enabled, issue keys present in the email subject will be used instead of using its JQL lookup.

Search in Subject

Where to search for the matching 'value' that is to be stored in a custom field, or to be searched for, for commenting instead of creating new issues if follow-up messages occur. If not checked, the Body is inferred as being the target text to scan.

Custom Field Name

The name of the custom field that will be assigned the identifying 'id' value from the remote system.

Value Type

As of JEMH 2.2.6 there are two methods for extracting values (called Value Types).  The Value Type of Regexp, combined Value Regexp is the most simple, and enables a single value to be extracted from an email's subject or body.

In more complex cases you may wish to combine extracted information from more than one part of the email.  Using the Value Type of Script allows a compound value to be created, through the parsing of a Velocity template.

Script TypeSince 3.3.46

The scripting type used for the Value Script. Allows installed script engines and languages as well as Velocity.

Value Regexp

The regular expression that matches the field in either Subject or Body, used for issue association.  This regular expression must return the actual value as capture group 1, for example, a regexp of: Ticket

1 #([0-9]{6})

would return a 6 digit numeric component of matching text. This value will be initially used to search for an unresolved issue with the given Custom Field containing the value, otherwise, used to populate that field during issue creation

Value Script

A custom script that extracts values from either the Subject or Body, used for issue association. This value will be initially used to search for an unresolved issue with the given Custom Field containing the value, otherwise, used to populate that field during issue creation.

Expected Match Value

For configuration time testing purposes, test case content is required, usually from a 'Basic Field Processor' description copy/paste, as the text that ends up in JIRA as an issue description will be the same text that will be scanned. The Expected Match Value will be what the eye determines is the appropriate text value that should be found by the Value Regexp (above), a mismatch or failure of any kind will be reported at configuration time.

Resolved Statuses

Identifies the optional statuses that should not be included in the search for a matching issue. Normally this would be Resolved,Closed

If this field is left empty, JEMH will fallback to excluding issues in Resolved and Closed statuses from matching, even though no statuses have been defined. This behaviour contradicts the user interface and will be improved in a future JEMH release: https://thepluginpeople.atlassian.net/browse/JEMH-6012

Creating a simple Value via Regexp

When the Value Type is Regexp, a text field is visible, enabling simple expressions with capture group support (only first capture group is usable).

1 \[#([0-9]*)\].*

Combined with Search In Subject, the following expression matches on the numeric part in the subject with final value 12345678. (the screenshot above shows the Expected Match Value for the compound key example)

1 Subject: [#12345678] This is a starting email template, update as required

Creating a compound value via Value Script

The standard Regexp value type is fine to use in most situations, but what if the value you want requires extracting information from more than one place in an email?  This is where the Script value type can help.  Through the use of scripting, you can extract and combine as many values as you want to create a "compound value".

Script Type options Since 3.3.46

Installed Script Engines can now be selected when creating a Value Script, in additional to legacy Velocity scripting.

To install Script Engines see:

Example Velocity Script

Velocity Script

Email Test Case content

Velocity Script

Email Test Case content

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 #set ( $quotedBody = $u.quoteReplacement($body) ) #set ( $pattern1 = $u.createPattern("\nsignificance=([0-9]+)\n") ) #set ( $matcher1 = $u.createMatcher($pattern1, $quotedBody) ) #if ( $matcher1.find() ) ## MATCHER1 Matched #set ( $firstFullGroupMatch = $matcher1.group() ) ## FIRST FULL GROUP : $firstFullGroupMatch #set ( $firstCaptureGroup = $matcher1.group(1) ) ## FIRST CAPTURE GROUP: $firstCaptureGroup #else ## :( No Matcher1 match #end #set ( $pattern2 = $u.createPattern("\nanother field=([0-9]+)\n") ) #set ( $matcher2 = $u.createMatcher($pattern2, $quotedBody) ) #if ( $matcher2.find() ) ## MATCHER2 Matched #set ( $secondGroupMatch = $matcher2.group() ) ## SECOND FULL GROUP: $secondGroupMatch #set ( $secondCaptureGroup = $matcher2.group(1) ) ## SECOND CAPTURE GROUP : $secondCaptureGroup #else ## :( No Matcher2 match #end ${firstCaptureGroup}-${secondCaptureGroup}
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 MIME-Version: 1.0 Received: by 10.223.112.12 with HTTP; Sat, 18 Jun 2011 22:42:26 -0700 (PDT) Date: Sun, 19 Jun 2011 17:42:26 +1200 Message-ID: <BANLkTinB1mfSh+GwOXGNWoL4SyDvOpdBoQ@mail.gmail.com> Subject: [#12345678] This is a starting email template, update as required From: andy@faraway.com To: notest@localhost Content-Type: text/plain; charset=UTF-8 This is a formatted email It has lots of text including some special fields significance=1234 More and more another field=56789 Even more and more, here is a key/value line split over two lines Open Date 04/18/2012 11:55:16

In the above script, lines starting with ## are Velocity comments that are non-functional.  Removing the the hash symbols so that the Preview feature () can be used to send the script to a REST service to execute over the currently persisted Test Case content and return the rendered output (including comments!).  This output is compared to the Expected Value Match so you know the expressions are right - IF  the inbound content is structured the same AND your HTML processing configuration within the Profile remain constant.

The test case above is the default example that is created automatically along with the first regexp configuration.  We make use of two values within the content:

1

1 \nsignificance=([0-9]+)\n

2

1 \nanother field=([0-9]+)\n

So the capture group (the first within the expression) is extracted and assigned:

1

1 $firstCaptureGroup = $matcher1.group(1)

2

1 $secondCaptureGroup = $matcher2.group(1)

The final part of the puzzle is on the last line of the script which actually renders the values into the template:

1 ${firstCaptureGroup}-${secondCaptureGroup}

When the final value is as expected, the comment can be hidden again (prefix with double #).  Remember that NEWLINES matter in velocity so don't leave any in the template.  JEMH will strip them out later anyway as the final compound Value must be a single line ideally contiguous text.

Script value preview

Example ECMAScript Since 3.3.46

Using the Script Type field an installed Engine using ECMAScript e.g. Oracle Nashorn can be used to create a script which may be preferable to using Velocity.

Using the same Email Test case as the previous example, a similar Script can be made that creates a compound value from multiple sources in the email.

ECMAScript

Email Test Case content

ECMAScript

Email Test Case content

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 //Variable for the Extracted Value var extractedVal; //first Pattern var pattern1 = Java.type("java.util.regex.Pattern").compile("\nsignificance=([0-9]+)"); var matcher1 = pattern1.matcher(body); if(matcher1.find()){ print("Matcher 1 Found a match"); //Setting the first Full Group var firstFullGroupMatch = matcher1.group(); print("First Full Group: " + firstFullGroupMatch); //first capture group var firstCaptureGroup = matcher1.group(1); print("First Capture Group: " + firstCaptureGroup); extractedVal = firstCaptureGroup; }else{ print("No Match found"); } //Second Pattern var pattern2 = Java.type("java.util.regex.Pattern").compile("\nanother field=([0-9]+)"); var matcher2 = pattern2.matcher(body); if(matcher2.find()){ print("Matcher 2 Found a match"); //Setting the second Full Group var secondFullGroupMatch = matcher2.group(); print("Second Full Group: " + secondFullGroupMatch); //second capture group var secondCaptureGroup = matcher2.group(1); print("Second capture Group: " + secondCaptureGroup); extractedVal = extractedVal + "-" + secondCaptureGroup; }else{ print("No Match found"); } //Final value print("Final value: " + extractedVal); //set the valueMatch to the extracted Value valueMatch.setMatch(extractedVal);
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 MIME-Version: 1.0 Received: by 10.223.112.12 with HTTP; Sat, 18 Jun 2011 22:42:26 -0700 (PDT) Date: Sun, 19 Jun 2011 17:42:26 +1200 Message-ID: <BANLkTinB1mfSh+GwOXGNWoL4SyDvOpdBoQ@mail.gmail.com> Subject: [#12345678] This is a starting email template, update as required From: andy@faraway.com To: notest@localhost Content-Type: text/plain; charset=UTF-8 This is a formatted email It has lots of text including some special fields significance=1234 More and more another field=56789 Even more and more, here is a key/value line split over two lines Open Date 04/18/2012 11:55:16

In the script above, the ‘//’ characters mark the line as a comment which will be ignored and are non functional, the ‘print();’ method calls however will output information to the console log, which is visible when using the Preview feature() , and in the JEMH Processing report, helping to debug scripts.

In this script the two values making up the final value are extracted using the following Regexp

1

1 \nsignificance=([0-9]+)

2

1 \nanother field=([0-9]+)

These first capture group (the value inside the brackets) is then added to the final value

1

1 2 var firstFullGroupMatch = matcher1.group(); extractedVal = firstCaptureGroup;

2

1 2 var secondCaptureGroup = matcher2.group(1); extractedVal = extractedVal + "-" + secondCaptureGroup;

Finally, the extracted value needs to be set as the ‘valueMatch’ to identify the extracted value as the outcome of the Script.

1

1 2 //set the valueMatch to the extracted Value valueMatch.setMatch(extractedVal);

The value set in this variable is the only output from the script that will be used for issue association. If this value is not set, the Value Script wont work.

Script console and preview

Using the preview button will output the console log of the script and show the value of ‘valueMatch’.

 

Test Case

This isn't a JEMH Test case.  Its a combination of (a) Headers and (b) post-processed email content (i.e. not HTML).

Field

Description

Field

Description

Test Case Email

The test case email content is NOT a full 'source' email, it is just a 'text block' that contains both subject and summary, but after processing by JEMH (exports from JEMH auditing/Test-Cases may be encoded). This field should be populated by Cut/Paste from an issue Description after processing, as that TEXT is what will be used to scan through and extract from

Adding a Test Case

Test Cases in the Regexp Field Processor are not to be confused with JEMH Test Cases. Here, they are a combination of two aspects:

  1. email headers

  2. decoded text body content

The configuration that is created when the processor is first enabled contains an example (see below).  You can use this as a basis for your own specific testing, updating the To: address to match your Profile's catchEmail address, and the From: address to match your remote system sender address.

It is important to make sure you adding just the TEXT (separated from the headers with an empty line).  Typically, you can obtain this TEXT either from a JIRA issue description of a created issue (when the field Processors are disabled) or from the text/plain part of a real email.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 MIME-Version: 1.0 Received: by 10.223.112.12 with HTTP; Sat, 18 Jun 2011 22:42:26 -0700 (PDT) Date: Sun, 19 Jun 2011 17:42:26 +1200 Message-ID: <BANLkTinB1mfSh+GwOXGNWoL4SyDvOpdBoQ@mail.gmail.com> Subject: [#12345678] This is a starting email template, update as required From: andy@faraway.com To: notest@localhost Content-Type: text/plain; charset=UTF-8 This is a formatted email It has lots of text including some special fields significance=1234 More and more another field=56789 Even more and more, here is a key/value line split over two lines Open Date 04/18/2012 11:55:16

Field Mappings

Editing a Field Mapping is done from the edit screen, by selecting the pen icon for the rule concerned. New mappings can be added via the New Field Mapping link.

Once edited, the following screen is shown, allowing the custom field to be nominated (must exist) and for the regexp to be supplied (must be valid syntax) as well as the Expected Value (must be found). NOTE the Test Case 'content' is shown at the bottom of the screen for reference when building the field mapping regexp.

If the validation does not succeed, the following error will be shown, indicating the regexp used, expected value and actual value, in this case the edit fails and needs to be redone.

Multi-value support

It is possible to use a Regexp pattern that matches more than one value.  If the Regexp configuration has Match Many enabled, all matching values will be used.  This Regexp is currently supported at FIELD level, and cannot be used as the issue selector (even if in the body).

1 (Red|Green|Blue)

Testing

With an issue already received and processed through the Basic Field Processor, it can be converted into a TestCase through the Auditing screen:

Ensure that the desired target profile is selected for the test case, and save.  Once saved, the TestCase can be processed by hovering the cursor over the test case and clicking the Run button (green play icon):

Assuming the Reporter of the issue as defined in either the Regexp Field Processor or the Global Default Reporter have appropriate permissions in the target project, an issue will be created:

Issue creation

Reviewing the created issue, as shown below, this shows that:

  1. The subject field was extracted and stored in the bugzillaId custom field.

  2. That custom field 1 and custom field 2 both have values, that match the expected values set earlier (because test case payload data was used).

  3. That custom field 3 does not contain a value - if the field mapping expression does not extract a value, the field will not get a value set by this field processor configuration.

Association with existing issues

JEMH uses the Privileged User (Profile>Security) to perform the search for existing issues with a matching custom field value. From Jira 7 this user needs to have Application Access and the Browse Projects permission.

When repeat emails come through with the same Custom Field Name value, a search is done for issues with that matching value, if found, as in the case just run, it is used to comment on with the payload of the incoming message. Re-running the test case (refresh the results page) to see if the issue gets commented on:

Viewing the issue now shows that the content has been added as a comment:

Related articles