Use Regexp Field Processor
- Andy Brook [Plugin People] (Unlicensed)
- Noah Griffiths
- Andy Brook [Plugin People]
- 1 Introduction
- 2 Configuration
- 2.1 Editing Configurations
- 2.1.1 Config
- 2.1.2 Issue Association
- 2.1.2.1 Creating a simple Value via Regexp
- 2.1.2.2 Creating a compound value via Value Script
- 2.1.2.3 Script Type options Since 3.3.46
- 2.1.2.4 Example Velocity Script
- 2.1.2.5 Script value preview
- 2.1.2.6 Example ECMAScript Since 3.3.46
- 2.1.2.7 Script console and preview
- 2.1.3 Test Case
- 2.1.3.1 Adding a Test Case
- 2.2 Field Mappings
- 2.2.1 Multi-value support
- 2.2.2 Testing
- 2.3 Issue creation
- 2.4 Association with existing issues
- 2.5 Setting comment/description to extracted value only
- 2.1 Editing Configurations
- 3 Related articles
Introduction
The Regexp Field Processor uses Regular Expressions (commonly known as "regexps") to extract values from email content. Issue fields are mapped to these values. In addition, the field processor can be used to associate emails to an existing issue by finding a common regexp-extracted field value.
It is primarily designed for processing emails that have predictable content. A common source for such emails is an automated system such as Bugzilla.
It is recommended to have at least a basic understanding of how Regular Expressions work before attempting to configure this field processor. Useful references can be found from Wikipedia, and this online tutorial.
Just need issue association?
If you just want to create comments from email replies to notifications, then this field processor is not needed. By default, JEMH will use issue keys found in the Subject (the pattern used to search for issue keys can be modified via the Profile>Email>Subject IssueKey (comment) Regexps setting.
The Regexp Field Processor can provide the same behaviour with a 'custom field' of 'issueKey' and the appropriate regular expression.
Just Need Field mappings?
By default the Regex Field Processor will attempt to associate issues by matching a specific Regexp pattern within the Subject. If you do not need this then you can do the following to stop the Regex Field Processor from associating issues. These configuration options are found within Profile > Regex Configuration > Issue Association
Remove the Default Regex from Value Regexp
Remove the Value from Expected Match Value
Configuration
In the target JEMH profile, ensure that Directive processing is enabled by going to Profile>Directives. Then, enable the Regexp Field Processor under Profile>Field Processors:
Once enabled, you should see the Regexp Field Processor section listed at the bottom of the main profile screen. When the Regexp Field Processor is enabled, an example configuration is added:
Editing Configurations
Editing the processor configuration is done by clicking on the pencil icon . Once in edit mode, the Regexp edit screen shows more details about each Match Configuration.
From JEMH 2.2.6 (JIRA 7.4+) the configuration is displayed as 3 tabs.
Config
This tab contains a range of fields that define when the particular configuration should be used.
Field | Description |
|---|---|
Projects | Restricts querying to defined projects when looking for issue association. The first project defined is used for creation of issues. If no projects are defined, another mechanism will be required in order to identify a project, such as Project Mappings or Directives. |
Use Election | If enabled, JEMH will use its standard Field Processor selection method - the field processor that finds the most fields is used. If this setting is disabled, then the Regexp Field Processor will always be the one used once an email has been found to match the From Address or To Address settings (see below). |
Allow Project Mappings | If enabled, Project Mappings are used to supply of additional custom field defaults, and potentially workflow advancement. About Custom Field Defaults & Rules If NO PROJECTS are scoped in the Regexp Config, the ALL RULES are tested, and custom field defaults etc are applied when such a Rule matches. If there PROJECTS are scoped in the Regexp Config, the RULES are NOT tested, as an optimization, and only the Project Mapping custom field values would be applied. |
Reporter | If a user is defined here it will be used as the reporter for issues created by this Field Processor. This takes precedence over profiles Default Reporter and would be used when the sender is not associated with a Jira user. |
To Address | If the email |
From Address | If the email |
Select body part type to process | Choose what part of email bodies are processed in order to find an issue association value (text or HTML). |
Match Many field Values | Enabling this setting means that all matches for Field Mappings will be extracted. If disabled, only the first found match will be extracted. |
Issue Association
The Regexp Field Processor can be used to associate an email with a JIRA issue:
On issue creation a uniquely identifiable value is extracted from the creating email through the use of a configured regular expression/script
This value is set on a configured custom field.
The next time an email is processed by the processor it will extract a value using the same steps
If an existing issue is found to already have this value in the custom field, and is not in a status defined under
Resolved Statuses, an issue association will be madeEnsure that the custom field set has the Search Template set to Free Text Searcher
Field | Description |
|---|---|
Require Value Match | Enabling this setting requires a valid remote system value to be found, meaning that now, many remote systems can be matched and have regexp field extracts that are applicable to that source. The only limitation is that the remote system keys must be unique in format. If you do not enable this, comment association by regexp will not work |
Use Issue Keys in the subject if valid | If enabled, issue keys present in the email subject will be used instead of using its JQL lookup. |
Search in Subject | Where to search for the matching 'value' that is to be stored in a custom field, or to be searched for, for commenting instead of creating new issues if follow-up messages occur. If not checked, the Body is inferred as being the target text to scan. |
Custom Field Name | The name of the custom field that will be assigned the identifying 'id' value from the remote system. |
Value Type | As of JEMH 2.2.6 there are two methods for extracting values (called Value Types). The Value Type of Regexp, combined Value Regexp is the most simple, and enables a single value to be extracted from an email's subject or body. In more complex cases you may wish to combine extracted information from more than one part of the email. Using the Value Type of Script allows a compound value to be created, through the parsing of a Velocity template. |
Script TypeSince 3.3.46 | The scripting type used for the Value Script. Allows installed script engines and languages as well as Velocity. |
Value Regexp | The regular expression that matches the field in either Subject or Body, used for issue association. This regular expression must return the actual value as capture group 1, for example, a regexp of: Ticket #([0-9]{6})would return a 6 digit numeric component of matching text. This value will be initially used to search for an unresolved issue with the given Custom Field containing the value, otherwise, used to populate that field during issue creation |
Value Script | A custom script that extracts values from either the Subject or Body, used for issue association. This value will be initially used to search for an unresolved issue with the given Custom Field containing the value, otherwise, used to populate that field during issue creation. |
Expected Match Value | For configuration time testing purposes, test case content is required, usually from a 'Basic Field Processor' description copy/paste, as the text that ends up in JIRA as an issue description will be the same text that will be scanned. The Expected Match Value will be what the eye determines is the appropriate text value that should be found by the Value Regexp (above), a mismatch or failure of any kind will be reported at configuration time. |
Resolved Statuses | Identifies the optional statuses that should not be included in the search for a matching issue. Normally this would be Resolved,Closed If this field is left empty, JEMH will fallback to excluding issues in Resolved and Closed statuses from matching, even though no statuses have been defined. This behaviour contradicts the user interface and will be improved in a future JEMH release: https://thepluginpeople.atlassian.net/browse/JEMH-6012 |
Creating a simple Value via Regexp
When the Value Type is Regexp, a text field is visible, enabling simple expressions with capture group support (only first capture group is usable).
\[#([0-9]*)\].*Combined with Search In Subject, the following expression matches on the numeric part in the subject with final value 12345678. (the screenshot above shows the Expected Match Value for the compound key example)
Subject: [#12345678] This is a starting email template, update as requiredCreating a compound value via Value Script
The standard Regexp value type is fine to use in most situations, but what if the value you want requires extracting information from more than one place in an email? This is where the Script value type can help. Through the use of scripting, you can extract and combine as many values as you want to create a "compound value".
Script Type options Since 3.3.46
Installed Script Engines can now be selected when creating a Value Script, in additional to legacy Velocity scripting.
To install Script Engines see:
Example Velocity Script
Velocity Script | Email Test Case content |
|---|---|
In the above script, lines starting with ## are Velocity comments that are non-functional. Removing the the hash symbols so that the Preview feature () can be used to send the script to a REST service to execute over the currently persisted Test Case content and return the rendered output (including comments!). This output is compared to the Expected Value Match so you know the expressions are right - IF the inbound content is structured the same AND your HTML processing configuration within the Profile remain constant.
The test case above is the default example that is created automatically along with the first regexp configuration. We make use of two values within the content:
1 | |
|---|---|
2 |
So the capture group (the first within the expression) is extracted and assigned:
1 | |
|---|---|
2 |
The final part of the puzzle is on the last line of the script which actually renders the values into the template:
When the final value is as expected, the comment can be hidden again (prefix with double #). Remember that NEWLINES matter in velocity so don't leave any in the template. JEMH will strip them out later anyway as the final compound Value must be a single line ideally contiguous text.
Script value preview
Example ECMAScript Since 3.3.46
Using the Script Type field an installed Engine using ECMAScript e.g. Oracle Nashorn can be used to create a script which may be preferable to using Velocity.
Using the same Email Test case as the previous example, a similar Script can be made that creates a compound value from multiple sources in the email.
ECMAScript | Email Test Case content |
|---|---|
In the script above, the ‘//’ characters mark the line as a comment which will be ignored and are non functional, the ‘print();’ method calls however will output information to the console log, which is visible when using the Preview feature() , and in the JEMH Processing report, helping to debug scripts.
In this script the two values making up the final value are extracted using the following Regexp
1 | |
|---|---|
2 |
These first capture group (the value inside the brackets) is then added to the final value
1 | |
|---|---|
2 |
Finally, the extracted value needs to be set as the ‘valueMatch’ to identify the extracted value as the outcome of the Script.
1 |
|---|
The value set in this variable is the only output from the script that will be used for issue association. If this value is not set, the Value Script wont work.
Script console and preview
Using the preview button will output the console log of the script and show the value of ‘valueMatch’.
Test Case
This isn't a JEMH Test case. Its a combination of (a) Headers and (b) post-processed email content (i.e. not HTML).
Field | Description |
|---|---|
Test Case Email | The test case email content is NOT a full 'source' email, it is just a 'text block' that contains both subject and summary, but after processing by JEMH (exports from JEMH auditing/Test-Cases may be encoded). This field should be populated by Cut/Paste from an issue Description after processing, as that TEXT is what will be used to scan through and extract from |
Adding a Test Case
Test Cases in the Regexp Field Processor are not to be confused with JEMH Test Cases. Here, they are a combination of two aspects:
email headers
decoded text body content
The configuration that is created when the processor is first enabled contains an example (see below). You can use this as a basis for your own specific testing, updating the To: address to match your Profile's catchEmail address, and the From: address to match your remote system sender address.
It is important to make sure you adding just the TEXT (separated from the headers with an empty line). Typically, you can obtain this TEXT either from a JIRA issue description of a created issue (when the field Processors are disabled) or from the text/plain part of a real email.
Field Mappings
Editing a Field Mapping is done from the edit screen, by selecting the pen icon for the rule concerned. New mappings can be added via the New Field Mapping link.
Once edited, the following screen is shown, allowing the custom field to be nominated (must exist) and for the regexp to be supplied (must be valid syntax) as well as the Expected Value (must be found). NOTE the Test Case 'content' is shown at the bottom of the screen for reference when building the field mapping regexp.
If the validation does not succeed, the following error will be shown, indicating the regexp used, expected value and actual value, in this case the edit fails and needs to be redone.
Multi-value support
It is possible to use a Regexp pattern that matches more than one value. If the Regexp configuration has Match Many enabled, all matching values will be used. This Regexp is currently supported at FIELD level, and cannot be used as the issue selector (even if in the body).
Testing
With an issue already received and processed through the Basic Field Processor, it can be converted into a TestCase through the Auditing screen:
Ensure that the desired target profile is selected for the test case, and save. Once saved, the TestCase can be processed by hovering the cursor over the test case and clicking the Run:
Assuming the Reporter of the issue is defined in either the Regexp Field Processor or the Global Default Reporter has appropriate permissions in the target project, an issue will be created:
Issue creation
Reviewing the created issue, as shown below, this shows that:
The subject field was extracted and stored in the bugzillaId custom field.
That custom field 1 and custom field 2 both have values, that match the expected values set earlier (because test case payload data was used).
That custom field 3 does not contain a value - if the field mapping expression does not extract a value, the field will not get a value set by this field processor configuration.
Association with existing issues
When repeat emails come through with the same Custom Field Name value, a search is done for issues with that matching value, if found, as in the case just run, it is used to comment on with the payload of the incoming message. Re-running the test case (refresh the results page) to see if the issue gets commented on:
Viewing the issue now shows that the content has been added as a comment:
Setting comment/description to extracted value only
You are able to set the comment/description fields to a value that you have extract from the email body this way you can keep only the relevant information within the issue description/comment. This is done by having a Field Mapping that has a Field Name of “comment”.
This is an example Regex Field Mapping
Here is an example Test Case.
Viewing the issue page shows that the description is set to the extracted value