Use Nagios Field Processor
- Andy Brook [Plugin People] (Unlicensed)
- Ryan Clifton
- Finn Gale
Scenario
You want to integrate Nagios with JIRA, but not only that:
Duplicates should stack up on the same issue as a comment
If a Nagios notification indicates the problem has been resolved the issue should be resolved automatically
Content of the Nagios email body happens to be of the KEY: VALUE format, mapping these to custom fields would be a bonus.
Example emails
Before setting up JIRA or JEMH, the format of the Nagios email needs a review, see CRITICAL and OK messages below.
Example CRITICAL notification
Subject: ** PROBLEM Service Alert: localhost/SSH is CRITICAL **
To: andy@localhost
X-mailer: mail (GNU Mailutils 2.2)
Date: Sat, 26 Nov 2011 16:53:45 +1300 (NZDT)
Message-Id: <20100129115054.BFAE31E8C@trippy>
From: nagios@localhost
Content-Type: text/plain; charset=UTF-8
***** Nagios *****
Notification Type: PROBLEM
Service: SSH
Host: localhost
Address: 127.0.0.1
State: CRITICAL
Date/Time: Sat Nov 26 16:53:45 NZDT 2011
Additional Info:
Connection refusedExample OK service restoration notification
Subject: ** RECOVERY Service Alert: localhost/SSH is OK **
To: andy@localhost
X-mailer: mail (GNU Mailutils 2.2)
Date: Sat, 26 Nov 2011 17:03:30 +1300 (NZDT)
Message-Id: <20100129115054.BFAE31E8C@trippy>
From: nagios@localhost
Content-Type: text/plain; charset=UTF-8
***** Nagios *****
Notification Type: RECOVERY
Service: SSH
Host: localhost
Address: 127.0.0.1
State: OK
Date/Time: Sat Nov 26 17:03:30 NZDT 2011
Additional Info:
SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0)The Nagios notification is structured, the summary specifically has its message type as the first keyword in the message, with a priority at the end. In the CRITICAL example, the message type is PROBLEM, the priority is CRITICAL. The recovery message example has a message type of RECOVERY, with status OK.
All Nagios messages are therefore related, JEMH refers to the related 'PROBLEM/RECOVERY' messages as Phrase Sets (of each Phrase). Each Phrase has an associated Priority, that is mapped to an appropriate JIRA Priority. When a message is received, the associated priority is set on a created issue, or updated if a related issues is found.
Related issues/matching
JEMH uses the subject of the email to match against pre-existing unresolved issues within a specific created time-frame with a 'matching' summary. A matching summary is not just a direct match (which would work) but is also all possible permutations (exchange of value) within the Phrase Set, for example, all the following subjects can relate to the same issue irrespective, so long as the Phrase is listed and the Priority is known.
** RECOVERY Service Alert: localhost/SSH is OK **Payload Mapping
JEMH aggregates the body parsing Colon Suffix Field Processor ( KEY: VALUE ), but with a variation that means every line that starts with KEY: VALUE will be extracted if there is a custom field existing of the same name. Multi-line values are not supported in this mode...
Custom field configuration
From the example email above, the following Custom Fields could be defined (and will be for this example). NOTE: The entire content of the message will still be used as the Issue Description or Comment.
Notification Type
Service
Host
Address
State
Date/Time
JEMH Configuration
First, edit the Field Processors:
Now enable the Nagios Field Processor:
Notice that when saved, JEMH is providing you some configuration advisories, this one is indicating that Directives are disabled, meaning that only Basic field processor will be available (must be set to Create or Comment for this field processor).
To enable Directives, go to Profile>Directives, and set Directive Processing Behaviour to On Create or Comment.
Now, looking at the bottom of the Nagios Field Processor configuration screen, an example configuration has been pre-populated. This will be good enough for the example messages here.
The automatically provisioned project is not valid, it needs fixing to a valid project
Nagios Related Phrases configuration
Within the Nagios configuration there is a default Phrase set that can be used for transitioning the related issues. However, you are able to create your own Phrase Sets to use.
To create a new Phrase Set you will need to:
Press “Add Phrase Set” - to create a new empty Phrase set
To add new Phases you will need to press “Add Phrase” and this will take you to a screen to configure the new Phrase.
You should then be able to see the new Phrase within the new Phrase set
Once this is configured, it will then mean that this Phrase set can be used for creating/updating Nagios issues within Jira.
Creating a Test Case
Paste the example text into a JEMH Test Case email, then save, NOTE: be sure to select the correct Profile that contains the new Nagios Configuration.
Once the test case has been saved, you will then be taken back to the test case list screen. Click the green run icon to execute the test case.
The test case will then be processed, and you will be presented with the Audit Event Details. It should result in an issue being created:
Looking at the created issue, the extracted content can be seen to populate the various Custom Fields created earlier, and that the whole body content message ended up as the issue description:
Subsequent Nagios Updates
To illustrate what happens with repeated 'down' or related messages, go back and re-run the Test Case, notice that the outcome is now a Comment not issue Created, under the same issue key:
Refreshing the issue screen, we now see that the repeat message has commented rather than created a new issue:
Issue Closure
Next we are going to illustrate how issues can be automatically resolved. As before, create and save a Test Case with the recovery example content provided at the top of this page:
Again, hit the green execute button, notice that the result picks out the same issue as before, marking as a comment:
Refreshing the issue screen again, you can see the issue has been automatically resolved!
Future messages of the same subject would create new issues.
Troubleshooting
Nagios Updates not commenting on existing issues
If Nagios messages are not commenting on existing issues and are instead creating new issues, there may be something in the email subject that Nagios provides that is breaking when a JQL query is run to find the associated existing issue. Currently, single hyphens that are not surrounded by characters will cause the association to fail.
Works | Does Not Work |
|---|---|
Phrases may need updating
Its possible that version or deployment configuration change Nagios Phrases. If new Issues are being created, likely, the JEMH Nagios Phrases need updating to include the new Phrases that caused additional issue creation.
Customer Nagios reporter
Nagios reporters in a ‘Customer’ project role must have access to the specified issue and workflow transition.
Basic Mail Processor Used Instead
It may be the case that your Nagios emails contain a reply-to header. If this is the case, you may need to change the From Address Parse Order setting to "From: then Reply-To:". Alternatively, you could try to configure Nagios to send emails with no reply-to header.