Use Audit History Reprocessing

Owned by Andy Brook [Plugin People] (Unlicensed)
Last updated: Nov 25, 2024 by Andy Brook [Plugin People]
4 min read

Background

JEMH Audit History saves a copy of the email to disk, allowing further processing (as of JEMH 1.4.7!)

Scenario

Audit History has collected a range of processing failures for a number of reasons:

  • An incorrectly formatted email may not be processable initially, but after serializing through audit history, is actually usable!

  • A configuration issue may have prevented processing the first time round

  • The outside chance that a bug in JEMH may have prevented correct processing!

What Does It Do

Re-processing uses some (currently) hard-coded matching rules to determine what should be Reprocessed.  A Reprocessed Audit History entry may or may not be marked 'Processed'.  Currently the following Reprocessing options are available:

  • Marked as 'Processed' but no Created Issue or Commented Issue is present (possible result of a buggy regular expression, not noticeable pre 1.4.7)

  • Exception is present

Configuring

The new Reprocessing panel in the JEMH Auditing screen looks like this:

The top form allows some rudimentary parameters to be set:

  • the Profile that should be used for the given email

  • the Reprocessing rule to be applied to locate emails

  • the MAX ID to be used (useful to limit the number processed in one go)

Contents of the form must be saved before any related operations will work.

 

The lower half shows a status window. The three operation buttons are:

  • Test - will run the currently configured Processing Type, and indicate how many records matched, without actually doing anything

  • Export - will download a spreadsheet capable CSV text file containing pertinent information about the records involved

  • Run now - will execute processing with parameters from the form

NOTE:

  • Execution may take considerable time, picking a lower MAX ID initially will limit the period of loading

  • Every record processed will re-recreate a new Audit History record, including a copy of the related email.  Watch disk consumption if processing large volumes of email.

  • If an email is processed due to rule matching, its audit history entry will be updated, indicating it is Reprocessed, and will refer to the new Audit Event ID that was the result of the reprocessing.

 

Automated reprocessing

Here is an example Audit History view, showing a few exceptions and a contrived buggy outcome entry:

Noting the maxId involved, the Reprocessing form is updated to reflect (Max ID of 230), and setting the rule type to withException (must be applied).

Checking rule matches

Click the Test button to run the rule and see the amount records that matched, note that 3 were found as shown below.

Download match list

For sanity, the matches can be downloaded as CSV by clicking the Export button:

Running the reanimation

Running the reanimation is just a click on the Run now button:

The process will be started in the background and a progress bar will by dynamically updated in the background:

Refreshing the page will show the previous failed entries marked as Re-processed:

Manual Reprocessing

Its also possible to manually trigger reprocessing, but this doesn't 'clean up' the original entry as the rules driven reprocessing do

Arranging for filtered mail to be reprocessed in bulk

Where a lot of mail has been ‘filtered’ you may want to rerun these, the only current way is to update the db table:

  1. Configure “JEMH > Auditing > Reprocessing” for with an exception logged

  2. Update the JEMH audit records in the db to be scoped in to the “exception” check; here, if you wanted to mark all audit records that had FORWARDED outcome and status FILTERED (syntax for postgresql, update as applicable for your db). You can scope the range of records affected through WHERE "ID" > 0 AND "ID" <= 100.

UPDATE public."AO_78C957_AUDITEVENTS" SET "EXCEPTION"='runme' WHERE "ID" > 0 AND "ID" <= 100 AND "FILTER_ACTION" = 'FORWARDED' AND "REPROCESSED" IS NOT TRUE AND "EXCEPTION" IS NULL;

Monitoring progress

When you have an active authorized http session, you can monitor what is going on through the browser:

http://server:8080/rest/jemh/1.0/api/auditing/reprocessing/status.json

Result:

{"auditId":0,"percent":0,"processingStatus":"notRunning","recordsProcessed":0,"recordsToProcess":0}

Related articles