Configure Audit History Purging

Owned by Andy Brook [Plugin People] (Unlicensed)
Oct 19, 2022
4 min read

Introduction

JEMH has two methods of removing audit history:

  1. Delete Older / Delete All

  2. Invoke a Purge

The first method is fast and uses date ranging queries to flush the database.  The second is more versatile but slower, and is dependant on database tables being present that refer to the audit files ID.  It is not possible to trigger the second method to purge history on disk if the database tables have been wiped.

Configurations

From the navigation menu click Auditing > Settings > Retention settings.

 

Low Volume

If your email volume is not large then it is possible to retain audit history for a longer period of time, as the overall volume would not be excessive, numbers for this would be for example purposes, up to 100 messages a day.

Consequences of this configuration:

  • Audit History volume in the database will grow proportional to the number of emails received.  Disk space will grow in line with the volume of email.

  • No mail will be purged by scheduled Purging that is within the last week, success or failure.

High Volume

If your volume is large, care must be taken.  If the retention period is too long, the volume of email may result in very large numbers of records in the database as well as multi-gigabyte disk usage.  There are several strategies that can be used to find an appropriate configuration:

  1. Reduce volume retention.  If your retention period is a week, consider lower it further.

  2. Alternatively, enabling Auto purge successes will reduce the volume of retained email substantially.

A suggested configuration would look like:

Consequences of this configuration:

  • All successful mail will be retained only for the duration of processing, and of event notification, all trace will be removed thereafter.

  • If the volume of failures is high, the purge process will be slowed by the volume of retrieving and checking failed records.

Monitoring the Purge Process

Since JEMH 1.4.10, the purge process can be monitored in real-time.  On page load, it checks to see how many Audit Events are candidates for purging, given Auto Delete by Age time constraints, and the percentage of matches versus total Audit History events.  The starting point DateTime for purging is also shown, and can be used to check against current Audit History.  If the Diagnostic screen is used bulk create test issues/events, the few issues per second can be seen clocking up by repeated application of the Check Matches button.



Progress Bars

During the purge process, progress bars are dynamically updated, as well as overall purge statistics (eg disk space reclaimed), to provide a live view of the executing process.

If the configuration above has Retain Failures, the count of these retained events, and the related percentage to total Audit History is also shown.  

 Purging is run automatically when the plugin is installed, once, according to the configured rules above.

After purging is complete, using the Check Matches, the count of Audit Events before the given Timestamp will be updated dynamically.

Manually Triggering Purging

Test Scenario

This example can be used to prove Audit History retention and purging:

1. Created 10K issues via the JEMH Diagnostics tab 
 

2. Check content of audithistory, listing the number of audit history files present: 

andy@sup-jira:/usr/local/apps/jira/home/data/jemh/auditmail$ ls -Rlas | grep txt |wc -l 10000


3. Advance system date couple of days (e.g. from 25th, the day the test content was created) with 
sudo date -s "May 28 19:56:56 BST 2013" 

4. trigger a purge with the following auto delete configuration, by clicking the Manual Purge button on the Auditing screen; 

Auto delete by Age >: Last Hour Auto Delete Purge Time: 00:00:00 Auto purge successes: OFF Retain Failures: ON


5. Monitoring the purge using repeated: 

andy@sup-jira:/usr/local/apps/jira/home/data/jemh/auditmail$ ls -Rlas | grep txt |wc -l


See the purge history dropping by a couple of hundred issues per 10S or so. 

6. Final Purge stats now show: 


This shows the purging is doing what it should, but it might be clear that copying old data into the folder won't be affected by the scheduled purge, only the Delete All. So, If tables have been deleted, use Delete All to start with, then verify there are no files remaining and your disk consumption due to auditing is 0. Then, trial it. If volume is a problem, consider enabling the 'Auto purge successes'.