Processing recovery emails with the advanced settings of the integration. [Example]


#1

OpsGenie allows you to fully customize, how your email integration (or any integration) is handling the incoming messages. Besides the full flexibility and control over the content of your alerts, you can also configure, which action (Ignore, Create, Ack, Close, Add Note) needs to be executed when a message arrives.

The more sophisticated controls are hidden from the simple settings page. The first step is to switch your integration to the Advanced view:

After switching to the Advanced Settings, new options will appear. On the left side, you’ll find the list of Actions. These are little containers, where you can capture rules. One action can be triggered by multiple rules, but one rule can only trigger one action.



These rules are matched in a top-down order - the first matching rule will execute the associated action, so it’s very important you either, set strict filters, or exclude messages. The rules contain two different section: Filters and Alert Fields. If the incoming email is matching the filter section of a rule, the alert will get created/updated with the information captured in the Alert Fields section.

You will need to use a Create Alert rule and a Close Alert rule to set up a recovery logic in your email integration.

Steps:

1.) Add a Create Alert rule (edit the existing one)

2.) Set up a filter. This is usually a condition which is true for any incoming alert emails. (e.g. “subject starts with Alert!”, or “subject contains ‘is DOWN’”, etc). It’s usually beneficial to exclude the recovery emails.

3.) Parse out a unique string from the email and set it as Alias, which identifies the alert. Find all the string processing methods under this link. (This is a very important step - you’ll need to be able to parse out the exact same string from the recovery email as well. This is what “connects” the recovery message later on to the open alert)

4.) Add a Close Alert rule

5.) Set up the filters. This should be true for all recovery emails. (e.g. subject contains “is UP”, or subject starts with “Resolved:”, etc)

6.) Parse out the very same string from the subject, what you parsed in the Create Alert rule, and set it as an alias

If you did correctly, an incoming alert email will match your Create Alert rule, and create the alert according to the Alert Fields section. If the alert is open, and a recovery email has been received, which matched the Close Alert rule - your alert will be closed automatically.

Example Case

Let’s say, we have a basic monitoring tool checking on our websites. Whenever it can’t reach the site, it will send us an email notification with the following subject:

DOWN alert: acmemain (http://acme.sample.com) is DOWN

and whenever the site is reachable again, we would receive the following recovery email:

UP alert: acmemain (http://acme.sample.com) is UP

In this case, we would like to create alerts, whenever a down message arrives, and we would like to close the open alert in OpsGenie, if the recovery message is received. The setup would look like this:

Create Alert rule:

Close Alert rule:

In this particular case, setting up the filters was really straightforward. However, let’s take a look on the string processing method we used in the alias field to parse out a unique identifier from the subject field:

{{ subject.substringBetween(“alert: “,” (”) }}

This will parse out the string, which is between “alert: " and the opening bracket " (”. In our example message, that would be “acmemain”, which was the name of the check.


Please note: if this string is not located at the exact same position in the subject of the recovery email, you would need to use a different string processing method in your Close Alert rule.