Supress alerts in period(s)


#1

We have one Splunk integration, and a lot’s of searches in Splunk that will generate alerts in OpsGenie.
One of the searches should not generate notifications during the night (2AM - 4AM) because of restarting of the system.

Is i possible to ignore only these notifications in this period? Only found a way to stop the complete integrations, supress all the notifications. Have also created a “autoclose policy” - that will close alerts every minutes, but this will create a lot of alerts.

Not sure if I’ve been able to describe the issue in an understandable way, but hopefully some of you understand:)


#2

You can achieve this easily as the following:

  1. Create a Notification policy which suppresses the notifications for the matching alert. Make sure that the conditions of the policy match only the alerts from your Splunk integration (with a condition like source equals Splunk) . For further information about the notification policies, you can refer here
  2. If you restart your system periodically (like every night 2AM - 4AM or every Monday night 2AM - 4AM), restrict the policy that you created on step 1 to this hours and enable the policy.
  3. If your restart times are not periodic, keep the policy disabled and configure a Maintenance that enables your Splunk integration. You can schedule maintenance periods by this way to activate your policy that suppresses Splunk notifications.

Best regards,
Kadir Türker Gülsoy
Software Engineer & Team Leader at OpsGenie


#3

Suppressing the notification is fine, but I’ll then need to be able to close the alert. Since I’m using aliases for these alerts - suppressing in a period - won’t unsuppress an existing alert.
I can’t disable the complete Splunk integration, since there are other alerts that are valid and need to be followed up.

Let me try to show an example

02:00: System restarts
02:00: Alert: System down - with alias - notification suppressed
02:10: Alert: System down - same alias --> Count increased
02:15: Alert; System down - same alias --> Count increased
02:20: Alert: System down - same alias --> Count increased
02:25: Alert: System down - same alias --> Count increased

04:00: Restart is finished. Alerts after this is “valid” and need to ble notified
04:05: Alert: System down - same alias --> Count increased

The question is then, how could I have the alert 04:05 to generate a notification? .


#4

Then there is an addition to the setup from my first answer :slight_smile:

  1. Create a Modify Policy with the same time interval & condition constraints. This policy will change the alias field of Splunk alerts. Let the updated alias be Splunk Alert To Be Suppressed
  2. Create a Notification policy which suppresses the notifications if an alert alias is equal to Splunk Alert To Be Suppressed . You can keep this policy enabled all the time, because this policy suppresses the notifications only if the modify policy from Step 1 works
  3. If you restart your system periodically (like every night 2AM - 4AM or every Monday night 2AM - 4AM), restrict the modify policy to this hours and always keep this policy enabled.
  4. If your restart times are not periodic, keep the modify policy disabled and configure a Maintenance that enables your Modify Policy. You can schedule maintenance periods by this way to activate this modify policy

In this case:
02:00 -> System restarts
02:00 -> Alert: System down. Alias = Splunk Alert To Be Suppressed. Alert is created and notifications are suppressed
02:15 -> Alert: System down. Alias = Splunk Alert To Be Suppressed. Count increased.

04:00 -> Restart is finished.
04:05 -> Alert: System down. Alias = Different than the previous one. Alert is created, and this time notifications are sent.

You can even empower this solution with an Auto-Close policy that closes all alerts with alias Splunk Alert To Be Suppressed automatically after a time period of your choice (if you don’t want to close them manually)

Best regards