Alerting multiple teams independently by replicating alerts


#1

We are frequently asked whether OpsGenie supports an incident response workflow where at least one member of multiple teams gets notified when there is a critical incident.

OpsGenie supports notifying multiple teams/users simultaneously. By default, OpsGenie continues to send notifications and escalate as necessary until an alert is acknowledged. The default workflow does not support the use case explained above since as soon as an alert is acknowledged by a user, OpsGenie stops the escalation process and stops sending notifications. Continue reading if you are looking for an answer to this: Can we notify each team in parallel but independently, so that when a team member acknowledges an alert, only the escalation and notifications for that team would end, but the escalations/notifications for the other teams continue?

The solution we developed leverages OpsGenie AWS Lambda integration to execute Python scripts when an alert is created/acknowledged/closed etc. When an alert for a critical incident is created, a Python script is triggered through the Lambda integration. This script creates a new alert (aka sub-alert) for each team listed in the original alert. Similarly, when a sub-alert is acknowledged by someone, a script is triggered to update the original alert to indicate that the corresponding team has been notified.

Please find the scripts and the detailed explanations in our GitHub repository. Your questions and comments are welcome!