Silent newrelic alerts for a few hours


#1

Hi

Today, during the night, we had someone playing with the site and generating errors some errors. As there are very low traffic during the night, newrelic triggered alerts for apdex and error rate to opsgenie that of course, notified us.

The problem was harmless (scriptkid) and we would like to ignore it until today morning, but ack or snooze the issue was useless, newrelic closed it after a few minutes, just to recreate it another few minutes later and invalidating any ack/snooze created… this during all night until he gave up or traffic increased enough for that alerts not be triggered

So the question is: how can we silent a type of alert for a few hours, even if the original alert is closed and then reopen as a new alert? Some “meta-ack/snooze” that match all alerts that match?

On our main setup, we use alertmanager and we can create silent based on the fields and regex, opsgenie have anything similar?

I think i could do this via the notification policy, but that is too low level, too permanent and i do not think that everyone have access to that feature. It also requires a browser, not something we can do in the smartphone app to quickly go back to sleep

If there is no feature for this, please consider adding a meta-snooze (regex or similar) so we can do this


#2

You may like to take a look at the maintance policy page that may help in this area. https://docs.opsgenie.com/docs/maintenance.

Thanks


#3

but that is basically creating a “Notification Policy”, it just make it more temporary.
Again, i do not know if everyone have even access to that and require a full browser… a “power snooze” is what i would like opsgenie to have. Without something like this, Opsgenie makes hard to silent flapping services


#4

If you can filter for these alerts you can make them a lower priority and suppress notifications until business hours. The alerts are still generated, but they won’t page you. That’s how I handle items like these.


#5

@rcouts That looks as a good solution, i already have low priority events being delayed until next day during the night … but how can i lower the priority? are you talking in the notification priority?


#6

Hi @danielmotaleite - I think the notification policy that @rcouts mentioned would work for this situation if I’m understanding correctly. With a notification policy, you can filter by alert content. So if there are only specific New Relic alerts you don’t want to receive notifications for during this time, you can filter by the alert content and choose to suppress notifications for these alerts. You can also restrict the policy to specified time intervals, so it will only apply to alerts that come in at night. Please let me know if this works for you, or if you have anymore questions.


#7

You can create the alert with a lowered priority in the advanced tab of the integration in question based on criteria of your choosing. By default Newrelic alerts inherit the priority from Newrelic, so you can configure it on that side as well.


#8

@Samir that would work and in the end it was what i have done… but notice that i wanted to do this only temporarily , during that night only… it is really a snooze, but that also apply to not yet created alerts.

Sadly, as far as i understand, the only real way to do this is using the notification policy… so this requires a full browser to change this, not forget to remove this next day… and i’m not sure if all users can even edit the notification policy, so if this happen in their call period, they will wake up every 10min

@rcouts the newrelic integration would work too if i wanted to permanently silent this… but i do not wanted that… between the integration or the notification policy, the policy is way easier to work with! :slight_smile:
Changing in the newrelic side, i failed to find a way to change the alert priority in their side

So where can i create a “feature request” to add a “snooze events like this” to the web interface and app :slight_smile: