It’s bound to happen: How to handle alert storms

No matter how careful you are. No matter how many times you’ve checked your parameters and state conditions. No matter how many times you tested it out of production.

It happens.

Some misconfigured rule - or even an event that happens much differently in a production environment then in test - begins firing off alerts. Maybe you don’t notice it right away - perhaps you haven’t setup notifications for this particular rule...