Alert Rules: Use-Cases and Examples
Index
- Changing Alert Summary
- Suppressing Incidents
- Changing Alert Type
- Changing Alert Entity ID
- Changing Alert Title
- Route to Different Escalation Policy/User
- Add a Specific Responder
- Assign Priority
- Adding Notes, Assigning Roles and Incident Urgency
- Assigning Incident Tags and SLAs
- Adding Tasks to Incident
- Using Alert Groups
- Creating new incidents for all alerts
For the purpose of the following examples, we'll reference a sample incoming alert, like the one below:
{
"message": "This is a sample incoming incident title",
"summary": "This is a sample incoming incident summary describing the issue",
"alert_type": "critical",
"entity_id": "42",
"payload":{
"status": "ACME Payments are failing"
"team": ["devops", "payment"]
"project": "kubeprod"
"severity": "1"
"release": {
"version": "2.3.1"
}
"module":{
"name": "payments"
}
}
Example 1: Modifying Incident Summary
You can modify the incident titles and summaries by using the alert data(message, summary, entity_id, payload) within placeholders like
The incident message is {{ message }}, or The incident summary is {{summary}} or Incident Created with Status - {{payload.status}} in release {{payload.release.version}}.
Let's say our payload also contains an additional field specifying the URL that caused the issue - "url": "www.acme.com/checkout/payments/issue-url/"
If we want to attach this field to our summary for additional context, we can modify the alert summary by using a rule with Payload Key Search as shown below:
Example 2: Modifying Incident Details - Changing Alert Summary
Example 3: Suppressing Incidents Outside Business Hours
The above issue can also be solved via maintenance windows.
Example 4: Suppressing Recurring Incidents
This Alert Rule would suppress similar recurring incidents that your team has already resolved in the last X seconds. A similar incident according to Zenduty is a previous incident created with the same entity_id as the new incoming alert. This can be applied in cases where the issue has been resolved in Zenduty but the alert source is still sending alerts shortly after a similar incident was resolved.
Example 5: Changing Alert Type for Non-Prod Alerts to Warnings
Example 6: Auto-Resolve Incidents
The above alert rule would automatically resolve an incident when receiving an alert with Closed mentioned in the title.
Example 7: Prevent Certain Incidents from Auto-Resolving
If there's an Incident with security mentioned in the title, the above rule will attach the Resolved Alert to the Incident as an Info Alert, prevent auto-resolution and attach a note letting responders know that there's manual investigation required.
Example 8: Sync Alert Entity ID with Integration Source
In the below example, if we have a key metric.name available in the payload JSON path, Zenduty will use its value as the entity_id; overriding the default entity_id.
Every alert coming from an integration source has an entity_id.
An entity_id points to a resource or an entity in the source application, for eg: the entity_id of an alert from Jira will be its Ticket ID.
There are some tools integrating with Zenduty that allow you to use a custom payload for notifications. In such cases, it is possible to sync the Zenduty Alert Entity ID with a user-specified value being sent by the integration, enabling easier association and collation if required.
Example 9: Changing Alert Entity ID of Incoming Alerts
The above rule would collate all incidents with a certain matching phrase into a single incident by changing their entity_id to a preset value. This may be useful to collect similar incidents with a single course-of-action fix or repetitive warnings together.
Hashed Entity ID
You can also hash your Entity ID by using the Hash Entity ID action. This allows integrations to accept Entity IDs longer than 32 characters, which are hashed and replace the original entity ID. The original entity ID is still viewable in the alert payload.
Example 10: Create New Incidents for Similar Alerts
The above rule will create a new incident if receiving a similar alert after 30 minutes since the first incident was created and add Recurring to the title.
Leaving the Change Alert Entity ID field blank assigns the incident a random entity_id, allowing you to override collation in such cases and create new Zenduty Incidents for recurring alerts. This particular rule is not recommended generally as it leads to an incremental amount of noisy alerts, however may solve some issues depending on the tooling.
Example 11: Change Alert Message to Add Severity
Example 12: Route to Different Escalation Policy
Example 13: Assign an incident to a specific User for specific Incidents
Example 14: Adding a Responder based on a value inside your payload:
In the above example, we're matching the team associated with the event and adding an additional responder in case the incident is regarding the DevOps team.
We can specify the indice that we'd like to be parsing whenever required.
So, if we want to know the secondary team associated with our event, we can change the above rule by replacing $.team[0] with $.team[1].
Example 15: Assigning Priority:
Example 16: Adding Notes, Assigning Roles and Incident Urgency
The Add Note action is only applied for the first alert that triggered the incident. For any alert that does not directly trigger an incident, the Add Note action will not be applied.
Example 17: Assigning Incident Tags and SLAs
Example 18: Add Tasks to Incident
Example 19: Using Alert Rule Groups to Change Urgency
You can use alert rule groups to build more intricate logical rules for niche use cases like the one above, in which the incident urgency will be set to low, if the project is kubenonprod OR if the project is kubeprod AND module.name is logging.
Example 20: Creating new incidents for all alerts(disabling default deduplication)
If you want to disable the default deduplication/grouping logic for an integration and want to trigger new incidents for all incoming alerts, then use the above alert rule. here we are catching all alert types in the condition, and using the action "Change entity Id" and keeping the value as blank/empty. This action will create a random and unique entity id for all alerts, which will in turn tell Zenduty to trigger new incidents for all incoming alerts.
You can also leverage RBAC with Alert Rules to route incidents to other teams or add global users as responders. Learn More
If you're still wondering whether Alert Rules could solve a particular use-case of yours, feel free to reach out to our team at contact@zenduty.com.