Skip to main content

Getting Started

Best Practice Guide

This guide offers best practices for setting up and using Zenduty to improve incident response workflows. You'll find tips for configuring alert management, escalations, on-call scheduling, and integrations. Following these guidelines helps enhance team coordination and response times, making the most of Zenduty’s capabilities.

Define Clear On-call Schedules and Escalation Policies

  • Organize your teams based on functional areas (e.g., customer support, engineering, operations). Each team should have designated members who are on-call during specific time slots. As much as possible, keep your shifts simple and predictable.
  • Ensure that if the primary on-call person doesn’t respond, the alert escalates to a secondary team member. Create multiple escalation levels to reduce the risk of an incident going unacknowledged.
    • Example: First, notify the on-call engineer; if no response within 5 minutes, escalate to a senior engineer, and then to management if necessary.

2. Integrate Monitoring Tools

  • Integrate all your monitoring tools like Datadog, New Relic, Prometheus, AWS CloudWatch, etc. These integrations allow the automatic creation of incidents in Zenduty based on predefined thresholds, ensuring that critical issues are flagged immediately.
  • Not all alerts require an incident - only those with a measurable customer or business impact and need to wake up your team. Configure your integrations to only alert Zenduty for high-severity incidents, reducing noise and preventing alert fatigue for your teams. Using SLA and priorities always helps in setting the right metric and alert thresholds.

3. Integrate Communication Tools

  • Integrate your incident management system with communication tools (Slack, MS Teams, and Google Chat) for real-time collaboration. For example, Zenduty can push incident alerts directly to a specific Slack channel where teams can coordinate responses.
  • Create dedicated war rooms in these communication platforms when a high-severity incident occurs. Zenduty can automatically launch these channels with the right teams involved.

3. Customise Notification Preferences

  • Zenduty allows notifications across multiple channels such as SMS, email, phone calls, and Slack. Ensure each team member configures their preferred notification methods for redundancy.
  • For high-severity incidents, ensure that notifications are sent via phone or SMS, while lower-priority issues can be handled through push notifications, email, or Slack.

4. Create Incident Response Playbooks

  • Document step-by-step response plans for common incidents (e.g., server outages, API failures, security breaches). Having these pre-defined guides can significantly reduce response times.
  • Set up Zenduty to automatically share relevant playbooks with the on-call team when a certain type of incident is detected. This eliminates the need to search for procedures during critical moments.

5. Implement Post-Incident Reviews with Postmortems

  • After resolving an incident, conduct a postmortem to analyze the root cause, response times, and team performance. Use Zenduty’s incident timeline to track events and response actions for a more detailed analysis.
  • Document insights and lessons learned from the incident, and share them with the wider team. This process helps improve future response times and prevents recurring issues.

6. Use Reporting and Analytics

  • Monitor key performance indicators (KPIs) like mean time to acknowledge (MTTA) and mean time to resolve (MTTR). Review these metrics regularly to identify bottlenecks in your incident response process.
  • Use Zenduty’s built-in analytics to identify recurring patterns and proactive measures that can prevent future incidents.

7. Enable Two-way Integrations with Support Systems

  • Connect Zenduty with ticketing systems like Freshdesk, Jira, or ServiceNow for seamless communication between customer support and engineering teams. This allows for immediate incident/ticket creation when support identifies an issue, and automatic updates when the incident is resolved.
  • Ensure that Zenduty syncs information back to these tools using bi-directional integrations so support teams have real-time visibility into the statuses of incidents, reducing communication gaps.

8. Use Incident Templates

  • Create alert rules to set predefined templates for common incident types (e.g., server crashes, network issues, database errors). This ensures consistency in the information logged and streamlines the incident creation process.
  • Use tags to categorize incidents by type, severity, and affected services. This helps in filtering incidents later for review, analysis, or postmortems.

9. Test and Validate Regularly

  • Schedule regular incident response drills to ensure that on-call teams are familiar with the tools, processes, and escalation procedures. Drills help uncover gaps in your workflows and ensure readiness.
  • Simulate an incident to confirm that escalation policies are working properly. Ensure that notifications are being received and escalated promptly.

10. Ensure 24/7 Global Coverage

  • If your organization operates across different time zones, configure Zenduty to provide 24/7 support by setting up on-call schedules based on team members' geographical locations. This prevents downtime during off-hours and ensures faster incident resolution.