Incident On-call Best Practices: Surviving on-call duty requires mastering best practices and essential tools. First, effective Incident Management escalation ensures critical issues are quickly addressed by the right personnel. Then, structured Outage Management minimizes downtime and swiftly restores services. Additionally, efficient Major Incident Management coordinates responses to high-impact incidents. Finally, smooth Change Management Processes prevent disruptions when implementing system changes. By utilizing these practices and tools, you maintain stability, quickly resolve issues, and ensure seamless service continuity.

Incident management from IT Service Desk: Service Management by Fancy Mills-Knebel and HDI

High-Level Incident On-call Best Practices for an Extended Weekend Shift-work

Shift Handover

1. Receive Handover: Onboarding team reviews all tickets, and thoroughly review handover notes and current incident status from the previous shift.
2. Verify Alerts: Check for any pending alerts, critical issues, or incidents requiring immediate attention.
3. Update Contact Information: Ensure all contact information for stakeholders and team members is current.

Incident Monitoring

1. Continuous Monitoring: Actively use monitoring tools and dashboards to track system health and performance.
2. Automated Alerts: Regularly check automated alerts for anomalies or potential incidents.
3. Proactive Checks: Perform proactive system checks to identify potential issues before they escalate.

Incident Detection and Triage

1. Acknowledge Alerts: Promptly acknowledge and categorize alerts based on severity (P1, P2, etc.).
2. Initial Assessment: Conduct an initial assessment to determine the scope and impact of the incident.
3. Prioritization: Prioritize incidents based on their impact on business operations and customer experience.

Incident Response

1. Notification: Immediately notify relevant stakeholders and teams about the incident.
2. Incident Logging: Accurately log the incident details in the incident management system.
3. Resource Assignment: Assign appropriate resources and team members to handle the incident.
4. Coordinate Response: Actively coordinate with technical teams to investigate and resolve the incident.

Communication Management

1. Stakeholder Updates: Provide regular updates to stakeholders, including status updates and estimated time of resolution.
2. Customer Communication: Communicate with customers about the incident and expected resolution time, if necessary.
3. Internal Communication: Maintain clear and consistent communication within the team to ensure everyone understands their roles and responsibilities.

Incident Resolution

Implement Fixes:
Collaborate with the technical team to implement fixes and resolve the incident.

Testing:
Ensure the implemented fixes are tested and verified.

Update Status:
Update the incident status to ‘Closed’ in the incident management system once resolved.

Post-Incident Review

Documentation:
Thoroughly document the incident, including root cause, resolution steps, and lessons learned.

Post-Mortem Meeting:
Conduct a post-mortem meeting with relevant stakeholders to review the incident.

Process Improvement:
Identify process improvements or preventive measures to avoid similar incidents in the future.

Handover to Next Shift

Prepare Handover Notes:
Document any ongoing incidents, pending actions, and important updates for the next shift.

Brief Incoming Incident Manager:
Brief the incoming incident manager on the current status and any critical issues.

Ensure Smooth Transition:
Ensure a smooth transition to maintain continuity in incident management.

Additional Best Practices Indicators for Incident On-call Best Practices

Compliance and Reporting:

• Ensure compliance with incident management policies and procedures. Generate and review incident reports.

Training and Mentoring:

• Provide guidance and support to team members, and conduct training sessions if necessary.

Escalation Management:

• Handle escalations promptly and effectively, involving senior management if required.

On Call Schedule:

Here is how an On-call schedule is set up in ServiceNow:

Shift Handover is key to Incident On-call Best Practices

• Review Handover Notes: Thoroughly review handover notes and the current incident status from the previous shift.
• Verify Alerts: Check for any pending alerts, critical issues, or incidents that need immediate attention.
• Update Contact Information: Confirm that all contact information for stakeholders and team members is up-to-date.

Incident Monitoring

• Engage in Continuous Monitoring: Use monitoring tools and dashboards to actively monitor system health and performance.
• Regularly Check Automated Alerts: Consistently check automated alerts for any anomalies or potential incidents.
• Perform Proactive Checks: Conduct proactive system checks to identify potential issues before they escalate.

Incident Detection and Triage

• Acknowledge and Categorize Alerts: Promptly acknowledge and categorize alerts based on severity (P1, P2, etc.).
• Conduct Initial Assessment: Quickly assess to understand the scope and impact of the incident.
• Prioritize Incidents: Prioritize incidents based on their impact on business operations and customer experience.

Incident Response

• Notify Relevant Stakeholders: Immediately notify relevant stakeholders and teams about the incident.
• Log Incident Details: Accurately log the incident details in the incident management system.
• Assign Appropriate Resources: Efficiently assign the necessary resources and team members to handle the incident.
• Coordinate Response: Actively coordinate with technical teams to investigate and resolve the incident.

Communication Management for Incident On-call Best Practices

• Provide Stakeholder Updates: Regularly update stakeholders, including status updates, progress made, next steps, and estimated time of resolution.
• Communicate with Customers: When necessary, communicate with customers about the incident and expected resolution time.
• Maintain Internal Communication: Ensure clear and consistent communication within the team so everyone is aware of their roles and responsibilities.

Incident Resolution

• Work on Implementing Fixes: Collaborate with the technical team to implement fixes and resolve the incident.
• Ensure Proper Testing: Ensure that the implemented fixes are tested and verified.
• Update Incident Status: Once resolved, update the incident status to ‘Closed’ in the incident management system.

Post-Incident Review

• Document the Incident: Thoroughly document the incident, including root cause, resolution steps, and lessons learned.
• Conduct Post-Mortem Meeting: Hold a post-mortem meeting with relevant stakeholders to review the incident.
• Identify Process Improvements: Pinpoint any process improvements or preventive measures to avoid similar incidents in the future.

Handover to Next Shift

• Prepare Handover Notes: Document any ongoing incidents, pending actions, and important updates for the next shift.
• Brief Incoming Incident Manager: Brief the incoming incident manager on the current status and any critical issues.
• Ensure Smooth Transition: Ensure a smooth transition to maintain continuity in incident management.

Tools:

What is Everbridge?

Everbridge is a critical event management tool. It helps organizations manage and respond quickly to incidents. It provides robust communication and coordination tools.

How Everbridge Enhances Incident On-call Best Practices

Everbridge streamlines on-call incident management, ensuring quick and efficient responses. Here’s how:

1. Incident Notification

• Automated Alerts: Everbridge sends automatic alerts to on-call teams during incidents, triggered by monitoring tools or manual input.
• Multi-Channel Notifications: Alerts are sent via SMS, email, phone calls, and mobile app notifications to ensure prompt communication.

2. Escalation Management

• Tiered Escalation: If the primary on-call person doesn’t respond, the alert escalates to the next person.
• Custom Escalation Paths: Define custom paths based on incident severity and team roles.

3. Response Coordination

• Real-Time Collaboration: Teams collaborate in real-time, sharing updates and coordinating responses.
• Conference Bridging: Automatically set up conference calls or virtual bridges for immediate communication.

4. Incident Management

• Incident Logging: Everbridge logs all incidents, notifications, and responses for post-incident analysis.
• Task Assignment: Incident managers assign tasks within the platform, ensuring accountability.

5. Reporting and Analytics

• Incident Reports: Generates detailed incident reports for analysis.
• Performance Metrics: Provides metrics on response times, notification success rates, and more.

Download and Install Everbridge for Incident On-call Best Practices

Please see the video below to learn about installing and registering for alerts in the Everbridge Mobile App (EMA).

Other Incident On-call Best Practices Resources

• After-Action Review Library – National Policing Institute
• CrowdStrike Outage: Global Chaos
• Guided After-Action Report
• Harnessing AI for Incident Management in ITSM (thinkhdi.com)
• HDI: Incident management | LinkedIn Learning
• How to plan for major incidents in ITSM | Axelos
• itSMF How AI is Evolving Critical Situation Handling
• IT Help Desk for Beginners (linkedin.com)
• IT Service Management Forum Best Practices
• ITSM Capability Model- Level 1
• Major Work Behind the Major Incident Process | IT Community (stanford.edu)
• One-IT: Effective Ticket Handling – Dawn C Simmons
• Predictive Intelligent Situational Awareness
• Proper Ticket Handling Imperative
• Productivity: Service Operations Workspace
• Lessons Learned: CrowdStrike Incident
• Security Incident Response Introduction
• Virtual-Agent Chatbot