{"id":25916,"date":"2020-05-11T13:51:00","date_gmt":"2020-05-11T13:51:00","guid":{"rendered":"https:\/\/kepner-tregoe.com\/the-essential-guide-to-incident-response\/"},"modified":"2026-05-28T14:46:51","modified_gmt":"2026-05-28T14:46:51","slug":"the-essential-guide-to-incident-response","status":"publish","type":"post","link":"https:\/\/kepner-tregoe.com\/nl\/blogs\/the-essential-guide-to-incident-response\/","title":{"rendered":"The Essential Guide to Incident Response"},"content":{"rendered":"

Sometimes it helps to go back to the basics. Such as reminding ourselves of the point of incident response (IR). The answer is simple: to keep the business running. But that simplicity is deceptive. This is an incredibly heavy responsibility, as you know if anything has ever gone wrong in your ability to respond to a major incident. According to Gartner<\/a>, every minute that your systems are down cost on average $5,600, adding up to more than $300,000 per hour. That\u2019s a lot of money, and a. lot of pressure.<\/p>\n

At KT, we put our collective heads together and came up with seven best practices for ensuring the success of your IR program. They include some operational, some technical, some organizational suggestions, but all of them contribute to building a first-class IR team.<\/p>\n

Why Incident Response?<\/h3>\n

ITIL describes an incident as any interruption or disturbance to normal IT services. We can make it more personal to your business, and say than an incident is any circumstance in which a system acts in a way that negatively impacts your customers. It doesn\u2019t have to be an outright system crash. Take a slow-performing email system. Does that constitute an incident? Using our definition, you bet it does, as slow emails mean slower response to customer service enquiries, delayed reactions to requests for proposals (RFPs), slowed product development, and just about every activity your business engages in for profit.<\/p>\n

IR is your process for responding to these incidents (and incidents are different from problems, which we will discuss later). Successful IR\u2014which means that it\u2019s both fast and effective\u2014results in improved worker and process efficiency, higher productivity, and, ultimately, higher revenues for the business. It really is a mission-critical operation.<\/p>\n

7 best practices for superb incident response<\/h3>\n

Here are eight best practices that will fine-tune your IR team to make it top-performing.<\/p>\n

1. Communicate, communicate, communicate<\/h4>\n

There has historically been a communication chasm between IT and the rest of the organization\u2014particularly between IT and users. This raises problems when attempting to deliver great IR, because many, if not most, of your incidents will be reported by your users. They must have an easy way to do this reporting, so you hear about incidents as soon as possible. Then you have to keep them informed in real-time as you resolve the incident. All this is necessary to gain their trust so they will work more closely with you\u2014a collaboration that is essential\u2014in future incidents. For starters, open up multiple channels to let users raise tickets easily. For example, they should be able to alert the IR team via email, chat, a portal, or an enterprise social network like Yammer. You should also create self-service mechanisms so users can solve the easy incidents.\u00a0 Make self-service easily accessible and educate users about the benefits of self-help and using the knowledge base to resolve issues on their own.<\/p>\n

Then, as the IR team works on fixing the incident, it\u2019s essential to keep everyone apprised of progress in real-time. There are two pieces of information that should be prominently displayed at all times: the incident status (current resolution state, including estimated time of completion), and the priority of the incident (how important it is to resolve the incident relative to other incidents.<\/p>\n

Automation can help, by sending automatic updates throughout the lifecycle of major incidents. Clear and visible notifications will also prevent users from raising duplicate tickets and overloading the help desk. Even if there\u2019s nothing to report, tell your stakeholders that, on an hourly or half-hourly basis. And have a dedicated line to respond to major incidents immediately and offer support to anyone affected.<\/p>\n

2. Adopt DevOps Processes<\/h4>\n

Before DevOps became mainstream, the IR team was basically in it for themselves. They, rather than the people who had actually built the systems, were responsible for all incidents. There was no feedback loop to the developers on how to fix repetitive interruptions to a particular application, for example. There was very little communication at all between the people who built the systems, and the ones responsible for fixing them when things went wrong. Indeed, one reason that DevOps was created was to eliminate these organizational silos. This is essential because of the complexity of today\u2019s systems\u2014they are all interconnected, and what affects one is likely to affect others.<\/p>\n

With a DevOps structure in place, developers do a better job in building their systems, because they now know they must also support them\u2014no more throwing problems over the wall for another group to worry about. IR teams have support, and, typically\u2014if DevOps is done right\u2014clear documentation of how to keep complex systems up and running.<\/p>\n

3. Sense when to \u201cswarm\u201d<\/h4>\n

Although most businesses have a \u201ctiered\u201d structure for dealing with incidents\u2014Tier 1 is the help desk, Tier 2 involves application specialists, and Tier 3 are generally the system uber-experts and developers\u2014you don\u2019t want to universally enforce this structure when solving major incidents. You want to give your team the freedom to \u201cswarm\u201d when necessary.<\/p>\n

This usually is necessary when an issue has a huge business impact. In such cases, you want to deviate from normal tiered IR processes. Swarming replaces that structure with a model of networked collaboration. \u00a0It originated at Cisco, which wrote about it in its 2008 white paper, \u201cDigital Swarming<\/a>.\u201d The concept was subsequently adopted by the Consortium for Service Innovation, and developed into a vision entitled \u201cIntelligent Swarming<\/a>..\u201d<\/p>\n

The general idea behind swarming is that instead of escalation, you bring everyone who might be able to help solve an incident into the IR team at the same time. There they brainstorm and bounce ideas off each other, and in general use the group dynamic to come up with fresh and innovative solutions to difficult IR issues.<\/p>\n

Core principles of swarming include:<\/p>\n