{"id":25654,"date":"2019-12-16T13:49:00","date_gmt":"2019-12-16T13:49:00","guid":{"rendered":"https:\/\/kepner-tregoe.com\/putting-together-a-top-notch-it-incident-management-team\/"},"modified":"2026-06-05T14:59:22","modified_gmt":"2026-06-05T14:59:22","slug":"putting-together-a-top-notch-it-incident-management-team","status":"publish","type":"post","link":"https:\/\/kepner-tregoe.com\/nl\/blogs\/putting-together-a-top-notch-it-incident-management-team\/","title":{"rendered":"Putting Together A Top-Notch IT Incident Management Team"},"content":{"rendered":"

A human error\u2014a very basic one\u2014caused British Airways to suffer an IT outage on May 27, 2017<\/a>, forcing it to cancel more than 400 flights and leaving 75,000 passengers stranded. An engineer had disconnected a power supply at a data center, and when it was plugged back in, a power surge caused major damage. Net cost to the airline: a whopping 80 million pounds<\/a> (about $102 million).<\/p>\n

This might sound like a lot of money\u2014and it is\u2014but according to Statista<\/a>, it\u2019s not unusual. The average cost per hour of downtime for 86% of enterprises is more than $300,000. And the hours add up quickly.<\/p>\n

<\/p>\n

The\u00a02019 IT Outage Impact Study<\/a>\u00a0found that the typical organization experienced 10 brownouts (where infrastructure or software performs at a degraded level) or outright outages over the past three years. \u00a0Those 10 incidents easily add up to millions of dollars.<\/p>\n

Not surprising, then, 80% of companies reported that the performance and availability of their IT infrastructure tops their list of concerns. More than half worry about experiencing an outage so devastating that it will make the mainstream news. And if some such event occurs, 53% expect heads will roll\u2014and that someone will lose his or her job.<\/p>\n

And as much as it would be nice to simply automate responses to IT issues, \u201cIncident response needs people, because successful incident response requires thinking,\u201d\u00a0wrote Bruce Schneier, in his blog, Schneier on Security<\/a>, back in 2014. What you need: an IT (major) incident management team with clearly defined roles and responsibilities, trained to fulfill those responsibilities by following a crisis-proven process while effectively communicating with managers, customers and subject-matter experts alike.<\/p>\n

The human side to outages<\/h4>\n

Herein lies the problem: staff and skills shortages are a significant challenge to effectively responding to incidents. Indeed, the Uptime Institute\u2019s 2019 study<\/a> is now calling the IT staffing problem a crisis. Sixty-one percent (61%) of respondents said they had difficulty retaining or recruiting staff \u2014 up from 55% the previous year.<\/p>\n

This matters because 60% of organizations believe that their most recent significant downtime event was preventable. If they had better management, processes, or configurations, the outage could have been avoided, they say. For outages that cost more than $1 million, this figure leapt to 74%.<\/p>\n

\u201cBy under-investing in training, failing to enforce policies, allowing procedures to grow outdated, and underestimating the importance of qualified staff, management sets the stage for a cascade of circumstances that leads to downtime,\u201d wrote Kevin Heslin, chief editor of the Uptime Institute Journal in a September 2019\u00a0blog post<\/a>about the survey.<\/p>\n

Staffing the IT incident management team<\/h4>\n

An incident is any unexpected event that disrupts normal operation of an IT service. \u00a0IT incident management is an area of IT service management (ITSM<\/a>) where the service is returned to normal ASAP. Many IT incident management teams use established ITSM frameworks such as IT infrastructure library (ITIL<\/a>\u00ae) or\u00a0COBIT<\/a>. Others use a combination of proprietary best practices established over time.<\/p>\n

Here are\u00a0some\u00a0of the most\u00a0common\u00a0IT incident management roles to hire and train for.<\/p>\n

(Major) Incident managers<\/h4>\n

These people need to be \u201cin control\u201d. When something goes wrong, they provide immediate structure, leadership and are ultimately responsible for bringing services back to normal.<\/p>\n