Sometimes the smallest change in a daily routine can have profound effects on a business. Consider the mystery of the weekly, Thursday afternoon, system outage.<\/p>\n
Situation<\/strong> \u2013 The IT systems at a stock brokerage house experienced slow transaction times followed by a complete outage of their transaction\u00a0system at 3:20pm on a Thursday afternoon. Rebooting the transaction system resolved the issue and got everyone back to work \u2026 until the same thing happened on the following Thursday afternoon.<\/p>\n Identical symptoms occurred over the following weeks. The issue was always resolved with a system restart but frustration was mounting, particularly on the trading floor, where lost time can result in lost profits. When this weekly event came to the attention of the board, they instructed the director of IT to make this issue a priority. He formed a problem-solving team to find the cause and they used the Kepner-Tregoe RCA methodology to guide their efforts.<\/p>\n Identify the Problem<\/strong> \u2013 Recognizing that to solve a problem you first have to clearly state what the problem is, the team began by separating and clarifying the problem from the generalized \u201ctransaction system is slow\u201d to the more specific \u201ctransactions are timing out\u201d. The team used this problem statement to focus their search for the information needed to find true cause rather than wasting time examining interesting but irrelevant information.<\/p>\n Describe the problem <\/strong>\u2013 A clear problem statement is necessary but not sufficient to rule out false causes and suggest probable causes. So, the team began gathering information about what, when, where, and the extent the problem was\u2014and was not\u2014observed.<\/p>\n By taking the time to first describe the problem, the team was able to quickly find the proximate cause and then find the systemic cause<\/span> Examining work rosters did not give them any viable leads, but talking to team leaders eventually identified a possible link. There was one staff member in the invoicing team who left early every Thursday afternoon to take her daughter to ballet class. Members of the problem-solving team interviewed her to find out how she interacted with the system. They discovered that just as she was leaving every day, she began running a report that she needed to use the following morning. Normally this would be run at 5:30 pm, as this was when she typically left work. At this time of the day the stock exchange was closed and few other people used the system. On Thursdays, as usual, she set the report to run when she left, but she left around 3:15pm. The one Thursday the issue did not occur coincided with a day her daughter was on a school trip and did not go to ballet.<\/p>\n By taking the time to first describe the problem, the team was able to quickly find the proximate cause and then find the systemic cause: the report was run without parameters so it searched the entire transaction database and the report had a higher priority than all other transactions\u2014not a problem when the stock market was closed. But at 3:15pm, this caused the already busy system to run extremely slowly and ultimately timeout, consequently dropping the connection to the stock exchange.<\/p>\n Remove the root cause<\/strong> \u2013 The quick fix was to instruct the staff member not to run the report when the stock market was open. She showed someone else how to run the report for her at the end of the day on Thursdays and this removed the proximate cause of the issue and prevented it from happening in the future. The team went on to address the systemic cause: running a report without parameters and consuming more transaction system capacity than necessary.<\/p>\n To remove the systemic cause, a development team made changes to the system to ensure reports required specific parameters and any report that could potentially impact system performance could not be run during stock exchange trading hours. Now stock trades sail through the IT system, even on Thursdays, while across town, a class of small girls in pink tights learn ballet. Mystery solved.<\/p>\n","protected":false},"excerpt":{"rendered":" Sometimes the smallest change in a daily routine can have profound effects on a business. Consider the mystery of the weekly, Thursday afternoon, system outage. Situation \u2013 The IT systems at a stock brokerage house experienced slow transaction times followed by a complete outage of their transaction\u00a0system at 3:20pm on a Thursday afternoon. Rebooting the […]<\/p>\n","protected":false},"author":5,"featured_media":988,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[101],"class_list":["post-25000","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-other"],"acf":[],"yoast_head":"\n\n
\nIdentify possible causes <\/strong>\u2013 with a robust problem description, the team was able to avoid the trap of considering all changes that could possibly affect the system; the cause they were looking for affected the entire system, but only on Thursdays between 3pm and 3:30pm. The predictable timing of this deviation during normal working hours suggested cause was likely due to some human interaction with the system. This became their focus.<\/p>\n