And saves millions
Lonnie Wilson, author of the Lean classic, How to Implement Lean Manufacturing, offers a case study of real-life, problem solving that saved millions of dollars, and then a deeper dive into troubleshooting methods. While the case study is specific to a Lean, refining situation, both the case study and overview of root cause analysis, offer insights that are relevant across technologies, processes and industries.
The Background
Recently I was working with a group of greenbelts who were part of their refinery’s Lean-Sigma (LS) program; I was their program advisor and mentor.
On one visit, I was approached by the Refinery Manager. He advised me they were headed for an unscheduled shutdown of the cat cracker because the VGO desulfurizer was prematurely reaching end of run (EOR) catalyst conditions. It seems the feed-effluent (F/E) exchangers were fouling at an unprecedented rate and a premature and very expensive shutdown was imminent. Knowing I was familiar with Kepner-Tregoe problem solving methods, he asked that we look at this problem and see if we could avoid the unscheduled shutdown he anticipated.
I spoke to the LS Program Manager and he arranged for the appropriate group to meet the next day. They were advised to block out the entire day. The next morning, 7:30 sharp, we started. They came with charts, tables and graphs galore; the volume of data they had gathered and compiled was truly impressive. The Program Manager and I came with a problem specification template. We went through introductions, I said a quick word about the method we were going to use, and we started.
The Technical Issue
People have told me numerous times that a problem well-defined is 50% solved. Rather I find that a problem well-defined is more likely 90% solved, and I have seen no process to define problems that even approaches the KT methodology.
I asked the lead process engineer to give me a recap of the history so I would be somewhat up to speed with the rest of the group. He said, “Formany years the fouling rate in the F/E exchangers was used as an early indicator for catalyst EOR conditions in this reactor. During most of the run, the reactor top bed pressure drop (DP) would show very little change. However, normally eight to ten months into the run, these heat exchangers would reach minimum efficiencies and we knew a shutdown was imminent. Shortly after reaching minimum efficiencies, the top bed pressure drop would increase dramatically; forcing the shutdown. Maximum top bed pressure drop was normally reaching in three to four months. When we shutdown we would skim the reactor top bed and do partial cleaning of the heat exchanger. This would extend the run to the next major shutdown. Due to a variety of issues, including safety, environmental and economic, the top bed skim option is no longer available. On the last shutdown we made several process changes including: improved water wash facilities; instrumentation and piping changes; and, because we were purchasing lower sulfur crudes, we loaded the reactor with a slightly less active and cheaper catalyst. These changes were designed to allow an 18-month uninterrupted run. However, now we are at seven months into the run and, with very low exchanger efficiencies, are facing an unscheduled shutdown within the next three to five months. We started this run in August and although we did not see it initially, due to our monitoring practices, heat exchanger efficiency dropped significantly. By November we were concerned and put this team together to revisit this issue. Now, five months later, heat exchanger efficiencies are at a minimum, but we have not seen the expected changes in reactor DP. We have invested over 100 man-days of engineering time and we are beginning to think we might not understand this problem”.
I asked that he explain the mechanism that causes the reactor DP to rise, causing the shutdown. He went on:
“The reactor feed is on the tube-side; reactor effluent is on the shell-side of the exchangers. The reaction is exothermic, so the reactor effluent is used to preheat the reactor feed. We get tube-side fouling reducing the exchanger efficiency. When the exchanger reaches a minimum efficiency, we believe the debris is then carried into the reactor rather than plating out on the exchangers. The DP on the top bed then rises causing the end of the run. We have seen this on the last four runs since the major modifications seven years ago.”
The Problem-Solving Session
Everyone then spent another 30 minutes scouring over the data. We then began to fill out the problem specification. At first it went slowly as everyone wanted to chase their favorite rabbit. Once we got the discipline to take it one step at a time, we moved quickly. Long before lunch we had specified the problem well enough that now everyone was convinced that, although they did not understand this mechanism, it surely was different from what they had previously seen. We started to break for lunch; but no one wanted to stop the brainstorming; we were on a roll. About an hour later we made a major breakthrough. We were answering one of the “dimensions” in the problem specification which addresses the “Extent” of the problem and, among other questions, it asks specifically “what is the trend?” At this time, I noticed something in the data and I asked the plant process engineer to make a simple plot of the change of heat transfer loss over time and plot it on the same graph as reactor outlet temperature over time. Eureka!!!
There are two major pitfalls I have seen that are a hindrance to good problem solving. Interestingly enough, one of those two pitfalls is not the difficulty in finding solutions. Rather the two pitfalls are poor problem definition and failing to use a disciplined, structured approach to solving the problem. – How to Implement Lean Manufacturing
We found something. Even though there was a lot of noise in the data, we could see a major inflection point in mid-February. Something had changed at this time. This then gave us a clear picture of a “distinction” and everyone knew we were on to something. We completed filling out the specification addressing all four dimensions of “what, where when and extent” and moved into “possible causes”. Now the logic was flowing faster than we could document it and we quickly focused on the “most likely cause”. It was now very clear that the reactor temperature drop was the key point. Soon enough we were able to connect the low temperature to lower than expected sulfur content in the feed. Since the reaction was exothermic, the lower sulfur feeds caused outlet temperatures to drop. We were able to confirm that this caused deposition of ammonium chloride and other salts. These salts were a known issue but had never been seen in these high concentrations nor had they been seen this far upstream in the reactor outlet. The lower outlet temperatures now allowed the salts to precipitate in the F/E exchangers rather than further downstream. We determined a temporary fix was to raise the inlet temperature until the outlet temperature was above salt sublimation temperatures. It was now mid-afternoon and, while several of us continued the documentation, two engineers went to the plant to test our hypothesis.
The Results
First, they increased inlet temperatures and within an hour they could tell the exchanger efficiencies had risen. We had confirmed the solution. Second, in a controlled experiment over the next days, the engineers lowered the temperature and raised it again to confirm we had developed a corrective action. Third, and finally, we raised the reactor outlet temperature substantially to “heat soak” the exchangers to see if we could improve the exchanger efficiencies; it worked and we got back some of the losses as some salts sublimed. After more analysis we reported that we had found the problem and with the cost of a little energy, we could run the plant to the next scheduled shutdown. The problem solving and decision making was a success. The next week the team reviewed the project, began working on the follow-up items and went through a formal Potential Problem Analysis. The application of the systematic problem solving technique to this problem was a huge success, saving a million-dollar emergency shutdown and untold millions in lost profit opportunities.
If you have read either of my books, How to Implement Lean Manufacturing or Lean Refining – How to Improve Performance in the Oil Industry, you will note a very important statement I make. This statement, surprisingly, IS NOT widely shared among the majority, but is clearly understood by experts and it becomes a painful reality to those who choose to ignore it.
There are two major pitfalls I have seen that are a hindrance to good problem solving. Interestingly enough, one of those two pitfalls is not the difficulty in finding solutions. Rather the two pitfalls are poor problem definition and failing to use a disciplined, structured approach to solving the problem.
The Kepner-Tregoe (KT) methodology is the very best I have ever seen to attack these two problems.
Problem Definition
First the problem statement is not just a statement, but it is further specified by what KT calls the four dimensions that “box in the problem.” Those four dimensions are:
- What is the problem?
- Where is the problem?
- When is the problem? And
- What is the extent of the problem?
As powerful as this is to get an understanding of the situation, there is yet another aspect of the specification. When you begin to specify each problem, asking “what, where, when and the extent” of the problem, the KT methodology includes the “is not” portion, and this is pure genius. For example, if you have a problem that has to do with defects of a certain nature, you likely have a small population of defects to use as you define these four dimensions. However, as soon as you include the “is not” analysis the size of your database just increased dramatically. For example, lets say you are having problems in a milling operation and there are five mills operating in parallel. You ask where are you making the defect? Then someone does some analysis and you find that you are only making them on Mill D. Then you ask the “is not” question which is, Where could we be making bad parts, but are not? And the answer is now Mill A, Mill B, Mill C and Mill E. Which leads to the next KT question which asks you to describe the differences and distinctions. Now all five mills become a source of problem solving data … and voila you have significantly improved your chances of finding the root cause. Although KT says they improve the problem specification with the four dimensions, once you consider the “is not” option for each dimension, I say they have 8 dimensions that lead to problem specification.
People have told me numerous times that a problem well-defined is 50% solved. Rather I find that a problem well-defined is more likely 90% solved, and I have seen no process to define problems that even approaches the KT methodology.
Structured Approach
Typical problem solving is too often “ready-fire and then aim.” Or said another way, one CEO, after seeing us solve a knotty problem that had bugged one of his refineries for years asked me, “Why is it we seem to have the time and resources to do it again and again, but not the time to do it right the first time?” I do not have simple answers to those issues but what I find is the answer lies somewhere enmeshed in the topics of focus, discipline and maturity.
This I know for sure. Your chances of making good decisions, finding true root causes and avoiding future problems lie squarely in the domain of people with intimate process knowledge using a structured approach. Again, the KT methodology is superior to any other method I have used. Starting with a clear problem specification and using logical analysis to find the root cause is only the beginning. Almost always, even if there is only one problem found, there are often many ways to solve it. This requires Decision Analysis, (DA) another KT staple. Finally, all decisions have risks and there are possible future problems, which the KT methodology addresses with Potential Problem Analysis (PPA). In the case just studied, if KT PPA had been used prior to this run, very likely the situation they faced would have never occurred. As we moved the team into PPA as part of the solution, we discussed this.
I asked the team, “Why did we not evaluate this on the changes that were made seven months earlier?” The answers were not very impressive and mostly they were somewhere between excited that we had solved this problem and embarrassed that it took seven months to realize they were trying to fix the wrong problem—especially since with the same team and the same data we wrapped this up in less than one day. However, now there was a new energy in that they had seen how the KT methodology could assist them in their day-to-day activities.
A few weeks later the LS program manager led them through another KT problem solving session. Again they solved it in one session; a seemingly unsolvable problem that had plagued them for months.
What about Lean Manufacturing?
Lean manufacturing is – The creation of a culture of continuous improvement and respect for people. And the “Means to Lean” is —
- Problem-solve your way to the Ideal State;
- Through the total elimination of waste;
- Using a fully-engaged workforce.
You will note it all starts with problem solving. Literally the vitality and the success of any lean transformation is dependent upon the ability to define and solve problems. While the KT technique is one of my favorites, I use other problem-solving techniques including the Six Questions of Continuous Improvement (see www.qc-ep.com), the statistical tools of Six Sigma, the Five Whys, the Shainin tools and others. However, when I have a really tough, a really large or a really consequential problem which are so common in an oil refinery, I count on the KT methodology to get me results … and it does; just as it did in this case.
Lean Refining and the Kepner-Tregoe Methodology
I have used the KT process since the early 70s when I was first an engineer for Chevron. Later as a manager I taught my supervisors how to use these methods and we would send managers to the KT training so we would have someone on staff to keep the process fresh and consistently used. When I wrote “Lean Refining – How to Improve Performance in the Oil Industry” (Industrial Press, 2017), which addressed the application of lean manufacturing strategies, tactics and skills in the oil industry, I determined that refineries needed to immediately create or augment six critical skill areas.
Six Initial Skill Focus Areas are:
- Leadership
- HK planning
- Leader standard work
- Problem solving
- Statistical techniques
- Meeting management and meeting facilitation
Later in the text, regarding problem solving I say:
“…the KT methodology, should be taught to a broad range of employees. All engineers, managers and those assigned specifically to problem solve should have KT training. In time, a lean support team will be staffed, and all these members must be KT proficient. In addition, you should have at least one KT-certified trainer for the facility.”
A Final Word About the KT Methodology
The KT process promotes yet another gain I have seen often; yet it is seldom discussed. After anyone works through the process, they are almost universally impressed with both the process and the results. Unfortunately, I have been in the control room on many occasions when we did not have the luxury of sitting down with a team of talented people to fill out the specification, develop possible causes and work through a problem. The problem was right there in our faces and we needed to act in the next few minutes or there would be serious problems. For those who practiced the KT process regularly, I found they were both faster and clearer thinkers in this type of a crisis. Following the methodology not only gives you better answers but it also trains you to think in a clear, logical and integrated process that assists you in any problem you might encounter anywhere.
In any Lean transformation, especially in an oil refinery, one key to success is the ability to solve problems. When it comes to addressing major issues of problem solving so you can find superior solutions, make better decisions and avoid future problems, I know of no system superior to the KT methodology. In a refinery, with a never-ending supply of large, complex and consequential problems, the KT skills should be widely used and supported. That will make a significant difference as you try to apply the lean strategies, tactics and skills and work to make your facility a world-class performer.
Lonnie Wilson is the author of How to Implement Lean Manufacturing (McGraw Hill 2009, 2015) which was recently translated into Chinese by Beijing University and McGraw Hill. He is also the author of Lean Refining: How to Improve Performance in the Oil Industry (Industrial Press, Inc. 2017). Founder of Quality Consultants in El Paso, TX, Lonnie was in refinery management at Chevron for 20 years. In addition to his expertise in evaluation, design of change and training in Lean Manufacturing, Lonnie is an expert problem solver, Six Sigma Master Blackbelt and trainer. In his not-so-spare time Lonnie is an avid reader and soccer fan. If you are looking for good books on lean or its related fields, check out “Lonnie’s library” on his website, www.qc-ep.com. He not only has an extensive library but each book is critiqued. Alternatively, if you wish to discuss those topics…or soccer…just give him a call.
About KT
Kepner-Tregoe has empowered thousands of companies to solve millions of problems. KT provides a datadriven, consistent, scalable approach to clients in Operations, Manufacturing, IT Service Management, Technical Support and Learning and Development. We empower you to solve problems. KT provides a unique combination of skills development and consulting services, designed specifically to reveal the root cause of problems and permanently address organizational challenges. Our approach to problem solving will deliver measurable results to any company looking to improve quality and effectiveness while reducing overall costs