The shutdown had occurred with no warning. The first indication of a problem was when the plant shut down due to a trip signal from the main turbine. The unexplained trip created quite a challenge since there are about 20 different logic signals that can lead to this trip, some occurring in conjunction with others.
Steve Bono, the plant support services supervisor, and I had completed our training to become Kepner-Tregoe Program Leaders at a Problem Solving & Decision Making (PSDM) Learning Development Institute program a week before the trip. After returning to the plant, we made a presentation to management on the use of process. The next day, when the turbine tripped, we were asked to help determine the cause.
Fitzpatrick is an 850-megawatt boiling water reactor. While it has the capacity to power over 750,000 homes, it mainly serves business and government customers and is designed to run at 100% power, full-time. The impact of a shutdown c
an be $500,000 per day in lost revenues and replacement power costs, so there is a huge financial incentive to determine cause and return the plant to service quickly and safely.
Harnessing power from the atom probably sounds complicated, and it is. The plant has over 65,000 components. However, the uranium fuel is just another method of producing steam to spin a large turbine and generator. The turbine turns from steam flow, much like a child’s pinwheel in the wind. In this case, the pinwheel is a long series of fan blades attached to a shaft that is almost the length of a football field.
Fitzpatrick operates under a license from the Nuclear Regulatory Commission (NRC). Under this authority, major equipment problems must be solved for cause before the reactor can be restarted. On such occasions, a team is formed and operates under great time pressure to return the plant to service. Despite having an experienced staff, some guessing normally occurs when there is uncertainty about the cause of an event. Time and money can be wasted taking unnecessary actions.
Establishing a team and getting to work
The search for cause began by assembling a team that consisted of mechanical, electrical, and turbine engineers, an instrument and control supervisor, and a lead engineer for the turbine systems. Initially, the team was frustrated. They had not been trained in rational process, and challenged why I wanted to know what the problem IS NOT. I used the four marker illustration for IS/IS NO T that Tom Doyle had shown us in our LDI class. That helped focus the team.
Over the next four hours, we went through three passes of Problem Analysis, refining the problem statement to home in on which part of the trip logic was the most probable. Several times, members of our plant management team stopped by to see how we were progressing and when we would be done. I explained where we were in the process and what inputs we needed to determine cause. Typically, these visits create anxiety in teams formed to solve problems under pressure. But our managers understood our need for quality data. Even though we wanted to know cause quickly, we all agreed to stick to the process.
Identifying cause
The problem specification pointed to turbine protection logic, indicating a simultaneous closure of all turbine stop valves and control valves (eight valves total). The turbine stop valves consist of four, 24-inch diameter valves, designed to stop the main turbine from reaching overspeed conditions and destroying itself. Turbine control valves are similarly sized, designed to regulate steam flow evenly to the high-pressure and low-pressure stages of the main turbine.
The manufacturer’s main turbine experts and several people at our plant rejected simultaneous closure as being highly improbable. Some felt that the cause was due to the installation of a row of new commutator brushes on the main generator, completed about 20 minutes before the trip. Others blamed a lightning strike that had occurred 12 hours earlier. However, the specification tended to rule out these “pet causes.” The problem specification was further complicated by the turbine speed, which had initially increased, an indication that the valves were still open.
The team stuck with the problem specification. We pushed for the inspection of each cable in the indicated valve logic.
Due to the initial reluctance to accept the identified cause in the valve protection logic, other inspections were performed first. On Saturday, the cables were inspected. A short was found in the cable that houses the signal for the valve position switches. This gave an indication that the valves had closed, even though they remained open. The result: a faulty trip signal to the turbine.
A spare cable was connected in place of the shorted cable. During our next scheduled refueling outage, a more detailed evaluation will be performed on the shorted cable to extend the cause of the cable short.
Determining cause for other problems
Meanwhile, Steve and I had more work to do. We used PA to determine the cause of additional problems experienced during the trip. These included main condenser vacuum degradation and other equipment problems. I worked the day shift and Steve worked the night shift. Each of us would take the problem specification as far as we could until we needed more data. As soon as new data arrived, whoever was working continued to pursue the problem. At shift change, Steve and I would pass off the PA to each other. Typically, shift turnover is difficult because information must be communicated clearly and quickly at a time when people are tired. By using KT forms to exchange data between shifts, turnover was faster than usual and we were more confident that relevant information had been communicated effectively.
Eventually the cause of the condenser vacuum degradation was verified to be a valve out of assumed position. While this was an unpopular cause and a number of people strongly disagreed with the results of the PA, it was verified to be true cause and corrected during plant start up.
The Benefits of Process Are Realized
The actual costs of the shutdown aren’t easy to quantify because downtime is used to perform other work; but the benefits of the use of process to our organization were immediate. KT rational process helped to raise the confidence of the NRC in our ability to identify the true cause of identified problems and helped plant management and NRC staff to work together more effectively.
Process continues to support efforts to improve teamwork and reduce departmental “silos,” while giving everyone involved a real sense of accomplishment. Following the shutdown, people recognized the advantages of having broad representation on teams that resolve complex issues. Interest in our workshops is growing. It is increasingly commonplace to see individuals from several areas working together to solve problems and make decisions.
High visibility success builds widespread support
One engineer spoke up at the end of a session and said, “I’ve been someone who hasn’t had time for you guys. I want to let you know that you can count on me to help you any time.” This engineer is an important ally. He Fitzpatrick has enormous influence on others at the plant.
People who had earlier complained that it took too long to get to cause now thanked me for sticking to process and have participated in KT Problem Solving and Decsion Making training. Several managers have petitioned to get more people trained in PSDM. Senior managers from Design, Operations, and Maintenance are using process to address long-standing equipment problems and other challenges. The Operations senior manager has now attended KT PSDM training
Plant management’s confidence in process is high. Over 125 members of Fitzpatrick’s management team have participated in a KT workshop to improve their skills. Managers are holding “KT kickoffs” ahead of training and “KT luncheons” afterwards to identify what’s going well and what needs improvement. A key item identified for improvement by workshop participants is to build KT rational process into plant procedures. The Design Change process, with an annual budget of $11 million, is now undergoing an enhancement based on Kepner-Tregoe processes. Other processes identified for improvements include Work Control and Corrective Action.
Support continues to grow. Steve and I are routinely asked to facilitate process, resulting in some creative schedule juggling. During a recent week, I helped with a PA (which turned out to be a Decision Analysis) to correct deficiencies with our nuclear instrumentation connectors and with another PA to identify the cause of a high failure rate of a steam line valve. Meanwhile, Steve was assisting with a DA to choose how best to improve equipment status control in the plant and a PA to identify the cause of leakage in a large valve in a plant emergency cooling system.
Our challenge is to keep the momentum going. I don’t believe things happen by fate and chance. I believe things happen for a reason. Our KT involvement began when industry regulators told us we needed better problem solving skills—no surprise to us. One of our managers, Dave Wallace, looked around the industry and heard good things about KT. Although he hadn’t taken the training personally, he gained approval to get Steve and me certified.
The turbine trip, just a week after our training, gave us an immediate and high visibility “opportunity” to demonstrate the power of process. That was a lot of pressure, but it proved the strength of the LDI and set us on the path to success. Now we look forward to capitalizing on the full value of process.
About Kepner-Tregoe:
Founded in 1958, and based on ground-breaking research regarding how people think, solve problems, and make decisions, Kepner-Tregoe provides a unique combination of training and consulting services to improve quality and effectiveness while reducing overall costs. The KT methodology is used at every level of client organizations: to implement strategy, achieve continuous improvement, increase customer satisfaction, and drive effective issue resolution throughout the organization.