Shuttle’s software quality head favours pragmatic approach to process improvement

by Prof Barry Dwolatzky

You’re reversing your car when, suddenly, you hear a loud crunch as you drive over something. You stop, jump out and run to see what you’ve hit. It’s your 5-year-old daughter’s tricycle crushed under your back wheel. Someone must have left it in the driveway yesterday. What do you do next?

a) Feel extremely relieved that your daughter wasn’t sitting on the tricycle when it got crushed under your car …. and then forget all about the incident;

b) Feel irritated that you’ll now have to buy another tricycle … and then forget all about the incident;

c) Feel angry that the tricycle was left in the driveway … and then forget all about the incident;

d) Spend the rest of the day thinking about how it could have happened that you reversed over a child’s tricycle (with or without a child on it). You analyse your routine, you think about precautions you will take in the future and you get your family together to discuss how a similar incident can be avoided in future.

To be honest I probably fall into category a), b) or c).  I would probably think, “thank heaven’s no one got hurt!” and then forget about it. Life is busy and who has time to worry about what might have happened.

I met someone recently who definitely falls into category d). His name is Ted Keller and he worked for IBM as the software quality manager on NASA’s Space Shuttle programme. Before each Shuttle launch a number of senior managers were required to sign a form certifying that the launch could proceed. In signing this form Ted Keller was making a very specific and terrifying statement. He was formally certifying that the Shuttle’s software was 100% correct and free of defects and errors. As he says, “Anyone who knows anything about large and complex software systems would probably have to be an idiot to make such a statement.” Not only was Ted Keller’s professional reputation on the line every time he signed the “Flight Readiness” certificate – he was also accepting responsibility for a $4 billion space vehicle and the lives of 7 astronauts, many of whom were his personal friends and neighbours.

How did Keller feel confident enough to sign the “Flight Readiness” form time after time? He attributes his faith in the Shuttle software to two things: (i) the capability and dedication of the team of software engineers who developed it, and (ii) the processes they followed to ensure that everything humanly possible had been done to ensure quality. Every time a defect was detected in software testing, causes were analysed and processes were changed to ensure that the same error never occurred again – category d) in the tricycle example. And it paid off. The Space Shuttle flew for 30 years and did not experience a single mission critical software defect. Keller’s confidence in his software was justified.

I met Ted Keller last week at the SEPG North America Conference in Albuquerque, USA, where he presented a really interesting paper. He spoke about his frustrations and successes in bringing the lessons learnt on the Shuttle programme to other domains. On leaving the Shuttle programme Keller began working as a consultant trying to convince software product companies, banks and others engaged in software development that they should follow NASA’s lead in developing high quality software. To his surprise managers and developers showed very little enthusiasm for the message he brought. In their domains quality just had to be “good enough”.

Over time Keller has changed his approach. He now begins by understanding the “pain points” experienced by software developers and their managers. While, in the case of the Shuttle, defect free software was the key business goal, other domains have their own business drivers.  These may include getting a product to market quickly, keeping a project within budget, or eliminating the need for team members to work overtime. In each case Keller has been able to draw on the lessons he learnt working on the Shuttle and come up with a process improvement strategy that helps an organization eliminate its specific pain points.

He listed the following as some of the lessons he has learnt over his long and distinguished career:

1. Don’t treat process improvement as a “textbook” activity. Textbooks provide guidance but DO NOT teach how to improve processes for any real world situation.
2. “Process tailoring” has evolved to “process crafting” to create a process improvement method that is appropriate to a specific organization.
3. It is important to understand which aspects of the organization’s performance can be changed. “Traditional” parameters such as cost, quality and even the skills of workers may not be alterable. Restrictions on education, population, time, materials, workable hours, travel, schedules, etc. can limit or prevent many “text-book” process improvement activities.
4. Before starting on a process improvement journey one should make certain of senior executives support.  It is not always true or obvious that there will be a positive ROI or that any financial benefit can be realized soon enough to justify the impacts and perceived risk to the business.
5. One of the most successful approaches to achieve process improvement buy-in and stakeholder participation (with or without Executive endorsement) is to build and advertise the intent as a remedy for serious “pain points” being experienced by the stakeholders.

In summary Ted Keller is not an advocate of textbook process improvement but recommends an agile, flexible and pragmatic approach. This is interesting and somewhat surprising coming from the man who successfully shaped the software development processes at the heart of the Shuttle programme.