Mar 19, 2009 / Zero Defects? Are You Kidding Me?
11 comments
Are you familiar with zero defects mentality? It looks very good from the first sight. Zero defects... Let me think... Well, cool! I like it! I’d like to have zero defects in my projects.
So, what is zero defects mentality? Here is the quote from Lean Software Development an Agile Toolkit book:
“One of the fastest ways to kill motivation is what is called in the US Army a zero defects mentality. A zero defects mentality is an atmosphere that tolerates absolutely no mistakes; perfection is required down to the smallest detail. The army considers a zero defects mentality to be a serious leadership problem, because it kills the initiative necessary for success on a battlefield” — Mary & Tom Poppendieck
Obviously, zero defects mentality is not something that HOUSE M.D. likes ;) Moreover, I think Dr. House hates zero defects mentality. It causes several unpleasant effects:
- Not enough courage to refactor complex, messy, buggy, but important piece of code.
- Can’t make important decision, instead make less risky, but wrong decision.
- Do everything to avoid responsibility, that leads to coward and stupid behavior.
"Zero defects" may sound good. But in reality you still have errors after production. Even in predictable and (quite) easily testable industries (hardware, automobile, etc.) there are problems with 100,000 power adapters that should be replaced (or hard drive problems, or engine problems, I bet you can continue the list).
How can we expect zero defects in software development? It is harder to test, harder to define in details, harder to predict. Software development is a complex adaptive system, we can't predict all effects. Bugs in production is a normal thing, and by “normal” I mean we can't bring them to zero. We can (and should) minimize them using all possible ways, but The Last Bug is a mirage.
There are several obvious strategies that may help:
- Test Driven Development. Nice side effect of TDD is a unit tests suite. You have tests for new code, and you have unit tests for old code. More tests — less bugs.
- Continuous integration. Instant feedback helps to identify problems early and fix them early. It saves time (and money) and reduces bugs in production.
- Automated regression functional tests suite. Unit tests are good, but you need something else. Functional tests emulates user behavior and find user interface errors, integration errors, etc. Needles to say you should have continuous integration in place to succeed with automated functional tests.
- Root cause analysis. There are several ways to fix a bug. You may just hack the code and apply a patch (please, don’t do it at home!). You may think deeper and fix the bug nicely. Also you may look into the root of the problem and fix it. Yes, it may take time, but you will prevent future bugs from this domain.
- High development skills. Ironically, high skills do not always reduce bugs rate in production. “Stars” may be strong in some areas (business layer), but may not polish things. It may lead to many small bugs, especially on UI.
Is it a good thing to have "Zero Defects" goal in the sprint? The answer is "Are you kidding me? Zero bugs? It's a miracle!"
Labels: agile, bug management
posted by Michael Dubakov @ 12:10 PM
11 Comments:
I think "Zero bugs? It's miracle!" is a big simplification. Do you mean you don't care about bugs at the end of a sprint? Of course you do! "Zero defects" mentality is not about having zero bugs (because you're never sure), but about being sure you DO ALL that's necessary not to have them at the end of a sprint.
All those practices you've mentioned are helping to achieve that "Zero defects" goal or to come as close as possible to zero bugs. And this is also what lean development (in my opinion) have in mind when talking about eliminating waste in terms of fixing bugs later instead of avoiding them during the production process.
I have seen many companies that don't have the "zero defects" mentality, and the result is that they are driven by bug fixing. So I think every developer should have a "zero defects" mentality, which IS NOT a "zero mistakes" mentality. We need to do mistakes to learn and be creative, but in the same time we need to do all that we can during the sprint to avoid/eliminate all possible defects.
A debugged program is one for which you have not yet found the conditions
that make it fail.
-- Jerry Ogdin
Every program has at least one bug and can be shortened by at least one
instruction -- from which, by induction, one can deduce that every
program can be reduced to one instruction which doesn't work.
@Marcin
Zero defects mentality is a fear to change anything and receive a bug report (or angry customer, or angry PO, etc). It is a quite usual thing in large waterfall projects. Agile paradigm and principles fights with this tendency and provide practices that transform the fear into courage.
I want to stress that you definitely should try to have as few bugs as possible, but there should be NO any extreme goal like zero bugs and NO punishment for bugs in production.
I think we agree on the need of continuous quality improvement and of course on that there should be no penalties for bugs slipping into production. Perhaps I was wrong on thinking about the meaning of "zero defects" mentality which for me was closer to the Lean's Jidoka and thus more positive.
In other hand what I want to stress is that if we admit that it's not possible or not worth to pursue the goal of eliminating all bugs during the development stage, then we will quickly stop doing that.
In fact I know companies that use that as an excuse to never start pursuing that goal. They prefer to wait for the bugs to come as it's completely acceptable for them to have hundreds of tickets with many levels of SLAs, etc. So they end up with the exactly opposite mentality EVIL! EVIL! You better have "zero defects" mentality but I should say - you better implement Jidoka.
@Marcin
No objections. I support your points.
Big difference between a zero defect mentality and a quality first mentality. Quality first implies quality is not a process but built into the process. Quality is a conscious effort that is part of infrastructure and design, not a stage at the end of the development. It is baked into the system and it is not about striving for a number, but rather striving for results.
I work a paper a couple of years ago, A False Measure of Success, that talks about a companies strive for zero defect, but at the expense of business needs. The technology was dictating the quality measures to the business rather than the other way around. May be an interesting read if anyone is interested in this topic.
http://onemoreagileblog.blogspot.com/2009/02/false-measure-of-success-revisited.html
"Zero defects mentality" is just a terrible name. It gets misinterpreted, and misapplied. As I understand it, the original intent was more along the lines of "Zero or one known must-fix defects at a time, or two, maybe, if you didn't get a chance to fix the first one before the second one was discovered."
It's probably better to say it this way: your priorities with regards to must-fix defects should be:
1. Prevention
2. Fixing
3. Tracking
If you put it that way, it seems obvious, except that most places don't live that way. They use tracking first, as a rug to sweep hundreds of defects under. Eventually, management puts its foot down and demands that the defect list is "too long" and must be tackled. Morale slips, if it hasn't already under the realization that the project is a stinkin' pile o' crap. Nobody likes fixing defects. That's why they were swept under the rug in the first place. Now many of the defects are months old, and whatever caused the defects is long forgotten, and it's time to fire up the ol' debugger to figure it out. It's gonna take a while.
There is also a team dynamic problem with zero-defect mentality. Who wants to be the one to stop having fun developing features, spend some ugly frustration-time fixing a bug that one of your co-workers probably caused, and then get docked in your performance evaluation for not developing as many features as your co-workers? Just log it in the tracking system, and continue coding like hell. It's the best individual strategy on a mostly-unmanaged team. And most teams are mostly unmanaged.
@Alan
Prevention -> Fixing -> Tracking. I like it :)
I think you touched a point when you called bugs "must fix defects". All bugs are not created equally: some are "stoppers", some are rare events based on specific usage scenarios and some just evaporate as development progresses, as then get refactored away.
The problem is to know how to tell them apart. I think if you are shipping code without known "issues", you either have not been looking hard enough or your product is either very simple or very mature. If you define zero bugs as "all the unit test pass", then your are not really looking.
We have just gone through our first public beta release of our new product. While integration testing we did discover a number of bugs, most of which had not been discovered during development. Bug detection is hard, but we must also categorize bugs by effects and deal immediately with certain classes of bugs. I really don't think zero defects, or no known defects makes sense from a product lifecycle point of view.
While someone has made the point that bugs have to be corrected as early as possible, I seriously doubt that this is useful. It has been my experience that early in product development, some issues are there just because all features have not been implemented or there implementation has not yet stabilized. These bug have a tendency to just go away as development progresses. Normally you have to watch them, but they should not be fixed before their time.
Software is implemented, maintained and abandoned. It takes profound insight to understand what bugs, smells, issues and features have to be addressed and at what time in a product lifecycle, if we are to realize products that are at once, useful to the customers, fulfill their requirements in a reliable way and can be done with a reasonable budget. I think religious answers to this hard problem, like "zero defects" guarantee failure.
Hi Michael,
thanks for another thought-provoking post! i enjoy reading your blog.
Hmm, several issues here.
1) First of all I think we should come to a culture where we do not call more or less serious flaws in IT systems 'bugs'. To me, it does not make sense to belittle the fact. Let's call them what they are: defects (I'm not going into the possible definitions of a defect here).
This said, look here: http://tr.im/iY1X scroll down a bit, and laugh.
And here's a funny cartoon: http://xkcd.com/319/
2) A couple of years ago I had the luck to be part of a team that delivered (close to) zero defects. I wrote down the 10 principles we used from a requirements perspective. See
http://tr.im/iXZF (short version on PlanetProject, more or less uncommented), or
http://tr.im/iYj9 and http://tr.im/iYlZ (longer, 2-part version on Raven's Brain).
Interesting enough, only 1 out of the 10 principles is directly related to a defect-finding activity. We had a 'zero defect mentality' in a way that we all seriously thought it would be a good idea to deliver excellent quality to the stakeholders. Fear or frustration was not at all part of the game. Mistakes were tolerable, just not in the product at delivery date. Frankly, we were a bit astonished to find out we successfully delivered a nearly defect-free non-trivial system over a couple of releases.
3) While it seems to be a good idea to use the different strategies that you proposed in the original post, I'm missing some notion of how effective the various defect-reducing strategies are. The only relevant quantitative research I know of come from metrics guru Capers Jones. If I remember correctly he states that each single strategy is only about 30% effective, meaning that you need to combine 6-8 strategies in order to end up with a 'professional' defect removal effectiveness. AND you cannot reach a net removal effectiveness of, say, 95% with testing alone. From an economic standpoint it is wise to reduce the number of defects that 'arrive' at testing in the first place, and this is done most effectively by formal inspections (of requirements, design and code).
Greetings!
Rolf
PS: if you find a defect in my comment, you can keep it ;-)
I think it all comes down to the team your work in, as no process is going to succeed without appropriate people. Zero defects mentality seems to be compelling, but frustrating the same time, too. We are all humans who tend to make mistakes, therefore the question should be how to handle those bugs coming up, instead, and which workflow to set up to address them the way that the time taken away by fixing wont affect the time needed for coding and advancing in the sprint or project. I agree that you have to combine more strategies to achieve your testing goals, but for a product of quality you don't need wipe out all the bugs, and it wont make your project succeed, but make sure you deliver a marketable product that you can sell.
Post a Comment
<< Home