A while back, Barry Boehm & Vic Basili wrote a nice summary of best ways to get better quality software. Their advice still applies to embedded systems today. Below is their list (in bold) with my commentary (the parts not in bold).
1. Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase
Bug fix cost gets worse as your software gets closer to deployment, because you have to not only spend a lot of time tracking down the source of the bug, but also retest the system after the fix. It is common to find situations in which a bug "escape" into field units costs dramatically more than 100x. Consider having to recall a fleet of cars to install a bug fix, or do maintenance visits to thousands of sites to manually install a fix. (Over-the-air fixes have their own problems, but that's a topic for another time.)
2. Current software projects spend about 40 to 50 percent of their effort on avoidable rework.
As the joke goes, the first 90% of the project is spent on design, and the second 90% on debugging. The cheapest way to debug is to avoid putting the bugs into the software in the first place. The next best way is to find them at the point of introduction (e.g., peer review of design before code is written) rather than at system test.
3. About 80 percent of avoidable rework comes from 20 percent of the defects
If you have a "bug farm" that's often not because the code is bad, but rather because the underlying requirements and design are bad. If one module has a lot of bugs you should rewrite the module rather than keep patching it. However, if during the rewrite you might well discover that the problem isn't really the code, but rather a bad design or unclear requirements. Writing new code for a bad design ultimately won't solve the problem.
4. About 80 percent of the defects come from 20 percent of the modules, and about half the modules are defect free.
In addition to comments for #3 above, modules with high cyclomatic complexity tend to be difficult to test and tend to be more bug-prone. Keeping a limit on complexity can help with this problem.
5. About 90 percent of the downtime comes from, at most, 10 percent of the defects.
It makes sense to weight testing on the areas that are the highest risk. For desktop software this often corresponds to the common use cases. For embedded systems and other mission-critical systems this also means testing failure detection and recovery for high-cost failures.
6. Peer reviews catch 60 percent of the defects.
Yes, really. Peer reviews catch the majority of defects. Why aren't you doing them? (If you are doing them, are they catching at least half the defects?)
7. Perspective-based reviews catch 35 percent more defects than nondirected reviews.
When you do reviews, give each reviewer a checklist with a different set of areas to think about or different role to play. For example, control flow, data flow, exception handling, testability, coding style.
8. Disciplined personal practices can reduce defect introduction rates by up to 75 percent.
As much fun as it is to be a coding cowboy, on average even the best programmer will introduce fewer bugs by following a methodical engineering practice rather than slinging code. As mentioned above, the cheapest bugs to fix are the ones that never made it into the code. Beyond this, there are practices such as PSP and TSP that are shown to dramatically improve quality without really changing productivity.
9. All other things being equal, it costs 50 percent more per source instruction to develop high-dependability software products than to develop low-dependability software products. However, the investment is more than worth it if the project involves significant operations and maintenance costs.
In other words, if a product recall puts your company out of business, it's worth investing in good software quality up front. In my experience if you are already shipping a mission-critical product you're already spending that 50 percent more (and then some), but still shipping defects. This isn't saying spend even more. Rather, doing peer reviews and some other basic software quality practices you can improve quality at the same cost you're already spending. Testing software into submission is simply not the way to go.
10. About 40 to 50 percent of user programs contain nontrivial defects.
If you have programmable features, your customers will have bugs in what they do. And don't forget that your financial management spreadsheets are user programs (i.e., can, and often do have bugs).
Items #1 - #10 from:
Boehm & Basili, Software Defect Reduction Top 10 List, IEEE Computer, Jan. 2001, pp. 135-137.
You can read the original three-page article here:
https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf
1. Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase
Bug fix cost gets worse as your software gets closer to deployment, because you have to not only spend a lot of time tracking down the source of the bug, but also retest the system after the fix. It is common to find situations in which a bug "escape" into field units costs dramatically more than 100x. Consider having to recall a fleet of cars to install a bug fix, or do maintenance visits to thousands of sites to manually install a fix. (Over-the-air fixes have their own problems, but that's a topic for another time.)
2. Current software projects spend about 40 to 50 percent of their effort on avoidable rework.
As the joke goes, the first 90% of the project is spent on design, and the second 90% on debugging. The cheapest way to debug is to avoid putting the bugs into the software in the first place. The next best way is to find them at the point of introduction (e.g., peer review of design before code is written) rather than at system test.
3. About 80 percent of avoidable rework comes from 20 percent of the defects
If you have a "bug farm" that's often not because the code is bad, but rather because the underlying requirements and design are bad. If one module has a lot of bugs you should rewrite the module rather than keep patching it. However, if during the rewrite you might well discover that the problem isn't really the code, but rather a bad design or unclear requirements. Writing new code for a bad design ultimately won't solve the problem.
4. About 80 percent of the defects come from 20 percent of the modules, and about half the modules are defect free.
In addition to comments for #3 above, modules with high cyclomatic complexity tend to be difficult to test and tend to be more bug-prone. Keeping a limit on complexity can help with this problem.
5. About 90 percent of the downtime comes from, at most, 10 percent of the defects.
It makes sense to weight testing on the areas that are the highest risk. For desktop software this often corresponds to the common use cases. For embedded systems and other mission-critical systems this also means testing failure detection and recovery for high-cost failures.
6. Peer reviews catch 60 percent of the defects.
Yes, really. Peer reviews catch the majority of defects. Why aren't you doing them? (If you are doing them, are they catching at least half the defects?)
7. Perspective-based reviews catch 35 percent more defects than nondirected reviews.
When you do reviews, give each reviewer a checklist with a different set of areas to think about or different role to play. For example, control flow, data flow, exception handling, testability, coding style.
8. Disciplined personal practices can reduce defect introduction rates by up to 75 percent.
As much fun as it is to be a coding cowboy, on average even the best programmer will introduce fewer bugs by following a methodical engineering practice rather than slinging code. As mentioned above, the cheapest bugs to fix are the ones that never made it into the code. Beyond this, there are practices such as PSP and TSP that are shown to dramatically improve quality without really changing productivity.
9. All other things being equal, it costs 50 percent more per source instruction to develop high-dependability software products than to develop low-dependability software products. However, the investment is more than worth it if the project involves significant operations and maintenance costs.
In other words, if a product recall puts your company out of business, it's worth investing in good software quality up front. In my experience if you are already shipping a mission-critical product you're already spending that 50 percent more (and then some), but still shipping defects. This isn't saying spend even more. Rather, doing peer reviews and some other basic software quality practices you can improve quality at the same cost you're already spending. Testing software into submission is simply not the way to go.
10. About 40 to 50 percent of user programs contain nontrivial defects.
If you have programmable features, your customers will have bugs in what they do. And don't forget that your financial management spreadsheets are user programs (i.e., can, and often do have bugs).
Items #1 - #10 from:
Boehm & Basili, Software Defect Reduction Top 10 List, IEEE Computer, Jan. 2001, pp. 135-137.
You can read the original three-page article here:
https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf
