It is always good to have someone occasionally look a wee bit harder at the things 'we all know'.

... figures are given in a Digital Equipment Company publication of the 1980s which suggested that it was 50% more expensive to fix a bug at the coding stage than at the design stage, 10 times more expensive at the integration stage, 60 times more if it’s caught in alpha testing and 100 times as expensive to fix in customer beta test.

Of course, in those days if a customer encountered a bug you’d have to plough through crash dumps arriving by international courier or send an engineer on site to sort out problems so it was definitely worth avoiding.

There's an inherent difficulty in measuring these figures in a meaningful way: whereas you can quantify the amount of time actually spent fixing bugs encountered in these different situations, it's difficult to reliably estimate the amount of time that would have been spent finding those same bugs at an earlier stage in the development process. It may be that the extra cost of fixing bugs in production is significantly offset by the amount of time it would have otherwise taken to uncover them in development, particularly in complex multi-threaded, multi-component systems.

Arguably, if you feed "lessons learned" from production bugs back into the development process you can mitigate a proportion of the cost, but how many shops are really that organised?

> [...] how many shops are really that organised?

Aren't all IT shops CMMI Level 5 self-certified these days?

Sociology is the study of societies.

Methodology is the study of of methods.

"I'm studying British sociology" is not the same as "I'm studying British society".

Any given way of doing something is a *method*, not a methodology.

Are you sure? :~

Definition of methodology

1 : a body of methods, rules, and postulates employed by a discipline : a particular procedure or set of procedures

* demonstrating library research methodology

* the issue is massive revision of teaching methodology

— Bob Samples

2 : the analysis of the principles or procedures of inquiry in a particular field

Oh, well. Bugs are OK then.

Bugs are not bad. You must accept your bugs.

Fixing things long after they have gone live can be an expensive process partly because of our modern bureaucratic way we develop software.

During development, the people writing the code, the people writing user stories, the project managers, management and the testers are all available and willing to iron out problems discovered during development. Funding is available, expertise is available as are the people who understand and codify the business logic. Releases are made frequently and sometimes regularly.

Compare that to the time when a bug is discovered. The team who did the work originally may very well not be working on the project anymore (or even be in the company). Depending on who is pushing to get the bug fixed, it may not get the weight it deserves from management. If there is a new team working on this product, they will all have to read up on how it should have been, on how the code is at the moment, what consequences might arise from this change. The testers will have to get to work on getting up to date on the changed functionality, if convenient or nicely-written test-cases are not available.

Now, to be sure, this extra work to push against the bureaucratic flow is not 100x more expensive, but it *is* more expensive than getting it done right while the original team were in full development mode.

One thing not mentioned in the cost here: Are you willing to bet your companies reputation and cash on not fixing a bug?

As several companies have recently found out, how expensive was that cyber attack? What did it cost your company in brand reputation? How much did it physically cost your clients to repair the damage?

How many lawsuits did your company just swallow because of a bug?

I'll put a paycheck on a Los Vegas bet that 100x would be cheep compared to the cost of a very ugly lawsuit train.

The Newsletter seems to focus on the cost involved to fix a software problem within the software team.

There are numerous other teams involved including testing, production, shipping, field engineers etc. (not to mention sales & marketing - ok I just did !).

If a defect is detected early and self contained within the development team and rectified within a timely manner the others teams need not be involved hence the costs can be relatively small.

However, defects are often like snakes and ladders causing a design to go back many steps & teams possibly to the very beginning: it doesn't take an expert to figure out the cost is generally cheaper the smaller the snake !

Of course reviewing code finds coding mistakes; the question is whether this a cost-effective approach to software product development.

Yes, mistakes fixed in later phases will involve more effort; some evidence: http://shape-of-code.coding-guidelines.com/2020/08/23/time-to-fix-when-mistake-discovered-in-a-later-project-phase/

but would it be more expensive, on average, to have fixed these mistakes earlier?

Products have a finite lifetime; some evidence: http://shape-of-code.coding-guidelines.com/2018/11/28/half-life-of-software-as-a-service-services/

Some coding mistakes are never experienced as faults by customers before the product is withdrawn; some evidence: http://shape-of-code.coding-guidelines.com/2020/12/20/many-coding-mistakes-are-not-immediately-detectable/

Figuring out the most cost-effective approach is very hard.

Many years ago I worked for IBM developing a System 390 product( on VS1/DOS). People had to record each week where they spent their time, For example, writing the spec, reviewing the spec, code, Unit test, FV prep, FV execution, FV Fix validation, System test prep, system test execution... system test fix validation. Golden code drop execution. There was also the cost of support. Time spent in PMRs, time spent diagnosing problems, time fixing problems. etc.

We had one guy whose job it was to collate all this stuff and come up with the numbers.

The numbers were pretty consistent. These numbers were reviewed by an executive who had oversight in many labs.

Dimension 1 - count the number of people

For a bug found in unit test, one person was affected.

For a bug in FV there was 2 people affected( developer and FV)

For a bug in Systems test there was 3 people affected( developer, FV, Systems test)

For a bug found by the customer there was IBM L2 support, IBM change team, then the original developer to migrate the fix, FVer to change the FV tests, Systems test to change the systems test. Bits of managers, bits of overhead ( the guy doing the collating).

For critical problems, factor in many calls to the customers, "call cooridinator" overhead, executive attention and 5 layers of management.

________________________________

Dimension 2 simple complex and subtle problems

I think (this is going back a few years) that we also classified the bug as for simple, complex, subtle.

Unit test found most simple bugs. Systems test found complex, and subtle, customers found many subtle and not many simple ones.

Complex and subtle took more time to design and run tests. (Some bugs you had to run for a week being really nasty to cause it)

So as well as bugs found by customer's taking more effort to fix, these bugs were harder. So an n**2 problem.

iow, the cost was a direct reflection of your org structure.

The British Post Office software - Horizons, was it? - comes to mind. Bugs denied, with vastly expensive consequences.

"...with vastly expensive consequences."

And not just money, lives were lost/destroyed.

Quite a long time ago we had a situation where the software checked out perfectly at every stage. It went into production, and everyone was happy for several years. Then just one customer complained about errors seemingly at random. It took some time to work out that not only was it just one customer, it was also just one machine. However, that machine had been fine with a previous version of the code (and was fine if we reverted it). Eventually our code virtuoso discovered that particular processor did something slightly different with float->int conversions, and in our new code the order of calculations had changed. To this day I don't know if it was the processor or our code that was actually at fault!

If you take a specification, create a test to see if that specification is met and then write software to pass that test, you will miss loads of bugs. Users in production will find them. Cant' beat real world use.

Bugs in undeployed development code have essentially no cost to the user . Once the code has been deployed then, exploits, etc. aside, there is the cost of information and planning, testing and applying patches, which may or may not require downtime.

If it weren't for the different legal situation, industry could provide a reasonable benchmark for when things like product recalls are required.

There is a fairly large difference between code bugs and design errors. I have been in IT since 1974, and I have always heard that it costs more to fix design errors in production that in development. A simple bug, may not cost any more to fix in production, though a complex bug, with interdependencies, may require a more complicated solution. However a fundamental design flaw (as we have seen with a number of major government projects), may cost enormous amounts to correct, and might well cause the entire project to fail. So I think this is a very broad claim that, as the author suggests, is not and indeed cannot be, verified by research, as you would have to classify many kinds of "bug", where they come from, the effect they would have on interconnected and dependent systems and processes, as well as the extent of the effort required to fix them. Fixing a code bug might cost very little, but manually verifying the integrity of the data affected by the bug could be very expensive.

I've always vaguely thought that the sort of quotes about an error discovered in step N after it was made being a power of N more expensive to correct was based on an assumed process of going back to the step where it was made, working through the whole development from there and the ultimate cost was based on the cost of each step so that the 100 times cost reflected a 10-fold cost from moving from one step to the next multiplied by a further 10-fold cost moving from that step to where you were. Whether the implied assumption was valid or that the costs were really an order of magnitude a matter for debate, of course. As the article implies it's the sort of assumption that gets easily thrown off the top of the head by an instructor in a course on development...

What is clear is that once something is in deployment the costs of fixing a bug might be insignificant compared to finding and fixing all the faulty data that's now out there.

As customers using Intel processors (other mfrs also have issues) have found out over the years, bugs embedded in the microcode, can be unfixable - if the manufacturer does not want to swallow the costs of replacing the faulty processors.

Costs of performance, maths accuracy, security, stability, etc., are usually incurred solely by the purchaser.

Mfrs can sometimes mitigate these bugs with patches, but again it's the customer who will suffer greatest.

It seems that some companies would rather not fix coding/hardware issues during development where it is potentially cheaper for all concerned, but continue to supply products that are *known* to have congenital faults anyway.

I now actively avoid Intel and Apple products due to their ongoing refusals to fix issues in their products and passing the costs, real and hidden, onto their customers victims. Especially in the case of Apple, when they will vehemently deny there is a problem and then charge customers for repairs to faulty designs. (Don't just take my word for it - have a look at Louis Rossmann's YT channel to see what sh*t Apple gets up to on a regular basis)