Fragile Agile development model is a symptom, not a source, of project failure
- Reference: 1718008205
- News link: https://www.theregister.co.uk/2024/06/10/agile_opinion_column/
- Source link:
The result: at least according to the study, [1]Agile projects fail exactly 268 percent more often. Not 267 percent or 269 percent, and how can you argue with precision.
This would seem to move the needle towards a perception of the methodology as a type of culthood, and here Agile hasn't always helped itself. The basic idea is that software projects are too complex to manage by creating a precise plan and sticking to it, so lots of feedback and ability to change has to be built in along the way. Agility, in other words. Sounds sensible enough, but the creators of the idea spiced it up by writing the [2]Agile Manifesto .
[3]
Manifestos are manifestly dangerous. They start out by describing how a new way of doing things can solve old problems, which is often necessary. Once the new ways are in place, though, the nature of the problems is going to change – and manifestos rarely change to match. There is much irony in the Agile Manifesto preaching pragmatism over prescription and openness to change over rigid adherence to overspecific plans.
[4]
[5]
To abuse a saying commonly used of politics and religion, Agile's problem isn't Agile, it's the people who believe in it. To be more specific, it's the people who believe that adopting a new, preferably fashionable methodology, can by its mere presence fix what went wrong in the past. Yes, our projects were delivered late, over-budget and parodies of what was promised, if they were delivered at all. Can't happen with Agile, it fixes all the problems.
It can and doesn't. Naughty Agile.
[6]
Look at this from the viewpoint of the people delivering the projects – those charged with putting finger to keyboard and making code happen. They have tasks to do, goals to meet and resources of time and other team members to help. They want to do a good job: there is little as satisfying as being a part of success, and little as dispiriting as being on Team Fail. Especially if it was clear what was going wrong and nobody had the power to change it. All this is true regardless of methodology: it's how a methodology helps or hinders the individual that makes the biggest difference.
The waterfall principle which provoked the rise of Agile had plenty of structural problems that made people party to failure without a chance to fix it. Detailed specifications turned into detailed schedules of highly compartmentalized tasks look great to management. They are also inherently vulnerable to the unexpected discovery, the incorrect assumption, and the change in circumstances. With little feedback other than cascading slippage and few remedies other than big reboots, the individual developer is locked in the cells of the project chart. You can make things worse, you can't make them better.
Agile sees all that, and it feels your pain. Design and specification documents are functional breakdowns of the essential components of a project, not necessarily detailed blueprints. Get the skeleton up and running, compare notes with others as you go along, and problems will be spotted much sooner - and everyone they touch can be part of the solution. Which is lovely, if that's all you've ever known and you understand the disciplines underlying the ideas. If you're writing a component that has to stand up before it's finished, the target is still to make the finished component on time, not to forsake that for the intermediate stages. If the pressure's on you to do the wrong thing, or there's no guidance in how to do the right, it won't end well. Demotivation once more.
[7]
As for letting the code lead the documentation and interacting across the project, these can and do go horribly wrong if you don't see them as tools in their own right with their own rules to be applied successfully. You can't just take a group of people used to working under one ill-fitting methodology and force them into another. If the company culture has been top-down management and plans handed to the tribe on tablets of stone, handing out another set of Thou Must and Thou Must Not will go about as well as things did in biblical times.
It may be true that Agile projects fail twice as often as others, although one study is just one study. That may be because the Agile Manifesto correctly identified the problems but didn't think through the implications of its remedies.
It's more likely that Agile's bad rap comes in the same way that Macbeth has become the "unlucky" play in theater lore, where even to say its name brings calamity. That happened because Macbeth was seen as such a guarantee of success that theaters in trouble put it on as a last ditch attempt to stay afloat. This rarely worked, so the play became associated with failure.
[8]Study finds 268% higher failure rates for Agile software projects
[9]Woo-hoo, UK ahead of Europe in this at least – enterprise IT automation
[10]Red Hat bins Bugzilla for RHEL issue tracking, jumps on Jira
[11]The Pentagon has no idea how to deal with bad cloud contracts, say auditors
And so it is with the huge slapdash edifice of corporate IT development, with so much cultural and managerial inertia that change will come not with new ideas, but the aging out of the old guard.
That, perhaps, we can find out, not by counting failed projects and grouping them by methodology alone, but by what sort of projects, what sort of organizations and what sort of people were involved.
Not for the first time, the IT industry stands desperately in need of serious analysis as the group of people with the most power over the future of us all and which is under the least amount of scientific scrutiny. Perhaps we can write that up as a manifesto. ®
Get our [12]Tech Resources
[1] https://www.theregister.com/2024/06/05/agile_failure_rates/
[2] https://agilemanifesto.org/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZmbOxPU4iEP3sAWm8JmNJAAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmbOxPU4iEP3sAWm8JmNJAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZmbOxPU4iEP3sAWm8JmNJAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmbOxPU4iEP3sAWm8JmNJAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZmbOxPU4iEP3sAWm8JmNJAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/06/05/agile_failure_rates/
[9] https://www.theregister.com/2023/11/07/red_hat_survey_automation/
[10] https://www.theregister.com/2023/09/29/red_hat_bugzilla_jira_migration/
[11] https://www.theregister.com/2023/09/14/the_pentagon_has_no_idea/
[12] https://whitepapers.theregister.com/
"the right tool for the job"
Or even the right tool for parts of the job. As I've frequently repeated, agile and "waterfall" are not mutually exclusive approaches. At the simplest level for example, agile is ideal for UI development because users typically can't envisage how it will work in abstract but can identify good and bad elements of it on sight. On the other hand, application security is very hard to ensure consistently across a big agile project unless a full specification of what is required is presented to all teams -- i.e. a "waterfall" specification has been created up front.
The big problem we face is that methodologies have become more important than results, and not only in the IT sphere. It's become the basis for much of education in general, as witnessed by a recent UK dept. ed. spokesman stating that grade boundary marks should be shifted downward "to maintain standards" (i.e. pass rates).
So this is a cultural phenomenon not a software development one. It just shows up most clearly in software development because we rely on its results so much.
Re: "the right tool for the job"
That.
In my line of business we do have certain things that are set in stone, and they should be, considering they are, in fact, laws.And then there are things we should be clear about beforehand, like how quickly do you want a result to be returned from the black boxen living in the underbellly of the datacentre. I'm pretty sure that ten minutes is too long. How about... 30 seconds? I don't know, nobody knows, nobody writes these things down. Would agile fix this? Maybe, if we could get users to interact with the half baked software, play around, and give feedback - that must be heard, not ignored or downplayed (gosh, how often have I seen that?).
From my point of view, if we could prioritise getting the whole stack to be more snappy instead of adding new features, giving the developers half a year time to do some housekeeping, let them pay off technical debt, that would be a Good Thing.
I think the fundamental problem is inability to make a plan and execute it, and it's getting steadily worse. Buzzwords and slogans are taking the place of planning, while smoke and mirrors are taking the place of execution.
If you can't plan, there is no methodology that will save you.
I suspect that Agile appeals to the buzzwords & slogans crowd, because it appears to provide justification for their deficiencies. You don't have to plan! In fact, it's better if you don't! This is not true, of course, and it only looks like that on the very surface. Unfortunately, that kind of mind does not look beyond the surface.
As a consequence, more Agile projects fail, but those same projects would probably have failed under any other methodology. Correlation, not causation.
My experience is that when allowed to figure it out for themselves, competent people will evolve a system that works quite well for their project. It won't be agile and it won't be waterfall and it won't be any of the other daft methodologies that have been trendy from time to time. But it will be more or less optimal.
Then what usually happens is that some clueless manager will come along and demand that everybody must conform to their particular project management religion. And it's usually completely unsuitable for the project. Then everybody gets fed up and productivity falls. Eventually, after a lot of complaints and to prevent the project from complete failure, the methodology gets incrementally tweaked until it's not far off what was being done before.
And a new manager comes along and we go around the whole cycle again.
I am of the opinion that agile is not a good methodology and should be avoided. However, I recently did some consultation for a small startup company and realized that they actually have no alternative. So little cashflow and very volatile investors.
So ultimately, the decision is between a terrible development methodology vs an abandoned project and potentially lost opportunity.
That said, often they result in the same thing. One just feeds developers for a little longer.
History lessons
If we're going to do potted history of methodologies, it's worth pointing out that one of the problems with Agile is that it delivered some significant early success to companies that read the manifesto and put some consideration into implementing the ideas (i.e. they actually thought about it).
Why is that success a problem? In the wake of a much needed shake-up to the industry, Agile went from a way to think about development to a buzzword, and on the back of that came a whole load of consultants and specialists who could (for a suitably large fee) teach your whole company how to do their special, branded version of Agile, complete with their special, subscription based Agile Tools (tm) that would give senior management Agile Metrics (tm), that they could use to improve productivity.
We've probably all sat through expert presentations (given by people who often failed out of software development) telling us how to do our jobs from the perspective of someone who genuinely hasn't got a clue what our company or business unit actually does. There will be graphics, special names to call parts of the process, a painfully complex workflow (but don't worry, you can buy.. erm.. subscribe to the software that will help with that) and more specialist teaching for people who's only job is to repeat the rituals, still without actually understanding what any of the technical staff they are shepherding around actually do. Does your 'Agile team' have more non-technical staff than actual code monkeys (you know, the ones who do the work)? Yeah... that.
So a good chunk of the fightback against Agile is not really about the core ideas and whether they work, but the way the term has been weaponised by an industry of grifters and rent-seekers, and (often willfully) misunderstood by senior management.
It always comes down to using the right tool for the job.
Sometimes Agile is the right development methodology for software. Sometimes it's the worst thing you can do to a software project.