Tool touted as 'first AI software engineer' is bad at its job, testers claim
- Reference: 1737620231
- News link: https://www.theregister.co.uk/2025/01/23/ai_developer_devin_poor_reviews/
- Source link:
The auto-coder is called “Devin” and was [1]introduced in March 2024. The bot’s creator, an outfit called Cognition AI, has made claims such as “Devin can build and deploy apps end to end," and "can autonomously find and fix bugs in codebases." The tool [2]reached general availability in December 2024, starting at $500 per month.
"Devin is an autonomous AI software engineer that can write, run and test code, helping software engineers work on personal tasks or their team projects," Cognition's [3]documentation declares. It "can review PRs, support code migrations, respond to on-call issues, build web applications, and even perform personal assistant tasks like ordering your lunch on DoorDash so you can stay locked in on your codebase."
[4]
The service uses Slack as its main interface for commands, which are sent to its computing environment, a Docker container that hosts a terminal, browser, code editor, and planner. The AI agent supports API integration with external services. This allows it, for example, to send email messages on a user's behalf via SendGrid.
[5]
[6]
Devin is a " [7]compound AI system ," meaning it relies on multiple underlying AI models, a set that has included OpenAI's GPT-4o and can be expected to evolve over time.
In theory, you should be able to ask it to undertake tasks like migrating code to [8]nbdev , a Jupyter Notebook development platform, and expect it to do so successfully. But that may be asking too much.
[9]
Early assessments of Devin have found problems. Cognition AI posted a [10]promo video that supposedly showed the AI coder autonomously completing projects on the freelancer-for-hire platform Upwork. Software developer [11]Carl Brown analyzed that vid and debunked it on his [12]Internet of Bugs YouTube channel .
The software agent was also called out by another YouTube code pundit for allegedly including [13]critical security issues .
Now, three data scientists affiliated with [14]Answer.AI , an AI research and development lab founded by Jeremy Howard and Eric Ries, have [15]tested Devin and found it completed just three out of 20 tasks successfully.
[16]Analysts say real datacenter emissions are a dirty secret
[17]Business value from GenAI remains elusive despite IT spending boom
[18]Google DeepMind CEO says 2025's the year we start popping pills AI helped invent
[19]OpenAI to blow through $500B on AI infrastructure for itself with help from pals
In an [20]analysis conducted earlier this month by [21]Hamel Husain , [22]Isaac Flath , and [23]Johno Whitaker , Devin started well, successfully pulling data from a Notion database into Google Sheets. The AI agent also managed to create a planet tracker for checking claims about the historical positions of Jupiter and Saturn.
But as the three researchers continued their testing, they encountered problems.
[24]
"Tasks that seemed straightforward often took days rather than hours, with Devin getting stuck in technical dead-ends or producing overly complex, unusable solutions," the researchers explain in their report. "Even more concerning was Devin’s tendency to press forward with tasks that weren’t actually possible."
As an example, they cited how Devin, when asked to deploy multiple applications to the infrastructure deployment platform [25]Railway , failed to understand this wasn't supported and spent more than a day trying approaches that didn't work and hallucinating non-existent features.
Of [26]20 tasks presented to Devin , the AI software engineer completed just three of them satisfactorily – the two cited above and a third challenge to research how to build a Discord bot in Python. Three other tasks produced inconclusive results, and 14 projects were outright failures.
The researchers said that Devin provided a polished user experience that was impressive when it worked.
"But that’s the problem – it rarely worked," they wrote.
"More concerning was our inability to predict which tasks would succeed. Even tasks similar to our early wins would fail in complex, time-consuming ways. The autonomous nature that seemed promising became a liability – Devin would spend days pursuing impossible solutions rather than recognizing fundamental blockers."
Cognition AI did not respond to a request for comment. ®
Get our [27]Tech Resources
[1] https://www.cognition.ai/blog/introducing-devin
[2] https://www.cognition.ai/blog/devin-generally-available
[3] https://docs.devin.ai/get-started/devin-intro
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z5IhStJudNbAEDmQc2wWAgAAAAg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z5IhStJudNbAEDmQc2wWAgAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z5IhStJudNbAEDmQc2wWAgAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://cognition.ai/blog/evaluating-coding-agents
[8] https://nbdev.fast.ai/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z5IhStJudNbAEDmQc2wWAgAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.youtube.com/watch?v=UTS2Hz96HYQ
[11] https://github.com/carlbrown
[12] https://www.youtube.com/watch?v=tNmgmwEtoWE
[13] https://www.youtube.com/watch?v=927W6zzvV-c
[14] https://answer.ai
[15] https://devin.ai/
[16] https://www.theregister.com/2025/01/22/datacenter_emissions_not_accurate/
[17] https://www.theregister.com/2025/01/22/business_value_genai_elusive/
[18] https://www.theregister.com/2025/01/22/google_deepmind_ai_drugs/
[19] https://www.theregister.com/2025/01/22/openai_stargate_ai_datacenter_company/
[20] https://www.answer.ai/posts/2025-01-08-devin.html#what-is-devin
[21] https://hamel.dev/
[22] https://isaac-flath.github.io/website/blog.html
[23] https://johnowhitaker.dev/
[24] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z5IhStJudNbAEDmQc2wWAgAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[25] https://railway.com/
[26] https://www.answer.ai/posts/2025-01-08-devin.html#appendix-tasks-attempted-with-devin
[27] https://whitepapers.theregister.com/
The Bastard AI Developer from Hell has landed.
Stop the AI Marketing spin
"As an example, they cited how Devin, when asked to deploy multiple applications to the infrastructure deployment platform Railway, failed to understand this wasn't supported and spent more than a day trying approaches that didn't work and hallucinating non-existent features."
They do not hallucinate, they output an error. They do not understand, there is no intelligence, they misinterprete the command (prompt). So the "AI" failed to interpret the user command correctly and continued running which produced errors in the output. Don't let marketing win.
Re: Stop the AI Marketing spin
Couldn't agree more.
Stop naming everything AI.
It's artificial alright, but it's all lacking intelligence at the moment.
Seriously, some of this shite is being pitched as if we've managed to create sentient, autonomous beings..... We really haven't!
Re: Stop the AI Marketing spin
at the moment?
Re: Stop the AI Marketing spin
Look on "hallucination" as a useful way to describe a specific error mode. Or do you complain about "buffer overrun" on the basis that it's got nothing to do with railways? Or about BSOD on the basis that only living things die?
Tasks that seemed straightforward often took days rather than hours, with Devin getting stuck in technical dead-ends or producing overly complex, unusable solutions
Sounds like a good fit for large Government IT projects. Perhaps Devin should be renamed Capita.
Obviously
If each one is 15% effective, I only need to enable 7 instances and I'm already at 105% of a wage drawing human.
"Cognition AI did not respond to a request for comment.'
With shitty results like that, no wonder.
Travelling Circus
« "Cognition AI did not respond to a request for comment." With shitty results like that, no wonder.»
A quick look at the genealogy of Cognition AI from Wiki "Originally the company was focused on cryptocurrency before moving to AI as it became a trend in Silicon Valley following the release of ChatGPT" suggests the outfit is a travelling circus.
Once the rubes who have been paying USD500/month realise they have duped by a lightly warmed over collation of other software and services that doesn't really work, the tent will come down and the clowns will move on to the next big thing.
Sure they did. They just assigned the task of responding to Devin.
"Devin’s tendency to press forward with tasks that weren’t actually possible."
IOW's it's hallucinating.
Just like every other LLM driven system.
"The researchers said that Devin provided a polished user experience that was impressive when it worked."
I think we know where the "Developers*" of this spent most of their cash. Maybe a "That won't work, here are additional instructions" button might be useful about now?
People have been trying this since at least the "Programmers Apprentice" project out of Stanford in the 80's. It was subsequently moved to Mitsubishi Research Labs where it might have been quite useful for their internal development. It did not use LLM but was built on hard reasoning and inference systems, back with something called the "Plan Calculus" to analyse programs and identify the clusters of code changes needed when you wanted to change the function of a module of code.
*TBH it sounds like it was stitched together from a bunch of other stuff.
Re: "Devin’s tendency to press forward with tasks that weren’t actually possible."
> TBH it sounds like it was stitched together from a bunch of other stuff.
Testing for bolt fastening head to neck - - Check!
Re: "Devin’s tendency to press forward with tasks that weren’t actually possible."
Let hand thread, oh dear.
The problem with current technology is that it looks impressive, but when it comes to the details...
It is officially part of my job to experiment with AI and find ways it can be useful. This includes both trying to integrate it in to our own custom software and also finding uses for our users.
We are rolling out copilot to users (limited at the moment due to cost, but users are asking to be included in the "trial"), and some of them are finding it useful. Summarising documents and generating boilerplate are the sorts of tasks it does well.
Integrating an LLM in to our own software has however been far less successful, as it basically requires you to implement every piece of functionality you want manually, in a non-deterministic "language". You can waste a lot of time tweaking it to be able to answer one question 99% of the time only to find it now gets confused about a different question that was working before.
As for code assistants... I left copilot installed in my IDE to see if it eventually became useful. Yesterday I spent 20 minutes trying to figure out where the Syntax error was in a simple 2 line function it output. It was very subtle. It sometimes gets it close enough that editing the code is quicker than typing it myself, but a lot of the time I end up wasting more time trying to fix it than it would have taken to write it correctly.
Summarising documents and generating boilerplate are the sorts of tasks it does well.
No, they aren't.
[1]When ChatGPT summarises, it actually does nothing of the kind.
[2]AI worse than humans in every way at summarising information, government trial finds
"Reviewers told the report’s authors that AI summaries often missed emphasis, nuance and context; included incorrect information or missed relevant information; and sometimes focused on auxiliary points or introduced irrelevant information."
[1] https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actually-does-nothing-of-the-kind/
[2] https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/?utm_source=pocket_shared
Re: Summarising documents and generating boilerplate are the sorts of tasks it does well.
I guess it depends what you mean by a summary. Most of the time they just want a list of key information from a document, or for it to rewrite meeting notes in to something with sentences, rather than expecting an insightful analysis.
But the key issue seems to be what people selling the things claim they can do compared to the reality. This is why you need to manage how you roll out such tools to users to ensure they actually pay attention to the output and do their own analysis of how useful it is, instead of assuming this new tool is magic.
"rather than recognizing fundamental blockers"
Obviously. It doesn't "recognize" anything. It's a statistical analysis machine put to very exacting use.
To recognize a fundamental blocker, you need experience and intelligence. This pseudo-AI has neither.
Cognition AI has set itself a tall task, and the market is not going to stand for a 15% success rate.
Maybe Devin is
working so badly because its codebase was written by... Devin.
Re: Maybe Devin is
We're Devin', (Devin', Devin', Devin') . . .
Can't get that out of my head now :(
Re: Maybe Devin is
... round 'em up, rawhide!
Not AI.
""Tasks that seemed straightforward often took days rather than hours, with Devin getting stuck in technical dead-ends or producing overly complex, unusable solutions," the researchers explain in their report. "Even more concerning was Devin’s tendency to press forward with tasks that weren’t actually possible."
As an example, they cited how Devin, when asked to deploy multiple applications to the infrastructure deployment platform Railway, failed to understand this wasn't supported and spent more than a day trying approaches that didn't work and hallucinating non-existent features."
So you mean that as soon as it actually required intelligence and inference, it wasn't able to do things?