UK Government Trial of M365 Copilot Finds No Clear Productivity Boost
- Reference: 0179011172
- News link: https://news.slashdot.org/story/25/09/04/2326232/uk-government-trial-of-m365-copilot-finds-no-clear-productivity-boost
- Source link:
> The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority of these allocated to volunteers and 30 percent to randomly selected participants. Some 300 of these people consented to their data being analyzed. An evaluation of time savings, quality assurance, and productivity was then calculated in [2]the assessment (PDF). Overall, 72 percent of users were satisfied or very satisfied with their digital assistant and voiced disappointment when the test ended. However, the reality of productivity gains was more nuanced than Microsoft's marketing materials might suggest. Around two-thirds of the employees in the trial used M365 at least once a week, and 30 percent used it at least once a day -- which doesn't sound like great value for money. [...]
>
> According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user. "Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day," the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as "very low," less than 1 percent and 3 percent per day, respectively. "PowerPoint and Excel were slightly more popular; both experienced peak activity of 7 percent of license holders using M365 Copilot in a single day within those applications," the study states. The three most popular tasks involved transcribing or summarizing a meeting, writing an email, and summarizing written comms. These also had the highest satisfaction levels, we're told.
>
> Participants were asked to record the time taken for each task with M365 Copilot compared to colleagues not involved in the trial. The assessment report adds: "Observed task sessions showed that M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users. Time savings observed for writing emails were extremely small. "However, M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users, conflicting time savings reported in the diary study for data analysis. PowerPoint slides [were] over 7 minutes faster on average, but to a worse quality and accuracy than non-users." This means corrective action was required.
>
> A cross-section of participants was asked questions in an interview -- qualitative findings -- and they claimed routine admin tasks could be carried out with greater efficiency with M365 Copilot, letting them "redirect time towards tasks seen as more strategic or of higher value, while others reported using these time savings to attend training sessions or take a lunchtime walk." Nevertheless, M365 Copilot did not necessarily make them more productive, the assessment found. This is something Microsoft has worked on with customers to quantify the benefits and justify the greater expense of a license for M365 Copilot.
[1] https://www.theregister.com/2025/09/04/m365_copilot_uk_government/
[2] https://assets.publishing.service.gov.uk/media/68adbe409e1cebdd2c96a19d/dbt-microsoft-365-copilot-evaluation.pdf
Re: (Score:1)
Wow, slightly disappointed that I got a flamebait mod point for what was essentially a joke about the questionable utility of any sort of government. This was absolutely not a jab at labour. History shows that all political parties are roughly as bad as each other, only in differing ways. Any good samaritans with a sense of humour who could mod this out of flamebait territory?
Re: (Score:3)
> Wow, slightly disappointed that I got a flamebait mod point for what was essentially a joke about the questionable utility of any sort of government.
The problem is such jokes have been done for years and they stopped being funny when we started electing governments, such as the Cameron and the Johnson governments, but even the Thatcher government, which treated those jokes as actual policy.
The British civil service has a bunch of problems. Massively conservative (small c), extremely establishment, very much from the public schools without experience of the lives of most of the country and so on. 85 IQs as someone below said and lack of individual produc
Re: (Score:2)
Well, thanks for taking the time reply to reply. Though I think quite the opposite is true. If you can't laugh about this stuff then it rapidly becomes very depressing. Which is why Have I Got News For You has been going strong for decades. Now I won't go as far as to say my joke was funny but flamebait it's not. But I'm effectively being lumped into the same category as this profanity spewing joker, who's not even British (either that or they haven't learned to spell arse correctly in British English).
Re: (Score:2)
> If you can't laugh about this stuff then it rapidly becomes very depressing.
Don't get me wrong, I'm not criticizing you for making the joke. "If you didn't laugh you'd have to cry" and all that. I'm just saying that now, when I think of people accusing government bureaucrats of doing nothing, what comes to mind is a baby dying of whooping cough because Robert Kennedy is a brain damaged idiot and the person who would have saved that baby has already been fired. Similarly with Ukrainians or UK troops dying because Farage managed to mess around the whole structure of Europe and Europe
Enskitification (Score:2, Interesting)
Microsoft peaked with Word 5 for Mac, nearly broke the company with version 6 and it has been downhill ever since. Pretty much the only useful feature in the last 30 years has been a multilingual spellchecker.
Oh and the major file format change just to flex muscle with Libreoffice developers that no one ever needed.
So yes, of course Powerpoint and Excel are going to see the best AI action because it is unusable without Googling half a dozen tutorials finding features hidden in stupid ribbons.
And yes I work
Re: (Score:3, Interesting)
I'd vote Word (and Excel) 4.x, but I appreciate your sentiment.
Re: (Score:3)
Enskitification -- when things become ridiculously comedic? Nice.
despite (Score:2)
> despite user satisfaction
Well we can't have that ! Shut it down!
Re: (Score:1)
Look, if the bureaucrat wastrel is happy talking to the robot, instead of getting in my way, let them have it. 0 productivity is better than negative productivity.
Wrong party measured? (Score:5, Interesting)
While the UK government might have no productivity boost, for Microsoft it has created:
Vendor lock-in
An endless stream of inside information about the UK government
An excuse to wreck the environment
Did the UK really think there was anything in it for them ?
Re:Wrong party measured? (Score:5, Interesting)
The UK government is already fully locked into Microsoft. They use it all... Windows, Sharepoint, Office, Active Directory, etc etc. Getting even one government department off MS would be a gargantuan task. Not to say it wouldn't be worth it, but it'll take a lot of work.
One thing I am curious about though is how the UK Government has 'squared the circle' of data sovereignty, now MS has said they can't guarantee it because US laws override local ones. What Copilot knows about the UK mist be extensive, say nothing of Sharepoint.
Sounds like a disaster. (Score:3)
As a direct test of the tool that sounds pretty underwhelming(and it's not a cheap upsell); but what seems really concerning is the second order effects. Your average office environment doesn't exactly lack for emails or bad powerpoint decks; and both get chiseled right out of the productivity of the people expected to read or sit through them. The more cynical sales types just go directly to selling you the inhuman centipede solution; where everyone else also needs a copilot license so they can summarize the increased volume of copilot-authored material; but that only bandaids the "if it's not worth writing why are you trying to write more of it?" problem.
You can never have too many meetings /s (Score:2)
You can never have too many meetings. Nothing useful can ever get done without having a meeting /s
Incentive (Score:3, Interesting)
As an employee, what is my incentive for increased productivity?
Are those saved hours just going to be used to give me more work?
Are some portion of those saved hours going to be used to reduce my work week without reducing my pay?
Are those saved hours going to be used as a basis for increasing my pay?
Are those saved hours, across multiple employees, going to be used to justify eliminating one or more employees?
How do I, the employee benefit?
Is there isn't a clear benefit to employees, why would they bother?
If the only benefit is "you get to keep your job", well, that's just working under the daily crack of the whip. Doesn't seem like a nice place to work.
Re:Incentive (Score:4, Insightful)
Sure, but idea all the economics-graduated idiots are salivating over is that their slaves get increased productivity from LLM use whether they want it or not.
What results like these show is simple: The current AI hype has burned all its straw and looks quite lacking after that. Hopefully it will end next year after a result. It could run longer though, because of all the morons that cannot admit how wrong they were.
Re: (Score:1)
That's why you increase efficiency under the table...
You do things to increase your efficiency, but don't tell anyone about them. Take less time to complete the same work, but submit the work on the same schedule and keep the difference for yourself.
Re: (Score:2)
Your questions are fair, but AI offers you the benefit of getting rid of crap tasks (e.g. writing minutes, editing transcripts) so you can put more time on tasks that require actual thoughts.
Re: (Score:2)
> As an employee, what is my incentive for increased productivity?
> Are those saved hours just going to be used to give me more work?
> Are some portion of those saved hours going to be used to reduce my work week without reducing my pay?
> Are those saved hours going to be used as a basis for increasing my pay?
> Are those saved hours, across multiple employees, going to be used to justify eliminating one or more employees?
> How do I, the employee benefit?
> Is there isn't a clear benefit to employees, why would they bother?
> If the only benefit is "you get to keep your job", well, that's just working under the daily crack of the whip. Doesn't seem like a nice place to work.
I'm glad you're not my employee.
Such a surprise (Score:2)
And with the lower quality, it is pretty clear that in the general office scenario, using LLMs ends up costing you money.
Re: (Score:2)
> And with the lower quality, it is pretty clear that in the general office scenario, using LLMs ends up costing you money.
Mistakes on Excel sheets have been responsible for billions of verified losses. This is also a great way to do plausibly deniable fraud. Just feed prompts into an offline AI model till you work out one which gives a subtly wrong answer in the benefit of your bonus.
productivity. UK? (Score:1)
productivity such a great word for analytical thinking robots. in the UK we have to remember that these are the same people that thought that the Japanese would never build a large motorcycle, and lost their entire motorcycle industry almost.
Re: (Score:3)
> productivity such a great word for analytical thinking robots. in the UK we have to remember that these are the same people that thought that the Japanese would never build a large motorcycle, and lost their entire motorcycle industry almost.
Oh you naive child. Less clear that with the Japanese and the motorcycle itself, but certainly by the time it came to cars, consumer and enterprise electronics, the rulers had worked out that if they got these things produced in the far east they could reduce the wages they had to pay to the people back home. Politically many of them discussed this as "inflation reduction" but practically they knew exactly what it meant. Moving people from high paying industrial jobs like welding into low paying service job
To summarise, AI is great at... (Score:5, Insightful)
Producing emails so you don't have to write them and digesting emails so you don't have to read them. It's as if the big lesson all along was that we could all learn to be more concise and respect each others time. Now about those meetings...
Or that a lot of people don't really need to exist (Score:2)
There are tons and tons of people that exist to summarize data and move it up to chain to the billionaire elites so that they can make sure they're a little empires are not collapsing. The closer you get to one of those billionaires the more you get compensated in order to encourage competence and obedience.
A lot of us are part of that chain and AI is very desirable to those billionaire elites because they are aware of their dependency on that chain and they aren't happy about it.
Copilot is not a good product (Score:2)
At all.
Because MS limits tokens and caches noise
Apart from that, the number of people with REAL jobs don't need LLMs to interject and assist. Marketing department loves LLMs because they're not particularly bright or creative, mostly employing 3rd parties to do the heavy lifting and now AI to do the rest.....
This is the bit the LLM cult is missing. If I already know how to do everything I need to do, why in the everloving fark would I slow myself down by roping in a sub-optimal Copilot product to take a s
Makes complex tasks worse (Score:2)
Copilot became available at the time I was doing my taxes, so I gave it a try. Nothing more complex than locating certain transactions in an excel sheet and a bit of math with the group. The thought... it should not be this hard came to mind as we crawled down the rabbit hole. Was grateful when I found there was an obscure way to turn it off and refuse the extra cost 'upgrade'. I can see that management is rubbing their hands over the thought of all the extra licensing fees. But from a simple productivity p
[Stares In Believability] (Score:2)
Ya don't say!
This actually seems very promising (Score:4, Interesting)
1) it did increase productivity for some tasks
2) people were only using it once a day
I suspect that with a little experience people will better know which tasks it works for and which it doesn't, while also using it more often.
For a brand new tool to show real potential in some tasks while also being currently neutral overall seems promising in the long run.
Re: (Score:2)
I agree. The summary evens mentions staff having time to attend training sessions or take a walk. These are real benefits being delivered even with a product that noone would argue is at the cutting edge of current AI. As you mention, with time and experience people will find ways to maximize these benefits even as the product itself improves.
Middle Man Analogy (Score:3)
Have you ever been in a situation when your team has a middle manager and that manager is responsible for facilitating work with an external team?
9 times out of 10, the outcome of such a scheme is poor because the manager overrides your instructions and suggestions, and filters out what's important when communicating with external parties.
This is exactly what role AI assistants play these days. They try to position themselves as the middleman between the person who wants shit done and external team who does it.
And just like with out middle manager, lots of critical detail is lost in translation, and the result is at best mediocre.
Oh, and just like with a real world middle manager, you pay them a lot of money for providing no real value whatsoever.
Emails and meetings (Score:2)
The finding that little time is saved with AI summaries of emails and meetings should be further analyzed.
Meetings take a huge number of hours every week for almost everyone. A lot of my meetings are online zoom-like meetings where information is disseminated and where I seldom need to say anything. Many of these meetings are effectively seminars. Having AI summarize these meetings so that I can do something else that is more productive would be immensely useful and would significantly improve efficiency an
So not going to lead to mass layoffs then (Score:2)
If AI only increases productivity incrementally (and this does match my own experience) it seems unlikely that AI will lead to massive layoffs, or starving workers. Maybe, just maybe, the hype was greater than the reality on the ground? Wait, the advertising oversold what the technology could deliver? Who would have thought!
Of course they couldn't (Score:2)
If you do nothing, like most government employees and civil servants, AI won't increase productivity....
As expected (Score:2)
AI tools can be useful for some things, but they are new and immature
Expecting "productivity gains" from early use of immature tech is silly
Also, copilot is not a very good AI
The emperor has no clothes. (Score:2)
The truth about Copilot - and AI in general - is starting to become obvious.
AI good for known tasks (Score:3)
And is bad for unknown tasks. I ask a bot for how to say setup a cloudformation template or other skeleton tasks, but actually coding my domain knowledge, I have to do. It simply can't do what it isn't trained on, i.e. new stuff not found on the Internet.
Re: (Score:2)
So about as good as LSD25
Re: (Score:3)
> It simply can't do what it isn't trained on, i.e. new stuff not found on the Internet.
This, I assume, is true, however I've also found it's really bad at things that are out there on the internet but are "obscure"
I was working on an asterisk dialplan, this is a very simple home setup, and chatgpt was "not helpful". It told me things that are actually official examples are wrong while not finding things that are out there.
OTOH, it probably didn't cost me any time either, first I tried chatgpt, worked throug
Re: (Score:3)
>> It simply can't do what it isn't trained on, i.e. new stuff not found on the Internet.
> This, I assume, is true, however I've also found it's really bad at things that are out there on the internet but are "obscure"
> I was working on an asterisk dialplan, this is a very simple home setup, and chatgpt was "not helpful". It told me things that are actually official examples are wrong while not finding things that are out there.
> OTOH, it probably didn't cost me any time either, first I tried chatgpt, worked through its suggestions, which didn't work, and then once I finally gave up and went back to duckduckgo, what I'd learned improved my searching and filtering and it didnt take me long to find the thread that actually had the information I needed (It didn't have an answer to my question but it was sufficiently closely related that I could see how to solve my issue).
DDG ai answers have been pretty good, at least the times I have used it. But that's not programming.
I find one use for it. Now that old school internet searches give either dozens of almost identical but not wanted results, or many commercial results. DDG ai answers don't seem to fixate on that.
Example - I vaguely remembered an accident, probably 6 or more years ago. that a man drove off a bridge after GPS told him to take that route. drove around barricades to drive off the bridge. I couldn't get any
Re: (Score:3)
> So I tried DDG ai, and got the answer in a second. I verified it manually, and now use it regularly.
Do you continue to check it regularly? I assume that you get "good enough" for you answers and just continue to use them unless it's either something that gives a doubt or something that is critical to you. In the context of a company or a government, where one person relies on the answers from another person that maybe aren't the core of the other person's expertise.
A while ago in another AI discussion on Slashdot I looked up bridge foundations and got a subtly wrong answer. The AI gave a simple, confident
Re: (Score:2)
It sounds like what they have done is an excellent shot across the bows of the LLM companies and lesson for the workers. "Use the LLMs for the side issues of your jobs, cutting down on tasks like reading ministerial directives which are repeated all over but which you aren't special at. Don't use LLMs for the actual heart of your job and the expert calculations that you do".
That makes it clear that with the current generation of LLMs they should be paying much much less, using it more selectively and teachi
Re:What is "productivity" in this context? (Score:4, Insightful)
> What exactly is being measured? How do you measure "productivity" in an organization that doesn't actually produce anything?
You make things up to suit whatever your pre existing motives were.
Re:AI good for known tasks (Score:5, Interesting)
It cannot get even simple things correct. I wanted to know what kind of a machine to which a serial number for a Mac belonged. Apple has a web site for this but I couldn't recall it. So googled Mac + serial number. First item up was googles ai-thingy's response. It told me something I knew was incorrect, but it did so very authoritatively. Further down is Apple's site. I typed in the serial number there and it got the machine correct. Never trust google's ai-thingy....bad ju-ju.
Re: (Score:3)
Pretty much this... You give a tool to a large bunch of people most of whom are neither doing things the tool is suited for, nor are they experienced in usage of the tool then you're going to conclude it's a bad tool.
If you want good results buy people the right tools for their job, and give them training on how to use the tools if they're not previously familiar with them.
Re:AI good for known tasks (Score:4, Interesting)
> And is bad for unknown tasks. I ask a bot for how to say setup a cloudformation template or other skeleton tasks, but actually coding my domain knowledge, I have to do. It simply can't do what it isn't trained on, i.e. new stuff not found on the Internet.
In that case it should be good for government departments and produce measurable productivity gains.
I don't believe that nonsense about government workers being lazy or Jobsworths. In my experience bureaucracy is a function of size, not ownership. I encountered more Jobsworths working for a mining giant than I ever met in the public service. However I digress, something that is more common in the public sector are repetitive and rule based work. If Application X meets conditions A, B, C and D or E... then approve, a whole lot of if, then, else and repeat.
Another problem is that the UK government is already pretty damned efficient. I got my first British passport last year (after gaining citizenship) and it too all of 10 days which included having to mail off various documents proving I was now a citizen and 3 of those days were bank holidays. Being a migrant meant I've dealt with more UK officialdom than the average Briton and in a much shorter space of time, a lifetime of things you do with the government only once has been squashed into the last 5 or 10 years. None of it was particularly difficult or onerous (except for having the misfortune to apply for settlement at the same time as Brexit). Certainly it's less painful than Australia and I've been reliably informed that Australia's worst government department isn't a patch on the Wicked Witch from the DMV. I suspect the UK is already operating at a very high level of automation.
Re: (Score:2)
Do you understand how voting in > 2 party systems works? What would you suggest, the runner up with even fewer votes forms a government?
Re: (Score:2)
Ah, found the guy who doesn't understand how instant-runoff voting works. They got 55.22% of the final vote.
Re: (Score:2)
"I don't believe that nonsense about government workers being lazy or Jobsworth"
Some are, some arn't.
"Another problem is that the UK government is already pretty damned efficient"
Says someone who's only been here a few years. Wait until you have to use the NHS for routine non-cancer conditions , or try and sort out a tax error with HMRC, or if you're a farmer try to get anything useful from Defra.
"Being a migrant meant I've dealt with more UK officialdom than the average Briton"
Don't kid yourself mate, you
Re: (Score:2)
" In my experience bureaucracy is a function of size, not ownership"
I've been in a few large corporations, latest one is the biggest. The bureaucracy statement is true from the 30+ years I've been working,
I've been subcontracted out a fair few times to government departments, and the only things done worse than private companies is the extremely understaffed departments and the sheer lack of money to improve anything.
Since the whole MBA fascination with profitable next quarter at any cost, that is the
Re: AI good for known tasks (Score:2)
Bureaucracy arise naturally in any organisation. The only thing that can stop is competition. (Among states: War) Private monopolies are at least as bad as government. That is why breaking up a monopoly will actually help the stock owners because the competing companies getting out of it will be much more lean.
Re:AI good for known tasks (Score:4, Interesting)
Yep. While many clueless ascribe "insight" or even "agency" to LLMs, what they really can do is just better search.
Re: AI good for known tasks (Score:3)
Not entirely. It's pretty good at "I have a bunch of headers for a library I'm not familiar with, do X." I could figure out X but it would take a couple of hours. If you drop those headers into the context, it can usually give you a pretty good first draft.
Re: (Score:2)
> It simply can't do what it isn't trained on, i.e. new stuff not found on the Internet.
So, train it on more?
A colleague of mine trained a custom GPT on a bunch of our internal documentation and code. I asked it a question about which of our internal APIs to use for a particular scenario and (after 2-3 minutes of burning presumably an ungodly amount of electricity) it correctly told me that such an API did not exist, but gave me the ones that do exist including a code snippet showing how they could be used.
Saved me at least 10-15 minutes of combing through libraries and a few minutes writi