UK government trial of M365 Copilot finds no clear productivity boost
(2025/09/04)
- Reference: 1756977314
- News link: https://www.theregister.co.uk/2025/09/04/m365_copilot_uk_government/
- Source link:
A UK government department's three-month trial of Microsoft's M365 Copilot has revealed no discernible gain in productivity – speeding up some tasks yet making others slower due to lower quality outputs.
The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority of these allocated to volunteers and 30 percent to randomly selected participants. Some 300 of these people consented to their data being analyzed.
Microsoft promises Copilot will be a 'moneymaker' in the long term [1]READ MORE
An evaluation of time savings, quality assurance, and productivity was then calculated in the [2]assessment .
Overall, 72 percent of users were satisfied or very satisfied with their digital assistant and voiced disappointment when the test ended. However, the reality of productivity gains was more nuanced than Microsoft's marketing materials might suggest.
Around two-thirds of the employees in the trial used M365 at least once a week, and 30 percent used it at least once a day – which doesn't sound like great value for money.
[3]
In the UK, commercial prices range from £4.90 per user per month to £18.10, depending on business plan. This means that across a government department, those expenses could quickly mount up.
[4]
[5]
According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user.
"Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day," the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as "very low," less than 1 percent and 3 percent per day, respectively.
[6]
"PowerPoint and Excel were slightly more popular; both experienced peak activity of 7 percent of license holders using M365 Copilot in a single day within those applications," the study states.
The three most popular tasks involved transcribing or summarizing a meeting, writing an email, and summarizing written comms. These also had the highest satisfaction levels, we're told.
Participants were asked to record the time taken for each task with M365 Copilot compared to colleagues not involved in the trial. The assessment report adds: "Observed task sessions showed that M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users. Time savings observed for writing emails were extremely small.
[7]
"However, M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users, conflicting time savings reported in the diary study for data analysis.
"PowerPoint slides [were] over 7 minutes faster on average, but to a worse quality and accuracy than non-users." This means corrective action was required.
[8]Barclays Bank signs 100K license Copilot deal with Microsoft
[9]Microsoft brings 365 suite on-prem as part of sovereign cloud push
[10]Microsoft crams Copilot AI directly into Excel cells
[11]Are you willing to pay $100K a year per developer on AI?
A cross-section of participants was asked questions in an interview – qualitative findings – and they claimed routine admin tasks could be carried out with greater efficiency with M365 Copilot, letting them "redirect time towards tasks seen as more strategic or of higher value, while others reported using these time savings to attend training sessions or take a lunchtime walk."
Nevertheless, M365 Copilot did not necessarily make them more productive, the assessment found. This is something Microsoft has worked on with customers to [12]quantify the benefits and justify the greater expense of a license for M365 Copilot.
"We did not find robust evidence to suggest that time savings are leading to improved productivity," the report says. "However, this was not a key aim of the evaluation and therefore limited data was collected to identify if time savings have led to productivity gains."
Microsoft can't guarantee data sovereignty – OVHcloud says 'We told you so' [13]READ MORE
And hallucinations? 22 percent of the Department for Business and Trade guinea pigs that responded to the assessors said they did identify hallucinations, 43 percent did not, and 11 percent were unsure.
Users reported mixed experiences with colleagues' attitudes, with some teams embracing their AI-augmented workers while others turned decidedly frosty. Line managers' views appeared to significantly influence adoption rates, proving that office politics remain refreshingly human.
The department is still crunching numbers on environmental costs and value for money, suggesting the full reckoning of AI's corporate invasion remains some way off. An MIT survey published last month, for example, found that 95 percent of companies that had collectively sunk $35-40 billion into generative AI had little to show for it.
For now, it seems M365 Copilot excels at the mundane while stumbling over the complex – an apt summary of GenAI in 2024. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2024/03/18/microsoft_copilot_moneymaker/
[2] https://assets.publishing.service.gov.uk/media/68adbe409e1cebdd2c96a19d/dbt-microsoft-365-copilot-evaluation.pdf
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2025/05/30/barclays_bank_sign_100k_license/
[9] https://www.theregister.com/2025/06/17/microsoft_365_on_prem_azure_local/
[10] https://www.theregister.com/2025/08/18/microsoft_adds_copilot_ai_formulas/
[11] https://www.theregister.com/2025/08/15/are_you_willing_to_pay/
[12] https://www.theregister.com/2024/03/18/microsoft_copilot_moneymaker/
[13] https://www.theregister.com/2025/08/27/ovhcloud_interview/
[14] https://whitepapers.theregister.com/
The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority of these allocated to volunteers and 30 percent to randomly selected participants. Some 300 of these people consented to their data being analyzed.
Microsoft promises Copilot will be a 'moneymaker' in the long term [1]READ MORE
An evaluation of time savings, quality assurance, and productivity was then calculated in the [2]assessment .
Overall, 72 percent of users were satisfied or very satisfied with their digital assistant and voiced disappointment when the test ended. However, the reality of productivity gains was more nuanced than Microsoft's marketing materials might suggest.
Around two-thirds of the employees in the trial used M365 at least once a week, and 30 percent used it at least once a day – which doesn't sound like great value for money.
[3]
In the UK, commercial prices range from £4.90 per user per month to £18.10, depending on business plan. This means that across a government department, those expenses could quickly mount up.
[4]
[5]
According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user.
"Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day," the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as "very low," less than 1 percent and 3 percent per day, respectively.
[6]
"PowerPoint and Excel were slightly more popular; both experienced peak activity of 7 percent of license holders using M365 Copilot in a single day within those applications," the study states.
The three most popular tasks involved transcribing or summarizing a meeting, writing an email, and summarizing written comms. These also had the highest satisfaction levels, we're told.
Participants were asked to record the time taken for each task with M365 Copilot compared to colleagues not involved in the trial. The assessment report adds: "Observed task sessions showed that M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users. Time savings observed for writing emails were extremely small.
[7]
"However, M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users, conflicting time savings reported in the diary study for data analysis.
"PowerPoint slides [were] over 7 minutes faster on average, but to a worse quality and accuracy than non-users." This means corrective action was required.
[8]Barclays Bank signs 100K license Copilot deal with Microsoft
[9]Microsoft brings 365 suite on-prem as part of sovereign cloud push
[10]Microsoft crams Copilot AI directly into Excel cells
[11]Are you willing to pay $100K a year per developer on AI?
A cross-section of participants was asked questions in an interview – qualitative findings – and they claimed routine admin tasks could be carried out with greater efficiency with M365 Copilot, letting them "redirect time towards tasks seen as more strategic or of higher value, while others reported using these time savings to attend training sessions or take a lunchtime walk."
Nevertheless, M365 Copilot did not necessarily make them more productive, the assessment found. This is something Microsoft has worked on with customers to [12]quantify the benefits and justify the greater expense of a license for M365 Copilot.
"We did not find robust evidence to suggest that time savings are leading to improved productivity," the report says. "However, this was not a key aim of the evaluation and therefore limited data was collected to identify if time savings have led to productivity gains."
Microsoft can't guarantee data sovereignty – OVHcloud says 'We told you so' [13]READ MORE
And hallucinations? 22 percent of the Department for Business and Trade guinea pigs that responded to the assessors said they did identify hallucinations, 43 percent did not, and 11 percent were unsure.
Users reported mixed experiences with colleagues' attitudes, with some teams embracing their AI-augmented workers while others turned decidedly frosty. Line managers' views appeared to significantly influence adoption rates, proving that office politics remain refreshingly human.
The department is still crunching numbers on environmental costs and value for money, suggesting the full reckoning of AI's corporate invasion remains some way off. An MIT survey published last month, for example, found that 95 percent of companies that had collectively sunk $35-40 billion into generative AI had little to show for it.
For now, it seems M365 Copilot excels at the mundane while stumbling over the complex – an apt summary of GenAI in 2024. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2024/03/18/microsoft_copilot_moneymaker/
[2] https://assets.publishing.service.gov.uk/media/68adbe409e1cebdd2c96a19d/dbt-microsoft-365-copilot-evaluation.pdf
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLljN91TEqysJS9x_ev1EAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2025/05/30/barclays_bank_sign_100k_license/
[9] https://www.theregister.com/2025/06/17/microsoft_365_on_prem_azure_local/
[10] https://www.theregister.com/2025/08/18/microsoft_adds_copilot_ai_formulas/
[11] https://www.theregister.com/2025/08/15/are_you_willing_to_pay/
[12] https://www.theregister.com/2024/03/18/microsoft_copilot_moneymaker/
[13] https://www.theregister.com/2025/08/27/ovhcloud_interview/
[14] https://whitepapers.theregister.com/
Crapilot
elsergiovolador
So the UK government paid Microsoft to learn that Copilot can spit out email drafts and meeting summaries but collapses on anything complex. And how exactly is an AI supposed to know the nuance of a call, or what was actually important, without losing it in a bland summary? Billions go into this hype, and the only thing Copilot does consistently is generate invoices - not productivity.
"Copilot excels at the mundane while stumbling over the complex"
Pascal Monett
And yet it is still marketed as "AI".
If there was any intelligence in there, it would help with the complex stuff and tell you to write your own mails.
22% identified hallucinations...
abend0c4
... and the rest were either unsure or didn't.
Unfortunately, that doesn't mean they weren't there.
This sort of survey is not exactly rigorous since there's no objective assessment of accuracy or the genuine equivalence of tasks undertaken by the two groups. Yet the productivity differences still seem to amount to little more than noise - with the added ingredient of potentially undetected hallucination.
It's not exactly a convincing value proposition.
Telling that users were disappointed when the play date with their mechanistic chums was over - perhaps all they need to do is pair up the staff working on these various tasks so they get some genuine personal interaction.
Productivity goes down
Trying to remove or disable the bugger
Ignoring it with its suggestions
Or in teams, if you unpin / uninstall, the constant nagging to putit back on