Don't panic: Your AI co-worker is here to help
(2024/06/04)
- Reference: 1717513271
- News link: https://www.theregister.co.uk/2024/06/04/genai_coworker/
- Source link:
Jake Heller, head of product for CoCounsel – an AI bot from multinational information conglomerate Thomson Reuters – has a tip for anyone selling AI software: Don't sell software.
Rather, he argues, sell AI as a co-worker. Corporations understand the cost of people, but not code, he reasons – and it has worked for him at least.
As he recounted at the [1]GenAI Summit 2024 in San Francisco last week, he was one of the founders of CaseText about a decade ago. When the startup began, it offered a crowdsourced legal database for legal research.
[2]
Then about two years ago, he was among those granted early access to OpenAI's GPT-4 and used it to develop CoCounsel, an AI legal assistance tool that debuted [3]on March 1 last year , two weeks before GPT-4 made a limited public debut. A few months later, CaseText was [4]acquired for $650 million by Thomson Reuters.
[5]
[6]
Heller argues that "building AI co-workers is going to be the largest opportunity of tech in our lifetime, measured by impact and measured by the amount of revenue, amount of money you can make building these kinds of technologies."
[7]
Jake Heller, head of product at CoCounsel, sees sunny days ahead – Click to enlarge
"When a lot of people hear 'AI co-workers,' especially the way I'll be talking about it a little later, it gets a little scary," he admitted. "What jobs will there be left for humans? I think it's actually bright rather than dim, for reasons that are a little bit different than a lot of the reasons people talk about."
When people talk about generative AI, said Heller, they often focus on the term "generative" – but he argues that it's the "intelligence" aspect that deserves attention. In the past two or three years, AI has got a lot more "intelligent" in ways that really matter, argues Heller, and it can take on real-world work.
Even two years ago, he said, AI models were reportedly passing bar exams – and doing so better than 70 and 90 percent of the people taking the tests. The AI [8]did better than 80 percent of those taking the Graduate Record Examinations (GRE), he asserted.
[9]
And this is old news, according to Heller. He expects that the technology as it exists today will lead to the emergence of AI co-workers – and he points to CoCounsel as an example of the genre.
"It can accomplish legal tasks at about a human level, if not a bit better, but at superhuman speeds," he claims. "Things that would take me, when I was a lawyer, many, many days to complete, it can do in 90 seconds."
For example, CoCounsel has proved itself capable when asked to find evidence of fraud in a large set of email messages – a common but boring task for lawyers. CoCounsel's results are good enough that in just a year and half on the market, 60 percent of elite law firms – the AM Law 100 – have adopted it. It's done 1.5 million tasks for attorneys – such as reviewing documents, reviewing contracts, and other research-oriented tasks, Heller claimed.
[10]
"A lot of people look at GenAI, and they think it's maybe hype, that there are all these kind of cool demos, but not real applications" observed Heller. "There are, in fact, real applications, real revenue for real customers, and real benefits."
[11]Microsoft to spend $3.2B on expanding cloud and AI in green energy-rich Sweden
[12]Google finally addresses those bizarre AI search results
[13]Surging datacenter power demand slows the demise of US coal plants
[14]IBM spin-off Kyndryl accused of discriminating on basis of age, race, disability
To build CoCounsel, the CaseText team defined [15]a set of skills that a junior lawyer might need – such as summarizing documents, editing a contract under guidance from instructors, or reviewing a contract for corporate compliance. They broke those skills down into steps, developed prompts, and worked out how to combine those skills in different ways.
The basic formula is simple, Heller noted, but the challenge is ensuring correctness.
"The difficulty really is making sure you should get the right answers," he said. "And I think part of the secret to how we became widely adopted in legal – and I think that will be true for many of the different professions and industries that you all serve – is an extremely robust testing methodology. Literally we've run over 600,000 tests, some of them against these individual prompts … and sometimes end-to-end, like if a request goes in and answer comes out, did [the AI model] hallucinate?"
Heller emphasized that only when his team had confidence in the correctness of CoCounsel's responses did they feel comfortable giving them to lawyers – whose jobs are on the line if errors occur. And one way the organization helped reassure customers is by providing sources they can read with a single click, to confirm what the AI model returns.
"The goal is not to take the lawyer's job away from them, but to assist them in all the tasks that they're doing – especially the stuff that takes a long time, but they and their customers get the least value out of," said Heller.
CoCounsel "was never pitched as software. We pitched it as a co-worker. The value isn't, 'this will make certain workflows a little more efficient, a little more effective for you.' The value is, 'now you have an additional member of your team.' And that lets you price in a way that follows value."
Heller went on to argue that he doesn't believe that AI co-workers will end up taking jobs away from people. "I think they will be upleveled to do more interesting stuff," he predicted.
That's been his experience with CoCounsel, having sold it to thousands of customers. "The number of layoffs that we've heard about as a result of the mass adoption of this technology in the legal industry is zero," he declared. Instead, it lets law firms say 'yes' to cases they'd have previously declined for lack of capacity.
That said, Heller acknowledged that what's economically beneficial in aggregate may not be positive for every person.
"Will this be net positive for every single person in society? Probably not," he conceded. "Even if it's truly net positive in the aggregate, I think some individuals in some job categories and some tasks are gonna be harder hit than others and faster than others. And so, we're going to rise to the challenge of massive potential for displacement, shifts, et cetera."
To address that, according to Heller, we need appropriate policies. "Does America, for example, have the capacity to react appropriately to this new technology?" he mused.
"I see a lot of shaking heads. Hopefully yes. But I think this will become one of the more important policy questions of the next decade." ®
Get our [16]Tech Resources
[1] https://genaisummit.ai
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://casetext.com/blog/cocounsel-first-generative-ai-legal-assistant-one-year/
[4] https://www.thomsonreuters.com/en/press-releases/2023/august/thomson-reuters-completes-acquisition-of-casetext-inc.html
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://regmedia.co.uk/2024/06/03/jake_heller_genai_summit_2024.jpg
[8] https://www.theregister.com/2023/03/14/openai_gpt4_ai/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/06/03/microsoft_spends_32b_on_expanding/
[12] https://www.theregister.com/2024/05/31/google_ai_search_update/
[13] https://www.theregister.com/2024/05/31/datacenter_power_crunch/
[14] https://www.theregister.com/2024/05/30/kyndryl_accused_of_discriminating/
[15] https://help.casetext.com/en/collections/9434490-cocounsel-core-skills
[16] https://whitepapers.theregister.com/
Rather, he argues, sell AI as a co-worker. Corporations understand the cost of people, but not code, he reasons – and it has worked for him at least.
As he recounted at the [1]GenAI Summit 2024 in San Francisco last week, he was one of the founders of CaseText about a decade ago. When the startup began, it offered a crowdsourced legal database for legal research.
[2]
Then about two years ago, he was among those granted early access to OpenAI's GPT-4 and used it to develop CoCounsel, an AI legal assistance tool that debuted [3]on March 1 last year , two weeks before GPT-4 made a limited public debut. A few months later, CaseText was [4]acquired for $650 million by Thomson Reuters.
[5]
[6]
Heller argues that "building AI co-workers is going to be the largest opportunity of tech in our lifetime, measured by impact and measured by the amount of revenue, amount of money you can make building these kinds of technologies."
[7]
Jake Heller, head of product at CoCounsel, sees sunny days ahead – Click to enlarge
"When a lot of people hear 'AI co-workers,' especially the way I'll be talking about it a little later, it gets a little scary," he admitted. "What jobs will there be left for humans? I think it's actually bright rather than dim, for reasons that are a little bit different than a lot of the reasons people talk about."
When people talk about generative AI, said Heller, they often focus on the term "generative" – but he argues that it's the "intelligence" aspect that deserves attention. In the past two or three years, AI has got a lot more "intelligent" in ways that really matter, argues Heller, and it can take on real-world work.
Even two years ago, he said, AI models were reportedly passing bar exams – and doing so better than 70 and 90 percent of the people taking the tests. The AI [8]did better than 80 percent of those taking the Graduate Record Examinations (GRE), he asserted.
[9]
And this is old news, according to Heller. He expects that the technology as it exists today will lead to the emergence of AI co-workers – and he points to CoCounsel as an example of the genre.
"It can accomplish legal tasks at about a human level, if not a bit better, but at superhuman speeds," he claims. "Things that would take me, when I was a lawyer, many, many days to complete, it can do in 90 seconds."
For example, CoCounsel has proved itself capable when asked to find evidence of fraud in a large set of email messages – a common but boring task for lawyers. CoCounsel's results are good enough that in just a year and half on the market, 60 percent of elite law firms – the AM Law 100 – have adopted it. It's done 1.5 million tasks for attorneys – such as reviewing documents, reviewing contracts, and other research-oriented tasks, Heller claimed.
[10]
"A lot of people look at GenAI, and they think it's maybe hype, that there are all these kind of cool demos, but not real applications" observed Heller. "There are, in fact, real applications, real revenue for real customers, and real benefits."
[11]Microsoft to spend $3.2B on expanding cloud and AI in green energy-rich Sweden
[12]Google finally addresses those bizarre AI search results
[13]Surging datacenter power demand slows the demise of US coal plants
[14]IBM spin-off Kyndryl accused of discriminating on basis of age, race, disability
To build CoCounsel, the CaseText team defined [15]a set of skills that a junior lawyer might need – such as summarizing documents, editing a contract under guidance from instructors, or reviewing a contract for corporate compliance. They broke those skills down into steps, developed prompts, and worked out how to combine those skills in different ways.
The basic formula is simple, Heller noted, but the challenge is ensuring correctness.
"The difficulty really is making sure you should get the right answers," he said. "And I think part of the secret to how we became widely adopted in legal – and I think that will be true for many of the different professions and industries that you all serve – is an extremely robust testing methodology. Literally we've run over 600,000 tests, some of them against these individual prompts … and sometimes end-to-end, like if a request goes in and answer comes out, did [the AI model] hallucinate?"
Heller emphasized that only when his team had confidence in the correctness of CoCounsel's responses did they feel comfortable giving them to lawyers – whose jobs are on the line if errors occur. And one way the organization helped reassure customers is by providing sources they can read with a single click, to confirm what the AI model returns.
"The goal is not to take the lawyer's job away from them, but to assist them in all the tasks that they're doing – especially the stuff that takes a long time, but they and their customers get the least value out of," said Heller.
CoCounsel "was never pitched as software. We pitched it as a co-worker. The value isn't, 'this will make certain workflows a little more efficient, a little more effective for you.' The value is, 'now you have an additional member of your team.' And that lets you price in a way that follows value."
Heller went on to argue that he doesn't believe that AI co-workers will end up taking jobs away from people. "I think they will be upleveled to do more interesting stuff," he predicted.
That's been his experience with CoCounsel, having sold it to thousands of customers. "The number of layoffs that we've heard about as a result of the mass adoption of this technology in the legal industry is zero," he declared. Instead, it lets law firms say 'yes' to cases they'd have previously declined for lack of capacity.
That said, Heller acknowledged that what's economically beneficial in aggregate may not be positive for every person.
"Will this be net positive for every single person in society? Probably not," he conceded. "Even if it's truly net positive in the aggregate, I think some individuals in some job categories and some tasks are gonna be harder hit than others and faster than others. And so, we're going to rise to the challenge of massive potential for displacement, shifts, et cetera."
To address that, according to Heller, we need appropriate policies. "Does America, for example, have the capacity to react appropriately to this new technology?" he mused.
"I see a lot of shaking heads. Hopefully yes. But I think this will become one of the more important policy questions of the next decade." ®
Get our [16]Tech Resources
[1] https://genaisummit.ai
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://casetext.com/blog/cocounsel-first-generative-ai-legal-assistant-one-year/
[4] https://www.thomsonreuters.com/en/press-releases/2023/august/thomson-reuters-completes-acquisition-of-casetext-inc.html
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://regmedia.co.uk/2024/06/03/jake_heller_genai_summit_2024.jpg
[8] https://www.theregister.com/2023/03/14/openai_gpt4_ai/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl86I-U4iEP3sAWm8JmDIgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/06/03/microsoft_spends_32b_on_expanding/
[12] https://www.theregister.com/2024/05/31/google_ai_search_update/
[13] https://www.theregister.com/2024/05/31/datacenter_power_crunch/
[14] https://www.theregister.com/2024/05/30/kyndryl_accused_of_discriminating/
[15] https://help.casetext.com/en/collections/9434490-cocounsel-core-skills
[16] https://whitepapers.theregister.com/
PB90210
"Things that would take me, when I was a lawyer, many, many days to complete, it can do in 90 seconds."
So no more megabucks multi page lawyer's bills... legal opinion for pennies!
Jimmy2Cows
Nope, just more profit for the suits.
Doctor Syntax
He's a marketroid. Rice-Davies applies
like if a request goes in and answer comes out, did [the AI model] hallucinate?
Jimmy2Cows
While it's great they are checking for this, do they also look for things the AI model missed? Things that are fraud, but have been crafted more carefully, deliberately hard for an expert human lawyer to uncover.
No good if they're relying on the AI output because they're sure it doesn't hallucinate, but then miss real instances.
co-worker, exactly how I use it at work
jdiebdhidbsusbvwbsidnsoskebid
This is exactly how I use LLMs where I work - as a coworker. More specifically, something I can delegate tasks to, such as writing summaries of long documents, coming up with key points from brainstorm notes or meeting minutes. Luckily, we have our own garden-walled LLMs so I can feed in corporately sensitive stuff safe in the knowledge that it won't ever get out to where it shouldn't.
And before anyone asks, I never use the LLM output without reviewing it thoroughly myself. I find it most useful for rewriting existing stuff that I feed it, not for coming up with new stuff in the first place, so the "LLM Hallucination" issue is a small one and easy to deal with.
Don't get high on your own supply
He gives legal work as an example, but [1]LLM has already failed in legal work .
He also said "Even if it's truly net positive in the aggregate, I think some individuals in some job categories and some tasks are gonna be harder hit than others and faster than others. And so, we're going to rise to the challenge of massive potential for displacement, shifts, et cetera."
"Some of you may die, but it's a sacrifice I am willing to make".
[1] https://www.theregister.com/2023/06/22/lawyers_fake_cases/