AI will come with a jagged edge

Some Gazette readers may believe that artificial intelligence software will never have a place in legal services; some may believe it will take over entirely from humans. In the middle, I suspect, the vast majority see AI as a tool to be pulled out of the box when necessary. 

The snag is that picking the right job for the current generation of AI tools - large language models - is not easy. To understand why, think of systems such as ChatGPT as very sophisticated predictive text generators. They work by detecting patterns in data throughout their virtual universe rather than selecting from inputs (the way the old so-called 'expert systems' worked). Hence ChatGPT's infamous 'hallucinations'.

These systems are good at tasks which were beyond previous generations of AI software, such as generating ideas, but hopeless at tasks like basic maths - or, indeed, mission-critical legal decisions. 

In an important new research paper*, researchers from Harvard Business School explore this 'jagged frontier' of AI's capabilities. It describes a large scale randomised trial of the impact of AI when used by 'knowledge workers' on realistic tasks both within and outside the frontier. Although the 758 subjects were not lawyers but management consultants from the prestigious firm Boston Consulting Group, some of the lessons will resonate with law firms. 

In brief, the study shows that large language models (specifically Chat GPT-4) 'are highly capable of causing significant increases in quality and productivity, or even completely automating some tasks' - but only within a rapidly changing frontier. 

In the experiment, participants (incentivised by office kudos and small gifts) were assigned two sets of tasks that would be involved in helping a fictional footwear manufacturer break into new markets. While both were designed to be comparably complex and realistic, the first task was selected to be within the technological frontier of GPT-4. The second was designed so that GPT-4 would make an error in its analysis, ensuring the work fell just outside the frontier.

The result? 'For tasks that are clearly within its frontier of capabilities, even those that historically demanded intensive human interaction, AI support provides huge performance benefits.' Outside the frontier was a different story; consultants who over-relied on AI ended up making potentially expensive mistakes. 'Not all users navigated the jagged frontier with equal adeptness,' the paper observes. 

Given these findings, it 'becomes incumbent upon human professionals to recalibrate their understanding of the frontier and for organizations to prepare for a new world of work combining humans and AI.' Navigating the frontier 'requires expertise, which will need to be built through formal education, on-the-job training and employee-driven upskilling.'

The Harvard study will generate some Schadenfreude over the prospect of management consultants, notorious for foisting IT-powered 'transformation' on other industries, getting a taste of their own medicine. But the general lesson seems transferable to other knowledge-based jobs. In particular, the warning to be wary of unsupervised AI in areas where 'correctness' is important. Like rather a lot of law, in fact. 

But importantly, we need 'to move beyond the dichotomous decision of adopting or not and instead to evaluate the value of different blends of human and AI for specific tasks and roles'. 

This matters for all 'knowledge workers' - journalists included. I was lucky enough to join the newspaper industry when it was on the cusp of a technological revolution after nearly a century of stasis. The radical transition in the technologies of printing, editing and newsgathering was sometimes a wild ride but it hasn't finished yet: at the Gazette we're now hoping to test the use of AI tools in a couple of ring-fenced tasks well inside the jagged frontier. 

Watch this space. It's an exciting one. 

*Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality

Michael Cross is the Gazette's news editor. He has followed developments in artificial intelligence for four decades