The list of tasks at which humans can do a better job than computers shortens today with the publication of a study showing an algorithm beating experienced lawyers in what is billed as ‘real world’ legal work. The exercise pitted 20 experienced US lawyers against an artificial intelligence system in going through contract clauses to spot issues such as liability for disclosure to third parties.
In a scoring system involving points for precision and taking into account false positives and negatives, the lawyers scored 85% while the algorithm managed 94%, according to Israel-based legal technology specialist LawGeex. The lawyers took an average of 92 minutes to complete their reviews; the AI engine took 26 seconds.
Academic studies showing computers outperforming humans at tasks such as predicting case outcomes have become routine in recent years. However, LawGeex says its exercise is significant because it tackles a ‘major business problem’. This is the need for businesses to retain a trained lawyer to review ‘even the most prosaic’ non-disclosure agreements (NDA) in commercial contracts.
The study involved five publicly available NDA agreements used by US companies. The human participants each had ‘significant commercial experience in this exact form of NDA review’. They were asked to annotate the five agreements from a list of 30 issues and given four hours for the task.
The algorithm had been trained with tens of thousands of NDAs but had not previously processed the five selected agreements.
Legal AI expert Professor Gillian Hadfield of the University of Southern California said the outcome probably understates the gain from AI. ‘The lawyers who reviewed these documents were fully focused on the task,’ she said. ‘It didn’t sink to the bottom of a to-do list, it didn’t get rushed through while waiting for a plane or with one eye on the clock to get out of the door to a meeting or to pick up kids.’
LawGeex described the study as a ‘major landmark’. However it stressed the technology ‘is not meant to be, nor indeed is it currently capable of being used as, a standalone tool.’