AI and the SFO: papering over the cracks?

The Serious Fraud Office has again faced scrutiny from the Commons justice committee, most recently in an October hearing which examined systemic problems in relation to disclosure. The committee heard from Sir David Calvert-Smith KC, Brian Altman KC and Anthony Rogers, who were responsible for reviewing the SFO’s handling of the Serco and Unaoil cases. It heard that the Calvert-Smith review into Unaoil and the Altman review into Serco reached the same conclusion: each case collapsed because of disclosure failures. 

In both reviews, serious organisational issues were identified, which were addressed in evidence from SFO director Lisa Osofsky and chief operating officer Michelle Crotty. Of note was the ‘endemic lack of resources’ plaguing the largest cases; the lack of incentivisation for individuals to take on disclosure officers’ duties; and pervasive staff resourcing and retention issues.

As part of a wider move to address its issues, the SFO intends to innovate its disclosure processes, which includes making greater use of artificial intelligence and machine-learning. However, technology alone cannot fix all its woes and the SFO must therefore approach AI with caution.

Sir David emphasised that the SFO is suffering from an ‘endemic lack of resources’ for its largest cases, which hampers its ability to deal with the volume of digital material it receives for each investigation – often upwards of 10 million documents. As he highlighted, defendants are often far better resourced than the SFO.

AI offers one solution to the issue of managing large volumes of material, not only reducing review time but also the number of staff needed and consequent costs. It automates what has long been the responsibility of ‘document reviewers’. For example, the technology can describe documents, determine their relevance and decide further keyword searches to run.

Despite this, the SFO is constrained in its use of AI by its own risk management procedures. This was acknowledged in its evidence: ‘Technology should not be the ultimate decision-maker in circumstances where a wrong decision could result in an unjust criminal justice outcome.’ Proposed safeguards for the use of AI included the use of human decision-makers, sample testing and then re-testing results.

In reality, AI can accelerate only parts of the disclosure process and, in cases with millions of documents, lengthy investigations will remain. Is it therefore time to hand over ‘the keys to the warehouse’? Or at least to allow the defence access to their own sections of it? Altman expressed sympathy for this idea, distinct from Calvert-Smith, but there is little doubt that cases would progress faster if the defence had earlier access to documents. Since many firms have already invested in or regularly use AI, receiving large quantities of documents from the SFO should no longer be as burdensome for a properly resourced defence. Due consideration would need to be given to publicly funded defendants however, to ensure that a disparity does not arise in terms of fairness and access to often expensive AI software.

The Altman review recommended the director ‘consider ways in which staff may be incentivised to take on’ the role of disclosure officer. The suggestion was made in light of the SFO’s process of picking existing staff for the role, without much choice in the matter. Training and support are necessary for employees to feel comfortable in the role, given its importance in ensuring that proceedings are fair.

This is important given the complex nature of AI and its increasing use by the SFO. Specialist training is necessary so that employees can use it and also explain its application to the defence and the court.

Still, training and support only go so far. As highlighted by Crotty, many document reviewers are temporary agency staff. The expendable nature of their employment is unlikely to incentivise them to develop in their role so the technology will still be operated unsatisfactorily and underutilised.

Crotty revealed that the SFO has a ‘permanent vacancy of between 20% and 25%’. Arguably AI’s efficiency benefits may plug staffing gaps by relieving case teams in reviewing material. Despite this, as Crotty admitted, with the growing cost of living, SFO salaries are becoming a challenge, making it hard to hire and retain staff.

Staff retention is a key issue if the SFO wants to benefit from AI. Its investigations are rarely shorter than a few years, so retaining the staff responsible for the input to and application of AI in a disclosure exercise is vital. A high turnover of those who can account for how and why the technology was applied will undermine efficiency.

Another issue with the proposed use of AI is that it butts heads with the SFO’s legislative disclosure obligations. This is because machine-learning uses algorithms to make decisions and, due to their nature, these are mostly ambiguous and unexplainable to humans. This is at odds with the Attorney General’s Guidance on Disclosure, which requires that details of sampling techniques used (including keyword searches) are provided to the defence and the court. This is a problem because the SFO will be able to explain only its ‘input’ into the software and the ‘output’ from that, but the part in between – where the AI has learnt and made decisions – will be unknown.

This is problematic for defence lawyers and the court. For example, if the SFO missed a relevant document, how will it identify a lacuna in the search terms applied which allowed for its absence? And how can the court trust that the AI has correctly applied the SFO’s input and the section 3 of the Criminal Procedure and Investigations Act 1996 test of relevance? It is possible that SFO cases will continue to collapse, now with the added risk of a glitch in the software undermining the entire disclosure effort.

The SFO must not grapple with the complexities of AI before rectifying more fundamental issues. Although AI should improve the SFO’s ability to deal with large volumes of material and relieve pressure of resources, the technology has limited benefit while other key issues remain.

Rachel Cook is of counsel at Peters & Peters. Trainee solicitor Abbie Melvin also contributed to this article