The legal profession is getting ever closer to a range of artificial intelligence companions. From litigants in person using ChatGPT to major law firms with sophisticated proprietary systems, the way we do law is changing. Two recent judgments highlight the risks and rewards of relying on GenAI.
In the first, a barrister was (again) referred to the Bar Standards Board for citing a hallucinated case and misleading the court, despite arguing that he had also been a ‘victim’ of AI. In the second, a judge was widely feted for announcing he had ‘grasped the nettle’ of using AI to deliver his judgment. As we see more examples of AI in practice, questions over the status of privilege in AI prompts and outputs and the consequences of AI misuse become more pressing.
Broken China: when AI fabricates a case
In MS v Secretary of State for the Home Department (Professional Conduct: AI Generated Documents) Bangladesh [2025] UKUT 305 (IAC), the Upper Tribunal referred a barrister to the Bar Standards Board under its Hamid jurisdiction, in similar circumstances to the recent referral in Ayinde.
Here, the deployment of the fabricated case (Y (China)) had actually led to permission being granted to appeal. At the appeal, the court asked the barrister for a copy of the relevant paragraph of Y (China), noting that the citation was for a different case about fresh claims, rather than delay, which was the issue in the appeal. Following unsuccessful attempts to resolve the issue, the Upper Tribunal provided the barrister with a copy of the Ayinde decision and took a break.
Unfortunately, upon returning, the barrister informed the tribunal that he had ‘undertaken ChatGPT research’ and that the case had been decided by ‘Pill and Sullivan LJJ and Sir Paul Kennedy’. After being ordered to provide a copy of the decision, he supplied ‘an internet print out with misleading statements including references to the fictitious Y (China) case’.
In this case, it appears that acute ill health contributed to the problem. Notably, while the barrister ultimately apologised, this was only after multiple attempts to explain away the incorrect citation (as well as the tribunal supplying a copy of the Ayinde judgment).
Implications
This case illustrates a broader risk: if AI-generated hallucinations are not caught early, they can propagate through the legal system, affecting decisions at multiple levels. In common law countries, which are reliant on binding precedent to create law, systems may be needed to prevent the spread of misinformation within the justice system. Are we now, for example, at the point where the courts will need to deploy software to verify accurate citations – in the same way that in the education sector, software is deployed to defeat Chat GPT ‘drafting’?
In Ayinde, the Divisional Court set out the existing guidance concerning the use of AI in court proceedings along with professional obligations. HMCTS now expects lawyers to know these new risks.
Importantly, MS, like Ayinde, shows that transparency and correction when challenged are crucial. Once someone is asked to produce an authority or correct a mistake, how they respond can affect whether their behaviour is viewed as negligent, misleading or something worse.
Judicious use of AI
At the other end of the scale, a recent decision in the First-tier Tribunal (Tax)illustrates how AI can be used to assist the judiciary without compromising legal integrity. In VP Evans (as executrix of HB Evans, deceased) & Ors v The Commissioners for HMRC [2025] UKFTT 1112 (TC), Judge McNall used his private judiciary Copilot account to summarise documents in preparing his judgment. He clarified that AI was not used for legal research and, critically, that the judgment applied ‘in the sense of the evaluative faculty, weighing up the arguments, and framing the terms of the order’ was entirely his. Noting that while judges are not generally obliged to describe the research or preparatory work done, he felt it was ‘appropriate’ to do so here.
AI – a privileged discussion?
As lawyers (and, possibly more problematically, clients) start to consult their AI companions on issues (in the same way you might bounce ideas around with a colleague), the status of those digital discussions needs to be considered. While inputs into a GenAI tool will generally be confidential (provided a public model is avoided), it is less clear whether they will always be privileged. As the MS case clearly demonstrates, GenAI may write like a lawyer, but it is not legally qualified. So AI chats will not benefit from privilege in the same way an email from a lawyer would.
While implementing strict retention and deletion policies will reduce concerns over disclosure, we now know from the US copyright infringement case of New York Times v OpenAI (1:23-cv-11195) that OpenAI can obtain and preserve its users’ data, even when users opted out of data retention or actively deleted their chats. While Sam Altman, CEO of OpenAI, has advocated for ‘AI Privilege’, arguing that interactions with AI should be treated with the same confidentiality as those with a lawyer or doctor, it remains to be seen what view the human judiciary will take.
For now, the priority must remain for lawyers (and the judiciary) to continue to understand and adhere to their professional and ethical obligations when using AI. In future hearings, we expect the courts will impose more than regulatory referrals for breach.
Charlotte Hill is a committee member of the London Solicitors Litigation Association and a partner at Penningtons Manches Cooper. Senior knowledge lawyer Harriet Campbell and knowledge paralegal James Isaac of PMC also contributed to this article
No comments yet