Christoph Engel, Yoan Hermstrüwer, and Johannes Kruse (all Max Planck Institute for Behavioral Economics) have posted Can doctrine harness silicon law clerks? on SSRN. Here is the abstract:
Large language models have arrived in the court room: not as decision-makers, but as decision-aids. Realistically, the human judge in the loop can only do so much. What can be done to prevent the AI clerk from surreptitiously introducing ideological bias, and from being manipulated by the parties? We investigate experimentally whether an institutional constraint that has a guiding effect on human judges also helps to discipline silicon law clerks. Specifically we study how strongly the obligation to reason in terms of established legal doctrine constrains AI outputs, on two channels: by directing attention to relevant precedents, and by imposing structure on the reasoning process. Six treatments are designed to isolate and reinforce these two channels through which legal doctrine operates. We emulate similarity-based reasoning with a Vector RAG and structural reasoning with a Graph RAG, and combine both interventions. We test seven LLMs, comparing both outcomes and a rich set of metrics for the quality of legal reasoning, using decisions by the European Court of Human Rights as ground truth. Even without RAG, frontier models produce impressive outputs. But RAG interventions align outcomes and explicit reasons significantly and substantially more with the human benchmarks.
Highly Recommended!
To receive new posts from Legal Theory Blog by email, get a free subscription to Legal Theory Stack.
Lawrence Solum