The Judicial Mind of Large Language Models

Large language models have arrived in the court room: not as decision-makers, but as decision-aids. Realistically, the human judge in the loop can only do so much. What can be done to prevent the AI assistant from surreptitiously introducing bias and from being manipulated by the parties? We address the first concern directly and the second indirectly, exploring whether doctrinal constraints reduce the surface for strategic manipulation. We investigate experimentally how strongly the obligation to reason in terms of established legal doctrine can constrain AI outputs through two channels: by directing attention to relevant precedents, and by imposing structure on the reasoning process. Six treatments are designed to isolate and reinforce these two channels through which legal doctrine operates. We emulate similarity-based reasoning with a vector RAG and structural reasoning with a graph RAG, and combine both interventions. We test seven LLMs, comparing both outcomes and a rich set of metrics for the quality of legal reasoning with decisions by the European Court of Human Rights. Even without RAG, frontier models produce impressive outputs. But RAG interventions significantly and substantially improve both accuracy and reasoning quality.

Yoan Hermstrüwer’s research explores how behavioral science, economics, and artificial intelligence can inform legal design, legal reasoning, and the solution of social problems. He applies empirical, experimental, and computational methods to study the interaction between humans and algorithmic decision-making, the application of market design to legal institutions, and the use of AI (machine learning, large language models) to analyze legal reasoning.

He publishes in peer-reviewed journals across economics and psychology (e.g., The Economic Journal; Games and Economic Behavior; Psychology, Public Policy, and Law), law journals (e.g., NYU Journal of Law & Liberty), and computer science proceedings (e.g., International Conference on Learning Representations; Natural Legal Language Processing Workshop).  For full bio, visit: https://www.ius.uzh.ch/de/staff/professorships/alphabetical/hermstruewer/team/person.html.

Moderator: Benjamin Chen, Associate Professor & Director of Law and Technology Centre, The University of Hong Kong Faculty of Law

To register, please go to https://hkuems1.hku.hk/hkuems/ec_regform.aspx?guest=Y&UEID=105135.