Matthew Reid Krell (California State University, Northridge – Department of Business Law) has posted LLMs Can’t Replace JDs on SSRN. Here is the abstract:
Proponents of “artificial intelligence” argue that their large language models can replicate cognition, largely by pretending that what LLMs do is the sum total of what brains do. I describe three cognitive tasks that LLMs are structurally incapable of achieving — intuition, legal inference, and scientific inference. Because these tasks are frequently confused among lawyers, I distinguish between them carefully and show how the current LLM structure will never be able to perform them, and how lawyers can avoid confusing them. In other words, AI will never be able to replace a lawyer. While there may be a role for LLMs in legal practice, the most expansive claims both of AI evangelists and AI fearmongers are simply not going to come to pass in the current environment.
Krell makes the strong claim that large language models are “structurally incapable” of reasoning and “will never” perform legal or scientific inference. That categorical architectural claim is contestable and rests on thin support: the central “LLMs never do ‘Aha!’” premise is sourced to a single unpublished manuscript, and the invocation of the transformer architecture (Vaswani et al., Attention Is All You Need) does not establish the in-principle limitation asserted, nor does it engage with in-context learning, retrieval-augmented generation, tool use, or the reasoning-model developments since 2022. The paper is on firmer ground where it draws on the peer-reviewed literature on hallucination (Kalai et al., Why Language Models Hallucinate) and on the unfaithfulness of chain-of-thought reasoning (Turpin et al.; Chen et al.) to argue that current LLM output is plausible but not reliably trustworthy or replicable—a narrower and better-supported claim than the categorical “cannot reason” thesis. Readers should weigh the structural-impossibility framing accordingly.