Pakistan study estimates $38.50 saved per dollar invested in training judges on an AI legal assistant

Pakistan has fewer than two judges per 100,000 residents (versus 22 in the EU and 30 in England and Wales), and by the end of 2024 had a very large backlog of pending cases, with the vast majority in trial courts. Judges work with minimal technology and no support staff, and before the study only about 25% had ever used a large language model like ChatGPT.

Researchers from ETH Zurich, Imperial College London, and the New Economic School ran a randomized field experiment across 1,559 judges in 118 courts (roughly half of Pakistan's trial court judges), testing JudgeGPT, a GPT-4-based assistant using retrieval augmented generation over 129,235 documents (128,292 rulings and 943 laws) to generate cited answers. One group received the tool plus six 90-minute lectures over three weeks taught by ETH Professor Elliott Ash; a second group got the same tool access but only a general seminar; a control group attended the seminar with no AI access.

Trained judges used JudgeGPT about four times as much as judges who had the same access and only a general seminar (nearly 60 logins and over 200 prompts after 40 weeks versus about 20 logins and fewer than 50 prompts). The report says AI access alone did little. Districts with more trained judges resolved more cases: at moderate exposure, about 1,848 more cases per year per district (a 6.3% increase), and even bottom-quartile districts cleared about 616 more. The researchers estimate savings of about $38.50 per dollar invested, based on what it would cost to hire enough extra judges to match the same output, and at least $10 per dollar under conservative estimates. Ruling quality held steady or improved: an LLM-based quality check validated by two Pakistani lawyers rated rulings from trained judges better in 59% of pairwise comparisons, up from 42% in the control group. The appeal rate per 1,000 resolved cases fell slightly, judges worked the same hours, and the study found no evidence that AI use increased gender or religious bias in judicial language.

• Tool access alone barely moved usage or outcomes — targeted training on which tasks suit the AI (and which don't) was the real multiplier, driving substantially higher usage and measurable case-resolution gains.
• Training shifted judges toward lower-risk tasks like editing and summarizing and away from open-ended legal questions where hallucination risk is higher, while keeping final decisions in human hands.
• The study ran on GPT-4, a pre-reasoning model, suggesting today's more capable models could yield even larger productivity gains.