Anonymous Immigration Legal AI Platform

The platform builds agentic workflows that generate draft immigration documents (O-1A, H-1B, green cards, etc.) for lawyer review and editing. Lawyer review time was the primary bottleneck to scaling throughput and margins. Each time AI workflows were updated—prompts, models, or tools—the team had no reliable way to detect quality regressions, forcing lawyers and engineers into expensive, inconsistent manual side-by-side comparisons. Post-deployment regressions surfaced even more problems.

Judgment Labs built custom post-trained LLM evaluators trained on the company's internal data—pairs of AI-generated rough drafts and lawyer-finalized drafts—to mimic lawyer review automatically. An LLM-as-jury system (N evaluators with different base models, weighted majority vote) further improved evaluator reliability. The judge was integrated into an automatic regression testing pipeline via the judgeval package, and later adapted as an Agent Behavior Monitoring (ABM) system to triage poorly generated documents before they reach lawyers.

The post-trained judge correctly identified the better document the vast majority of the time (vs. near-chance baseline performance), matching or exceeding lawyer-level accuracy. Lawyer review time dropped by more than 85%, saving 100+ hours per month across the caseload. The team shipped 2 new agent releases 3 months ahead of schedule and now deploys updates 3x faster, while supporting 20% more caseload with the same team size. The ABM system, live for three weeks at publication, surfaced over 40 cases with factual contradictions, misquoted citations, and misconstrued evidence.

• Training an LLM judge on real lawyer edit pairs (rough draft → final draft) captures domain-specific quality signals far better than generic out-of-the-box LLM judges.
• An LLM-as-jury ensemble with weighted majority voting reduces evaluator inconsistency and improves alignment with human expert judgment at scale.
• Automated regression testing that eliminates the need for human comparison unlocks faster iteration cycles; the same judge model can be repurposed as a production monitoring / early-warning system.