A Novel Evaluation Strategy for Domain-Specific Legal QA: Application to Bangladeshi Law with RAG

Abstract

Retrieval-Augmented Generation, or RAG, has been identified as a promising approach for question answering tasks, particularly for high-risk fields such as law, for which accuracy, interpretability, and contextuality are essential. Current evaluation approaches for RAG models are general natural language evaluation scores, which do not embody specific legal reasoning attributes such as temporal validity, jurisdictional validity, procedural adequacy, or reconciliation of numerical constraints. This paper introduces a legal-aware evaluation framework for RAG models, and we build an AI legal assistant for Bangladeshi laws. We introduce a novel evaluation framework with seven complementary metrics: (1) Gold Answer baseline correctness, (2) BERTScore semantic similarity, (3) Numeric Accuracy, (4) Temporal Validity, (5) LLM Judge reasoning quality, (6) Legal Validity statute checking, and (7) Robustness. Evaluation results show our legal-aware framework is successful and measures key unexplored failing dimensions for quality evaluation, giving a CLR score of 0.824 on 20 test queries under the Baseline configuration, with perfect scores on Temporal Validity (1.00) and Legal Validity (1.00), and strong BERTScore semantic similarity (0.861), and improved Numerical Accuracy following word-form normalization of Bangladeshi legal quantity expressions, confirming both system reliability and the framework’s ability to surface domain-specific failure modes invisible to standard NLP evaluation.