Introducing SaulLM-7B, the first publicly available large language model (LLM) specifically designed for legal applications. It addresses the unique challenges of legal text by extensively pretraining on legal corpora from English-speaking jurisdictions like the USA, Canada, the UK, and Europe. SaulLM-7B aims to comprehend the complexities of legal documents and adapt to evolving legal discourse.
The SaulLM-7B family includes a 7-billion-parameter LLM tailored to legal text, and SaulLM-7B-Instruct, an instruction-tuned variant engineered to outperform existing models on legal tasks. It introduces an improved evaluation protocol, LegalBench-Instruct, for assessing the legal proficiency of LLMs. This protocol includes tasks from the popular MMLU benchmark, focusing on international law, professional law, and jurisprudence. SaulLM-7B and SaulLM-7B-Instruct, along with evaluation code, are released under the MIT License to foster widespread adoption and innovation within the legal domain.
The methodology involves continued pretraining on a high-quality legal dataset, followed by instruction fine-tuning using both general and legal instructions. The dataset is curated from various sources, including legal texts, conversational data, and instructional prompts. Data cleaning and deduplication processes ensure the quality of the training data.
Experimental results demonstrate that SaulLM-7B and SaulLM-7B-Instruct outperform existing models on legal benchmarks, showcasing significant improvements in legal proficiency. SaulLM-7B exhibits superior performance in understanding legal nuances and excels in tasks that require legal-specific knowledge. Additionally, SaulLM-7B demonstrates lower perplexity scores compared to baseline models across different types of legal documents.
Overall, SaulLM-7B represents a significant advancement in the application of LLMs to the legal domain, offering state-of-the-art performance and paving the way for further innovation in legal language understanding and application.