Popular AI chatbot GPT-4 has surpassed the performance of many aspiring lawyers on the legal ethics examination required by nearly every state for the practice of law, as indicated in a recent study published on November 16.
As per a report unveiled on Thursday by LegalOn Technologies, a provider of AI software for contract evaluation, GPT-4 achieved a 74% accuracy rate on the simulated Multistate Professional Responsibility Exam (MPRE), outperforming the estimated 68% average among human examinees nationwide.
The study suggests the potential for developing AI tools in the future to assist lawyers in adhering to ethical standards and fulfilling their professional duties as needed.
This research contributes to the expanding body of literature exploring AI applications in legal education and licensure. An earlier study revealed that the predecessor of GPT-4 received passing grades, though not exceptional, on law school final exams. Recent findings indicate that GPT-4 successfully completes the bar exam. However, a study conducted this month found that while using GPT-4 improved the speed of legal writing tasks, it did not enhance the quality of work produced by law students.
GPT-4, a substantial language model supported by Microsoft-backed OpenAI, generates human-like language in response to user queries.
The National Conference of Bar Examiners, responsible for the MPRE, stated through a spokesperson that it could not verify the claims made in the LegalOn study regarding GPT-4’s ethics assessment.
Sophie Martin, the director of the National Conference, emphasized that “the legal profession is continuously integrating technology into its operations, a trend that will persist. Attorneys possess a unique skill set that AI currently cannot replicate.”
In addition to passing the bar exam, law students in all states except Wisconsin must pass the 60-question MPRE before gaining admission to the bar. The exam covers topics such as conflicts of interest, attorney-client relationships, and confidentiality and is typically taken during law school.
GPT-4 exhibited exceptional performance with a 91% accuracy rate in questions related to conflicts of interest and an 88% accuracy rate in queries concerning attorney-client relationships. However, its accuracy was slightly lower in questions regarding communication about legal services, safekeeping of funds, and other assets, with correct responses given 71% to 72% of the time.
The study concludes that “this research demonstrates, for the first time, that advanced AI models can effectively apply established ethical principles as proficiently as aspiring legal professionals.”
