New Study Questions Bar Exam’s Predictive Validity for Lawyer Effectiveness
On Friday, November 1, 2024, Sage Journals published a study that examines the predictive validity of bar exam outcomes on the effectiveness of newly admitted lawyers.
The study, titled “Putting the Bar to the Test: An Examination of the Predictive Validity of Bar Exam Outcomes on Lawyering Effectiveness,” is the first of its kind to analyze the relationship between bar exam scores and the performance of new lawyers. It combines data from three distinct sources: the bar results from the State Bar of Nevada, a survey of recently admitted attorneys, and evaluations from supervisors, peers, and judges regarding the effectiveness of these new lawyers in their practice.
The research found that performance on both the Multistate Bar Examination (MBE) and the essay components of the Nevada Bar Exam demonstrated a weak correlation with assessed lawyering effectiveness. This raises questions about the reliability of bar exams as indicators of a lawyer’s competence and ability to serve clients effectively.
Bar exams are a requirement for law school graduates in the United States seeking to practice law. The National Conference of Bar Examiners (NCBE) states that these exams aim to ensure that only individuals who exhibit minimum competence are licensed. Traditionally, bar exams assess knowledge of legal doctrine, procedural rules, analytical reasoning, and writing skills. However, the current study suggests that these skills, while related to legal practice, do not encompass the full range of competencies that make an effective attorney.
The authors highlight persistent racial disparities in bar passage rates, citing American Bar Association (ABA) data which shows that, in 2021, White law school graduates had significantly higher first-time pass rates compared to their Black, Hispanic, and Native American counterparts. These disparities indicate potential barriers to entry into the legal profession for underrepresented groups.
The study utilized Tobit regression analysis on survey data from over 500 newly licensed attorneys in Nevada. This analysis sought to determine how well bar exam performance could explain the ratings given to these lawyers by their peers, supervisors, and judges. The findings indicated that while some positive correlations were present, they were minimal and lacked practical significance.
The study also notes that the bar exam is set to undergo substantial changes with the introduction of the NextGen Bar Examination in 2026. Despite these changes, the fundamental components, such as multiple-choice questions and essay writing, will remain largely the same, suggesting that the current study’s findings may still apply to the new format.
Moreover, the research has implications for jurisdictions considering adjustments to their cut scores—the minimum passing scores for the bar exam. The authors suggest that the lack of a strong relationship between bar exam scores and lawyering effectiveness may encourage jurisdictions to reconsider their cut scores or explore alternative pathways for legal licensure.
The bar exam typically consists of multiple components: the MBE, which is a multiple-choice test; the Multistate Essay Examination (MEE), which includes essay questions; and a performance test (MPT) which assesses practical skills. The MBE constitutes 50% of the total score, while the MEE and MPT account for 30% and 20%, respectively. However, cut scores for passing the exam vary significantly by jurisdiction, with some states requiring scores as low as 260 and others as high as 270 out of a possible 400 points.
The study also discusses the challenges of using attorney discipline rates as a measure of lawyering effectiveness. It points out that such measures are limited and do not accurately reflect the competencies required to practice law effectively. Previous studies have indicated that even among disciplined attorneys, the reasons for discipline may not align with the skills assessed by the bar exam.
In conclusion, the study underscores the need for ongoing scrutiny of the bar exam’s effectiveness as a measure of lawyer competence. The authors call for rigorous validation testing of bar exams to ensure that they accurately assess the skills necessary for effective lawyering, particularly in light of ongoing discussions about racial disparities and access to the legal profession.
“More research is needed, but this study finds that while the bar is serving as a significant barrier to the practice of law, there is little indication that it is a robust indicator of what it takes to be a “good” lawyer,” the authors said.
This research contributes to a growing body of literature questioning the efficacy of standardized testing in the legal field and emphasizes the importance of evaluating the skills that truly predict success in legal practice.
Source: Sage Journals
