Why Idaho May Not Be a Good Indicator of What Future October Remote Bar Exam Results Will Look Like
Idaho was one of the few jurisdictions that offered both an in-person bar exam in July 2020 as well as a subsequent remote bar exam in October, providing a comparison between the in-person and remote administrations.
Based on information culled from various places (the NCBE, the state boards of law examiners, social media outlets, etc.), only 28.6% passed the October 2020 Idaho remote exam after the state saw a much-higher bar passage rate of 76.7% on the July 2020 in-person bar exam
This is a very significant decline—nearly a 50 percentage point difference. But the situation in Idaho should not be viewed of what the bar passage rates will be in other jurisdictions that administered the October remote bar exam.
1. Timing of the announcement
The Idaho Supreme Court announced on July 20, 2020—just one week before the administration of the July 2020 in-person bar exam—that it would give applicants the ability to withdraw from the July 2020 in-person bar exam and take the October remote bar exam instead.
The Court permitted applicants who were eligible to sit for the July 2020 in-person bar exam, as well as those who had previously applied for July 2020 in-person bar exam but had already deferred to take the February 2021 bar exam, to take the Oct. 5-6 remote exam in lieu of the July 2020 or February 2021 bar exam.
Given the late timing of the announcement, two groups of applicants likely took advantage of the October remote bar exam, and both these groups likely contributed to a lower passage rate on the remote exam.
First, applicants who were registered for the July 2020 in-person bar exam but deferred to take the Oct. 5-6 remote exam likely led to a lower bar passage rate on the remote exam because, if these applicants were unprepared to sit for the July bar exam with one week remaining to prepare for the July exam, then the additional studying time for the October remote exam might not have helped these applicants. Second, applicants who had deferred the July exam and planned to take the February 2021 exam, but then decided to move up the exam to October 2020, essentially had their study time cut by several months as a result.
It’s impossible to know why or who decided to take the October remote bar exam, but it is not unreasonable to assume that the applicants taking the July 2020 in-person bar exam was a better prepared group of test-takers given the timing of the announcement.
Most jurisdictions that adopted the October remote bar exam did so by the beginning of July and did so in conjunction with cancelling the July in-person exam, ensuring that the October remote bar exam examinees were not distinguished from another group of bar exam takers.
2. Grading of the Idaho bar exam
Before the administration of the October remote bar exam, the Idaho Supreme Court announced that the passing score will be 68% of the total points possible, consistent with how all Idaho bar exams are scored under Idaho Bar Commission Rule 217.
The problem, however, with relying on consistent passing score standards, as the Idaho Supreme Court did, is that the October remote bar exam was not similar to previous Idaho bar exams also subject to the 68% point requirement.
Idaho first administered the Uniform Bar Exam in February 2012. (The UBE consists of 200 multiple-choice questions, two Multistate Performance Tests, and six Multistate essays.) Since Idaho’s adoption of the UBE, it has administered the UBE a total of 17 times. The October remote bar exam, however, was not a UBE and, in fact, contained only one-half of the parts of the UBE (100 multiple-choice questions, one MPT, and three essays). Likely because the remote exam was not in the form or format of prior administrations, the NCBE announced that it would not equate the MBE, calculate scaled scores for the written components, or provide UBE and MBE score transfer services.
The Idaho Supreme Court setting the passing score at 68% of the total points possible on the October remote exam appears to be done more out of consistency and ease rather than an examination of actual examinee performance on the October exam and a historical comparison of the October results with the results of prior UBE administrations.
It should be noted that many of the UBE jurisdictions that had cancelled their July in-person administrations and adopted the October remote exam announced that they would hire a subject matter expert psychometrician to provide grading and scaling services to ensure the validity, reliability, and fairness of the process for the October remote exam. As the Connecticut Bar Examining Committee announced, “The expert psychometrician will convert scores on the 100 item multiple choice exam to a 200 point score using a raw to scale conversion plan based on NCBE’s historic raw to equated scale score transformations. This will ensure high exam reliability while maintaining a standard that is reflective of past exams. All jurisdictions agreeing to reciprocity for the October 2020 remote bar examination have retained the same expert psychometrician and have agreed to the same grading and scaling methodology.”
Idaho was not a member of this group of jurisdictions.
Reciprocal jurisdictions include: Connecticut, District of Columbia, Illinois, Kentucky, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Ohio, Oregon, Tennessee, and Vermont.
3. Comparison of combined July/October with previous July results
The Idaho Supreme Court announced on Oct. 21, 2020, that eight applicants passed the October remote exam. It did not, however, release other specific numbers, such as the number of test-takers or bar passage rates. However, a review of online discussion boards indicate that that 28 applicants sat for the October remote bar exam, resulting in a 28.6% bar passage rate (8 out of 28). By comparison, the NCBE announced that 76.7% of the examinees passed the July 2020 in-person bar exam, with a review of online discussion boards indicating that 92 out of 120 examinees passed the exam.
By comparison, here’s a summary of July bar passage rates on the Idaho bar exam from July 2016 to July 2019:
Over the four July administrations of the Idaho bar exam from 2016 to 2019, the average July bar passage rate was 70.2% with an average of 142 examinees per administration.
Looking at the July 2020 and October 2020 exams together, the results are very similar to prior July-only statistics between 2016 and 2019. A total of 148 examinees sat for both July 2020 and October 2020 bar exams. This number is in line with the number of examinees in the past—it’s not the highest number of test-takers, and it’s not the lowest. And it’s just six examinees off from the four-year July average of 142 examinees from 2016 to 2019.
To better reflect the bar passage rate for July 2020, the numbers for July 2020 and October 2020 should be examined together. After all, as noted above, examinees who sat for the October remote bar exam must have been eligible to sit for the July 2020 exam. But in the absence of COVID-19, those eligible applicants for the July 2020 exam would not have been permitted to defer their applications to the July exam to either the October 2020 or February 2021 exams.
While not released by the Idaho Supreme Court, here are the likely numbers for the Idaho bar exam for July 2020 and October 2020:
By combining the two most recent administrations, the July/October 2020 bar passage rate was 67.6%. This number is in line with prior bar July-only passage rates in the past—it’s not the highest passage rate, nor is it the lowest. And it’s just 2.6 percentage points off from the four-year July average bar passage rate of 70.2% from 2016 to 2019.
While some observers might want to focus on the 28.6% bar passage rate on the October remote exam, it’s important to note that the combined July/October 2020 bar passage rate was, in fact, 4.0 percentage points higher than the previous July administration in 2019.
