While working on test validation projects I often get asked: How long should my test be? The answer is pretty simple: The test needs to be long enough to cover all the important content of the learning system. And how do you do that? You do it by writing the questions to the learning objectives. Depending on the importance and complexity of each learning objective it could be one question or maybe as many as five. As a consequence of this process your test might have a nice round number of questions that easily divides into 100%, like 20 or 25, resulting in a whole number of points per question, or it could be a number that doesn’t easily divide into 100%, like 17 or 32. We have a bias for a whole number of points per question but that is really an artifact of hand scoring (though the test takers do like it as well). Today it doesn’t matter so much. Computers are very good at doing the math for you.
I was recently working on a test validation project that got me thinking about test length, points per question and passing score. Most of my clients have set their passing scores at numbers divisible by five: 80%, 85% or 90%. This isn’t exactly kosher since the passing score should really be determined by something like the Angoff process, which could result in a non-round passing score, like 87%. But trainers and test takers alike don’t like “odd” passing scores, so we often round them to the nearest multiple of five.
So what relationship does passing score have to test length? A 90% passing score is pretty common for my clients so let’s work with that as an example. If we give a 10 question test then a student can get one question wrong and pass the test, but two questions wrong fails the test. What about an 11 question test? Same result. A 15 question test? Same. All the way up to 19 questions. But let’s add one more question. At 20 questions the student can now get two questions incorrect but still pass the test.
So think about that: A 19 question test is much harder to pass than a 10 question test, which kind of makes sense, but by adding just one more question (from 19 to 20) you have made passing significantly easier! Counterintuitive but true. Another break occurs after 29 questions (at 29 questions the student fails with three wrong; at 30 questions the student passes with three wrong) and so on.
Here’s a handy table showing some of the breaks for three different passing scores:
|80 % Passing Score||85% Passing Score||90% Passing Score|
|Number of Questions in Exam||Number Permitted Incorrect||Number of Questions in Exam||Number Permitted Incorrect||Number of Questions in Exam||Number Permitted Incorrect|
At any of these break points the number of incorrect answers permitted jumps by one question. So, the key takeaway here is: Think about exam length and its consequences the next time you create a test.