In discussions with clients I notice that some people inadvertently use the terms randomizing and subsetting interchangeably, though they are really quite different with entirely different consequences for exam validity.
Randomization means that all test takers get the same questions but in different order.
Subsetting means that each test taker gets a different subset of questions from a larger item pool, though unless the item pool is very large compared to the test size some (or many) questions will appear on multiple individuals’ exams.
To make things more confusing, most exams that employ subsetting also randomize the questions.
Why subset or randomize? Simple: To discourage cheating. If you could administer a 100%-secure test there would be no reason to do either.
So, if you do need to make it more difficult to cheat which should you use and when? It depends on the type of exam you are giving.
For low stakes exams (exams with no consequences) and learning exercises feel free to do either or both. But for high stakes exams (exams with consequences) you can randomize the questions, but you should never subset the questions.
Why? Because unless you have performed a test equivalency process, which is not a simple thing to do, you cannot guarantee that the exam taken by Person A is the same level of difficulty as the exam taken by Person B. If Person A’s exam turned out to be more difficult than Person B’s, and B passes but A does not, you are opening yourself up to legal jeopardy.
What about randomization with no subsetting? Does the order in which test takers see the questions affect exam difficulty? There is research evidence that shows it does not, so feel free to randomize the questions.
Many testing systems also give you the ability to randomize question choices (i.e. for each question, test takers see the choices in different order). Does this affect question difficulty? Again, research evidence suggests it does not.
So, bottom line: You can always randomize questions and question choices (with some caveats), but be careful with subsetting.
intelalearning.wordpress.com 
Hi Steven. Very interesting. I would love to know what research you’ve read that indicates that randomizing Q order or Q answer options doesn’t appear to affect Q difficulty. Thanks much for this post. It’s a
LikeLike
Thanks for the question Laurie. I was going to dash off a quick reply of the form: Look at studies X and Y — but the more I investigate the more I realize the answer is a little complicated. From what I have read I still believe my assertion is correct, but I think the answer deserves a fuller treatment than what I can put in a comment, so I’ll write it up as my next blog post.
LikeLike
Great. I look forward to it. Thank you for the response.
LikeLike