In my workshops on assessment writing I provide ten rules for writing valid questions, I discuss the role of learning objectives and I emphasize the importance of writing questions at a range of Revised Bloom’s Taxonomy levels — but I’ve come to realize that all of that, while important, is not enough. Above all, question writers have to write meaningful questions. What do I mean by a meaningful question? It’s a question that is relevant to the learner’s need-to-know.
My first inkling of this distinction was a few years ago when I was going through an Angoff cut score setting process for a sales representative exam at a pharmaceutical company. The training departments of life science companies tend to have two kinds of employees: career learning professionals, often with backgrounds in instructional design, and rotational employees who are typically former sales representatives doing a one to three year rotation in training who will then go on to other positions within the company, commonly in sales management.
All of the Angoff judges recruited for me were of the former-sales rep variety of employee. As we were going through the question review process we came across a number of questions that they wanted to discard because (as they explained it): “That’s a valid question but in a million years that information would never come up in a sales call.” And that’s when I understood the concept of a meaningful question.
When you are writing questions put yourself in the position of the learner/employee. Is this something the learner will actually need to know on the job? Is this something you would expect the learner to remember a few months from now? If the answers aren’t “yes” and “yes,” the question may not be meaningful.