The Problem with Rating Scales

What’s wrong with rating scales? A lot. They are ubiquitous in learning and assessment, appearing primarily in two types of evaluations: Level One “smile sheets”Skills evaluations The same criticisms apply to both usages, though the proposed solutions are different. In Level One evaluations rating scales are primarily used as responses to statements about a learning …

Does Question Order Affect Exam Difficulty?

In my last blog post I made the assertion that randomizing questions does not affect exam difficulty. In other words, if Person A gets a set of exam questions in one order and Person B gets the same questions but in a different order, I asserted that both exams will be of the same difficulty. …

Randomizing vs. Subsetting Exam Questions

In discussions with clients I notice that some people inadvertently use the terms randomizing and subsetting interchangeably, though they are really quite different with entirely different consequences for exam validity. Randomization means that all test takers get the same questions but in different order. Subsetting means that each test taker gets a different subset of …

Validating Questions and Exams. How Much is Enough?

I do a lot of exam validation and one of the questions I am frequently asked is: Do we need to validate ALL of our questions and exams? The answer is: It depends on what you are using the questions and exams for. Based on many years of experience here are my best practice guidelines: …

Storytelling Works. Doesn’t It?

It is conventional wisdom in the learning community that storytelling is an effective form of training. I think we all have an intuitive sense that storytelling engages the emotions of our learners and helps them understand concepts in context. But what does the research show? If you google “storytelling and learning” you will find the …

Did Someone Actually Invent the Multiple-Choice Question?

Often, when a product or service has existed throughout our entire lifetime it is hard to imagine what life was like before its invention. I remember asking my parents what they did before there were TVs. Similarly, children growing up today will have a hard time conceiving of a world before the mobile phone or …

Webinars and Traditional eLearning are Fine, But They Are Not Enough

Prior to the current pandemic, corporate learning was a mixture of live and virtual training. Today, and for the foreseeable future, all learning is and will be virtual. So, it is important to know that while webinars and traditional eLearning can be effective, they are not enough to ensure knowledge retention and real-world application. You …

It’s Now More Important Than Ever To Validate Assessments

Now that all learning has moved online (traditional eLearning, microlearning. virtual coaching, virtual classrooms, etc.) it’s important to look at the flip side of the learning coin: Assessments. As we’ve discussed many times in this blog there are two types of assessments: Assessments for learning -- sometimes called formative assessments Assessments of learning – sometimes …

A Practical Guide to Using Microlearning to Create a Learning Campaign

About six months ago we ran a series of blog posts on Learning Campaigns (here, here here and here). The central idea is that for microlearning to be effective you need to combine the microlearning event with an opportunity for retrieval practice. And, depending on your learning goals, you might want to include a final …

Just Because It’s Short Doesn’t Mean You Will Remember It

Here at Intela, we are big fans of microlearning. There is a huge amount of research demonstrating that presenting small nuggets of information spaced over time improves learning retention. Much of this is due to limitations on our working memory: Working memory is short-term (no more than a few minutes) and limited – capable of …