Reporting Level One Evaluations

In our last blog post we pointed out the many problems with rating scales. We hope we convinced you that doing traditional math on these scales is a mathematically invalid exercise. So, the question becomes: How do we report results from Level One and Skills Evaluations? The answer is somewhat different for each. We will look first at Level One Evaluations.

First and foremost, we must acknowledge that much of our thinking in this area is influenced by the work of Dr. Will Thalheimer and his book “Performance-Focused Smile Sheets:”

We can’t recommend this book too strongly — with one caveat, and that caveat comes from Will: He has an expanded second edition of the book arriving this spring and he recommends waiting for this edition.

The bulk of Will’s book is a discussion of how to improve the actual questions you ask on the smile sheets. It’s not our purpose to discuss this important topic here. Just buy and read his book. Instead, we are focusing on how to meaningfully report results.

Let’s return to the example from our last post. As a reminder, we pointed out the malpractice involved when we report the results of Level One evaluations as averages:

All three Level One questions produce the same rating average but have very different details. In particular, focus on the question “I was satisfied with the course overall.” Half the learners strongly agreed with this statement, but half strongly disagreed. When we report the average of this question as 3.0 (usually considered “acceptable”), consider the information we have lost! So, how do we not lose it?

Answer: Get rid of the artificial, mathematically incorrect, rating scale (1-5) and do not think of these instruments as Level One Evaluations or Smile Sheets! Think of them as Learner Surveys, because that is what they are. And how do we report on surveys? With bar or pie charts.

So, in the above case, with, for example, 12 learners responding to our survey, we would report the results as follows:

Now we can see the strangely skewed results clearly. Maybe we don’t have a nice, concise (and meaningless!) result like 3.0, but we do have information-rich, actionable results. We are great believers in analytics, but in this case our need to have a simple, single, numeric result leads us astray.