How Do We Know if a Training Technique is Effective?

What training methodologies do you use? Do you know if they are effective compared to other ways of delivering the same training? Unfortunately most of us don’t know. We base our training decisions on gut feels, common sense, what we have “always” done, what others do, and a general idea of “what seems to work.” But if training is to be more than doing “what seems to work” we need to ground our decisions in research-based science.  For this reason I spend a lot of time reading research papers. Now, I understand that most trainers don’t. It takes time and let’s face it, research papers are not the most exciting reading. Also, they are full of arcane statistics that can be difficult to understand.

I rarely write about statistics, even though understanding statistics at some level is important to anyone who wants to move beyond common sense and discover what actually works. So, this week, knowing my readership will plunge temporarily, I will address a relatively simple but very important topic in statistics called effect size.

If you read any research papers at all you will frequently see references to effect size. Fortunately effect size is not that difficult to understand. Effect size helps us answer a very important question: If I apply a “treatment” (e.g. a new training technique) to a group how much of a difference will I see from the control group that doesn’t get this “treatment?”

Effect size can be negative (that’s not desirable at all!). It can be zero or close to zero (no effect). Or, it can be positive. In general of course the bigger the effect size the better.

Effect size is a measure of how many “standard deviations” the mean of the treatment group is above the mean of the control group (assuming the treatment effect is positive). What does this mean in practice? To make this simple here are some examples:

  • An effect size of .5 means that the average person in the treatment group scored better than 69% of the people in the control group.
  • An effect size of 1 means that the average person in the treatment group scored better than 84% of the people in the control group.
  • An effect size of 1.6 means that the average person in the treatment group scored better than 95% of the people in the control group.
  • An effect size of 2 means that the average person in the treatment group scored better than 98% of the people in the control group.

So, what’s a ‘big” effect size? This is an over simplification, but with lots of caveats:

  • An effect size of .2 or lower is small
  • An effect size of .5 is moderate
  • An effect size of .8 or greater is large

So, the next time someone is promoting a new learning methodology ask if there is any research that demonstrates that it works.

Data is not the answer to all questions and I’m not against common sense, but remember, common sense tells us that the world is flat and the Sun rotates around the Earth.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s