How Intela Helps You Detect Test Collusion (Part One)

For trainers who do remote, unproctored, testing (e.g. testing a distributed sales force) one nagging question is always: “Are my learners cheating?” Usually followed by the question: “If they are, can I tell?” The answers are (in order): maybe and maybe.

It turns out there are three ways to detect cheating: one pretty simple, which is the subject of this blog post, a second, more statistical, but not that difficult, which will be the subject of a future blog post and a third, highly statistical, which is well beyond the capabilities of 99.99999% of the people who read this blog.

What does exam cheating look like? Usually, it involves some form of collusion. In a classroom, collusion can be voluntary (Student A intentionally gives the answers to Student B) or involuntary (Student B copies Student A’s answers). In remote testing collusion is always voluntary, because, in general, the test takers will not physically be together when they take the test.

If the test takers do get together, they can pool their knowledge, and take the test, literally, at the same time, answering the questions together. But this is difficult to do if you have randomized the order of the questions (something we always recommend). If you have randomized the order of the questions, then the test takers will have to take the test sequentially, one after the other.

Or, alternately, if the test takers are not physically together a learner who has already taken the test can help other test takers via a phone call or Skype.

A third scenario, which happens when test creators use the same questions repeatedly over time, is that the questions and answers get shared by the learner community.

No matter how it occurs, when collusion does occur there is often one tell-tale sign: Learners who have help often complete the test much more quickly than learners who do not. (There’s no read and think time.)

Intela provides an easy way to see if this is occurring. For every exam, Intela provides a scatterplot of score vs. elapsed test time. Here’s a sample:

Screen Shot 2017-05-04 at 2.31.55 PM

Along the Y–axis is score and along the X-axis is time on test. This is just a short sample test of five questions, but it illustrates the point. There are two learners who scored 80% in under 20 seconds. That is clearly not possible if the test taker were seeing the questions for the first time and actually had to read the questions and weigh the choices before answering.

If this were an actual exam this would be suspicious behavior. Further, in Intela, rolling over the points on the graph tells you who those test takers are.

So, this doesn’t prove collusion, but it should certainly lead you to investigate further.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s