LTEM, the Learning-Transfer Evaluation Model, was designed as an alternative to the Kirkpatrick-Katzell Four-Level Model of learning evaluation. It was designed specifically to better align learning evaluation with the science of human learning. One way in which LTEM is superior to the Four-Level Model is in the way it highlights gradations of learning outcomes. Where the Four-Level model crammed all “Learning” outcomes into one box (that is, “Level 2”), LTEM separates learning outcomes into Tier-4 Knowledge, Tier-5 Decision-Making Competence, and Tier-6 Task Competence. This simple, yet incredibly powerful categorization, changes everything in terms of learning evaluation. First and foremost, it pushes us to go beyond inconsequential knowledge checks in our learning evaluations (and in our learning designs as well). To learn more about how LTEM creates additional benefits, you can click on this link, where you can access the model and a 34-page report for free, compliments of  me, Will Thalheimer, and Work-Learning Research, Inc.

Using LTEM in Credentialing

LTEM can also be used in credentialing—or less formally in specifying the rigorousness of our learning experiences. So for example, if our training course only asks questions about terminology or facts in its assessments, than we can say that the course provides a Tier-4 credential. If our course asks learners to successfully complete a series of scenario-based decisions, we can say that the course provides a Tier-5 credential.

Wow! Think of the power of naming the credential level of our learning experiences. Not only will it give us—and our business stakeholders—a clear sense of the strength of our learning initiatives, but it will drive our instructional designs to meet high standards of effectiveness. It will also begin to set the bar higher. Let’s admit a dirty truth. Too many of our training programs are just warmed-over presentations that do very little to help our learners make critical decisions or improve their actual skills. By focusing on credentialing, we focus on effectiveness!

 

Using LTEM Credentialing at Work-Learning Research

For the last several months, I’ve been developing an online course to teach learning professionals how to transform their learner surveys into Performance-Focused Smile Sheets. As part of this development process, I realized that I needed more than one learning experience—at least one to introduce the topic and one to give people extensive practice. I also wanted to provide people with a credential each time they successfully completed a learning experience. Finally, I wanted to make the credential meaningful. As the LTEM model suggests, attendance is NOT a meaningful benchmark. Neither is learner satisfaction. Nor is knowledge regurgitation.

Suddenly, it struck me. LTEM already provided a perfect delineation for meaningful credentialing. Tier-5 Decision-Making Competence would provide credentialing for the first learning experience. For people to earn their credential they would have to perform successfully in responding to realistic decision-making scenarios. Tier-6 Task Competence would provide credentialing for the second, application-focused learning experience. Additional credentials would only be earned if people could show results at Tier-7 and/or Tier-8 (Transfer to Work Performance and associated Transfer Effects).

 

 

These learning experiences are not quite ready for enrollment—they should be ready within a week or two—but if you’d like to be informed of their release, please sign up for Will’s News. You can do that here, while also seeing silly pictures of me.

 

How You Can Use LTEM Credentialing to Assess Learning Experiences that Don’t Use LTEM

LTEM is practically brand new, having only been released to the public a year ago. So, while many organizations are gaining a competitive advantage by exploring its use, most of our learning infrastructure has yet to be transformed. In this transitional period, each of us has to use our wisdom to assess what’s already out there. How about you give it a try?

Two-Day Classroom Workshop — What Tier Credential?

What about a two-day workshop that gives people credit for completing the experience? Where would that be on the LTEM framework?

Here’s a graphic to help. Or you can access the full model by clicking here.

The two-day workshop would be credentialed at a Tier-1 level, signifying that the experience credentials learners by measuring their attendance or completion.

Two-Day Classroom Workshop with Posttest — What Tier Credential?

What if the same two-day workshop also added a test focused on whether the learners understood the content—and provided the test a week after the program. Note that in the LTEM model, credentialing is encouraged at Tiers 4, 5, and 6 to include assessments that show learners are able to remember, not just comprehend in the short term.

If the workshop added this posttest, we’d credential it at Tier-4, Knowledge Retention.

Half-Day Online Program with Performance-Focused Smile Sheet — What Tier Credential?

What if there was a half day workshop that used one of my Performance-Focused Smile Sheets to evaluate success. At what Tier would this be credentialed?

It would be credentialed at Tier-3, or Tier-3A if we wanted to delineate between learner surveys that assess learning effectiveness and those that don’t.

Three-Session Online Program with Traditional Smile Sheet — What Tier Credential?

This format—using three 90-minute sessions with a traditional smile sheet—is the most common form of credentialing in the workplace learning industry right now. Go look around at those that are providing credentials. They are providing credentials using relatively short presentations and a smile sheet at the end. If this is what they provide, what credentialing Tier do they deserve? Tier-3 or Tier-3B! That’s right! That’s it. They only tell us that learners are satisfied with the learning experience. They don’t tell us whether they can make important decisions or whether they can utilize new skills.

What is this credential really worth?

You can decide for yourself, but I think it could be worth more, if only those making the money provided credentialing at Tier-5, Tier-6, and beyond.

With LTEM we can begin to demand more!

 

Work-Learning Research and Will Thalheimer can Help!

People tell me I need to stop giving stuff away for free, or at least I ought to be more proactive in seeking customers. So, this is a reminder that I am available to help you improve your learning and learning evaluation strategies and tactics. Please reach out to me at my nifty contact form by clicking here.

Series of Four Interviews

I was recently interviewed by Jeffrey Dalto of Convergence Training. Jeffrey is a big fan of research-based practice. He did a great job compiling the interviews.

Click on the title of each one to read the interview:

Anders Ericsson and Robert Pool — one of whom is the world's leading expert on how expertise develops (Ericsson) — have critiqued Malcolm Gladwell's popularization of the 10,000 Hours Rule in a Salon article, adapted from their new book, Peak: Secrets from the New Science of Expertise.

 

 

 Here are the main points from their article:

  1. "Gladwell did get one thing right, and it is worth repeating because it’s crucial: becoming accomplished in any field in which there is a well-established history of people working to become experts requires a tremendous amount of effort exerted over many years. It may not require exactly ten thousand hours, but it will take a lot."
  2. "There is nothing special or magical about ten thousand hours." It can be more or less. Indeed, in some fields it may take twice as long to reach world-class status.
  3. The number of hours to become an expert varies from field to field.
  4. It's not just about practice (or time spent in an activity). Rather, it is about a very specific form of practice — "deliberate practice, which involves constantly pushing oneself beyond one’s comfort zone, following training activities designed by an expert to develop specific abilities, and using feedback to identify weaknesses and work on them."
  5. There are zero research studies that show that anyone who puts in some requisite number of hours (be it 10,000 or less or more), will achieve preeminent expertise. And, let me add my conclusion: There may be — indeed there are likely to be — other factors that influence the development of expertise, including such things as innate abilities, health, environmental stressors, related experiences, nurturance, et cetera. As the authors stress, not everyone can become an expert in a particular field.
  6. Almost always, people can radically improve their performance in a skill through deliberate practice.

Anders Ericsson is amazing — having been doing great research for decades, putting in certainly more than 10,000 hours, I must think. I've just ordered the book, and I recommend that you order it too!

And here is a nice audio clip with Ericsson.

 

Next Thursday March 10th, I'll be speaking on Performance-Focused Smile Sheets at the Charlotte, North Carolina chapter of ISPI.

Click here for the details…

Wow! What a week! I published my first book on Tuesday and have been hearing from well wishers ever since.

Here are some related links you might find interesting:

And here are some random visuals, but maybe related.

  • In the rush of purchases, Amazon briefly ran out of my book, telling folks they'd have to wait 1-3 weeks — their signal that they have no stock. Later, they found some copies…but the genie was out of the bottle.

Amazon showing sold out

 

 

  • The animal kingdom seems to be behind the book:

Bozarth's Corgi

Olah's Cat

 

 

  • Even one of the United States' presidential candidates has spoken up:

Bernie2

 

What a week!

 

Thank you all!!

 

= Will Thalheimer

 

There appears to be more and more momentum for CPA's to be able to earn credentials for micro-learning.

Click to read more…

Researchers at MIT have coined the term "Wait-Learning" — learning at a time when a person would otherwise be waiting, and hence wasting time… Their research work involves foreign-language learning.

They surmised that instant messaging provided an excellent application to test whether a program could enable wait-learning for language vocabulary. Often while chatting, conversations feels asynchronous; the person who just sent a message waits for a reply.

They built a program, called WaitChatter, that works in Google Chat. It's an experimental program, only able to teach Spanish and French vocabulary to English speakers. They experimented with WaitChatter and got positive results, which they published online in an ACM publication.

Here's what the authors said about the amount of learning:

"In just two weeks of casual usage, participants were on average able to recall 57 new words, equivalent to approximately four words per day."

TechCrunch has a nice article explaining how WaitChatter works.

WaitChatter is not ready for prime time. It's an experimental program and it only works in Chrome and only if you disable Google Hangouts and go back to Google Chat. Still, several concepts about WaitChatter and the concept of wait-learning are intriguing:

  1. Wait-Learning, though not an original concept, is a good one…We learning professionals ought to figure out how to maximize efficiencies in this way. Of course, we'll want to make sure that the additional learning doesn't compromise the main task. We know multitasking is illusory, often hurting one task or another, so we'll need to be careful.
  2. Embedding learning opportunities in other applications may enable such efficiencies, if we do it carefully.
  3. Part of the vocabulary learned was learned based on the words that came up in the chat. So for example, if the word "dog" came up in the chat, WordChatter might focus on the Spanish equivalent "el perro." We know from the general research on learning that alignment between the learning context and the performance context produce learning and remembering benefits, and the authors cite research that such contextual learning benefits language learners as well.

 

The swell of interest in short learning nuggets has another article touting its benefits, this one from Training Industry Magazine and Manjit Sekhon (of Intrepid Learning).

The article makes no mention of the spacing effect, but it does talk of threading nuggets in an intentional sequence. The article emphasizes shorter attention spans of learners, and the benefit of keeping people on the job.

More evidence that subscription-learning and shorter learning in general is on the rise.

 

Clark Quinn and I have been grappling with FUN-da-mental issues in the learning space over the years, and we finally decided to publish some of our dialogue.

In the latest conversation, Clark and I discuss how the tools in the learning field often don't send the right messages about how to design learning–that they unintentionally push us toward poor instructional designs.

You can read the discussion on Clark's world-renowned blog by CLICKING HERE.

——

——

——

Or, read an earlier discussion on how professionalized we are by clicking here.

 

Below is the feedback for the Learning-Objective Quick Quiz. If you haven’t taken the quiz yet, I recommend you do so before you read the feedback. Science says you’ll learn more and think more deeply if you do! And don’t you want to think more deeply?

I’m impressed with your persistence!

The feedback file is available by clicking here.

While you’re here, feel free to look around. This is my blog…

= Will Thalheimer