Tag Archive for: elearning guild

Thanks to the eLearning Guild for Guild Master Award

, ,

I want to thank David Kelly and the eLearning Guild for awarding me the prestigious title of Guild Master.

Guild Masters including an amazing list of folks, including lots of research-to-practice legends like Ruth Clark, Julie Dirksen, Clark Quinn, Jane Bozarth, Karl Kapp, and others who utilize research-based recommendations in their work.

Delighted to be included!

 

 

What Learning Evaluation Metrics are We Using? Indexed by LTEM!

, ,

For years, we have used the Kirkpatrick-Katzell Four-Level Model to evaluate workplace learning. With this taxonomy as our guide, we have concluded that the most common form of learning evaluation is learner surveys, that the next most common evaluation is learning, then on-the-job behavior, then organizational results.

The truth is more complicated.

In some recent research I led with the eLearning Guild and Jane Bozarth, we used the LTEM model to look for further differentiation. We found it.

Here’s some of the insights from the graphic above:

  • Learner surveys are NOT the most common form of learning evaluation. Program completion and attendance are more common, being done on most training programs in about 83% of organizations.
  • Learners surveys are still very popular, with 72% of respondents saying that they are used in more than one-third of their learning programs.
  • When we measure learning, we go beyond simple quizzes and knowledge checks.
    • Tier 5 assessments, measuring the ability to make realistic decisions, were reported by 24% of respondents to be used in more than one-third of their learning programs.
    • Tier 6 assessments, measuring realistic task performance (during learning), were reported by about 32% of respondents to be used in more than one-third of their learning programs.
    • Unfortunately, we messed up and forgot to include an option on Tier 4 Knowledge questions. However, previous eLearning Guild research in the 2007, 2008, and 2010 found that the percentage of respondents who reported that they measured memory recall of critical information was 60%, 60%, and 63% respectively.
  • Only about 20% of respondents said their organizations are measuring work performance.
  • Only about 16% of respondents said their organizations are measuring the organizational results from learning.
  • Interestingly, where the Four-Level Model puts all types of Results into one bucket, the LTEM framework encourages us to look at other results besides business results.
    • About 12% said their organizations were looking at the effect of the learning on the learner’s success and well-being.
    • Only about 3% said they were measuring the effects of learning on coworkers/family/friends.
    • Only about 3% said they were measuring the effects of learning on the community or society (as has been recommended by Roger Kaufman for years).
    • Only about 1% reported measuring the effects of learning on the environs.

 

Opportunities

The biggest opportunity—or the juiciest low-hanging fruit—is that we can stop just using Tier-1 attendance and Tier-3 learner-perception measures.

We can also begin to go beyond our 60%-rate in measuring Tier-4 knowledge and do more Tier-5 and Tier-6 assessments. As I’ve advocated for years, Tier-5 assessments using well-constructed scenario-based questions are the perfect balance of power and cost. They are aligned with the research on learning, they have moderate costs in terms of resources, and learners see them as challenging and interesting rather than punitive and unhelpful like they often see knowledge checks.

We can also begin to emphasize more Tier-7 evaluations. Shouldn’t we know whether our learning interventions are actually transferring to the workplace? The same is true for Tier-8 measures. We should look for strategic opportunities here—being mindful to the incredible costs of doing good Tier-8 evaluations. We should also consider looking beyond business results—as these are not the only effects our learning interventions are having.

Finally, we can use LTEM to help guide our learning-development efforts and our learning evaluations. By using LTEM, we are prompted to see things that have been hidden from us for decades.

 

The Original eLearning Guild Report

To get the original eLearning Guild report, click here.

 

The LTEM Model

To get the LTEM Model and the 34-page report that goes with it, click here.

New Research on Learning Evaluation — Conducted with The eLearning Guild

,

Released Today: Research Report on Learning Evaluation Conducted with The eLearning Guild.

Report Title: Evaluating Learning: Insights from Learning Professionals.

I am delighted to announce that a research effort that I led in conjunction with Dr. Jane Bozarth and the eLearning Guild has been released today. I’ll be blogging about our findings over the next couple of months.

This is a major report — packed into 39 pages — and should be read by everyone in the workplace learning field interested in learning evaluation!

Just a teaser here:

We asked folks to consider the last three learning programs their units developed and to reflect on the learning-evaluation approaches they used.

While a majority were generally happy with their evaluation methods on these recent learning programs, about 40% where dissatisfied. Later, in a more general question about whether learning professionals are able to do the learning measurement they want to do, fully 52% said they were NOT able to do the kind of evaluation they thought was right to do.

In the full report, available only to Guild members, we dig down and explore the practices and perspectives that drive our learning-evaluation efforts. I encourage you to get the full report, as it touches on the methods we use, how we communicate with senior business leaders, what we’d like to do differently, and what we think we’re good at. Also, the report concludes with 12 powerful action strategies for getting the most out of our learning-evaluation efforts.

You can get the full report by clicking here.

 

 

Learning Solutions 2014 — Come Join Me!

, ,

If you're going to the eLearning Guild's Learning Solutions Conference this coming week in Orlando, come join me–and say hello!

I'll be speaking in three sessions:

Featured Session (F2)
Subscription Learning: A Fundamentally Different Form of eLearning

Time: Wednesday March 19, 10:45AM

Details on the session

Slides for the session

Over 300 people are expected to attend. Get there early for a good seat!

Concurrent Session (105)
Serious eLearning Manifesto (Also with Clark Quinn and Michael Allen)

Time: Wednesday March 19, 1:00PM

Details on the session

We will hand out paper version of the Manifesto at the session (there are no slides)

Morning Buzz (MB31)

Time: Thursday March 20, 7:15AM

A casual conversation about the eLearning Manifesto and Instructional Design

Note: Look for Clark Quinn, or Michael Allen's name (as mine is not listed), but I'll be there!

 

Featured Session this coming week at the eLearning Guild’s Learning Solutions Conference

This week, on March 19th, Wednesday at 10:45 AM, Will Thalheimer (that's me) will be speaking on Subscription Learning at the eLearning Guild's Learning Solutions Conference.

300 people are expected to attend, so come early for your seat!!

Details: Learning Solutions Featured Session F2

Slides for the session

Google Ads for Subscription Learning

Does Subscription Learning have legs?

It's too early to tell, but the eLearning Guild is buying Google Ads on "subscription learning."

Capture of Guild Advertisement

The Pluses and Minuses of Social Media and User-generated Content

At the recent eLearning Guild conference in Orlando, I was asked to lead an Espresso Cafe roundtable discussion on a topic of interest.

My topic: The Pluses and Minuses of Social Media and User-generated Content.

I promised folks from my three sessions that I'd post all the results. Here they are:

Plusses:

  1. Users engaged.
  2. Relevant to the users.
  3. Not-distracting, real-world.
  4. Enables learning when training experts not available.
  5. Can augment online courses.
  6. Can capture water-cooler talk (that would have happened anyway).
  7. Opportunity to debunk inaccuracies.
  8. Capture institutional knowledge.
  9. Enables the use of internal experts for informal learning.
  10. Because informal, can be more comfortable to use for people of different languages and/or cultures. Or different socio-economic groups as well.
  11. More of an equal exchange. Leveling the playing field. Creating more democratic or egalitarian organizations.
  12. Novel, interesting.
  13. Quick feedback on what doesn't work.
  14. Not corporate-down, so more likely to be attended to without skepticism, jadedness, etc.
  15. Opportunity to connect with customers.
  16. Keep up with younger workers coming in.
  17. Headquarters experts may not be as trusted as those who work on the ground.
  18. Timely, instant updates.
  19. Get details from someone who actually does the job.
  20. Emotional connection.
  21. Convenience.
  22. No geographic boundaries.
  23. RSS feeds enables more targeted info.
  24. Employees may be able to affect policy.
  25. Could make us improve our policies for fear of law suits. (Like this: stuff that's posted can be used in court. Organization then has impetus to make changes quickly).
  26. Questions coming first is a good learning design.
  27. Can give organization more of a sense of what's going on in the field.
  28. Cheap.
  29. Builds community if people are tackling serious issues together.
  30. Feeling engaged.
  31. Employees have instant access to experts.
  32. Another data source.
  33. Develop connections. Know who knows who AND who knows what.
  34. Enables virtual relationships.
  35. More reflective–learners have to reflect to write, to learn deeper.
  36. Wisdom of the crowd.
  37. Opens up links to other things. Sets agenda, letting people know that there are other things.
  38. Generate buzz.
  39. Smile sheets shared. (Rate my teacher. Rate my professor).
  40. Best practices are distributed.
  41. Will make things easier. Info at fingertips.

Minuses:

  1. Might have to get used to it.
  2. How do you make it usable?
  3. Duplicate information.
  4. How to make pertinent information instantly accessible.
  5. Opening up floodgates.
  6. Cultural hurdles and disconnects.
  7. Competes with other channels of information.
  8. Perhaps top-level buy-in is required.
  9. A big distraction. Time user.
  10. Productivity drain.
  11. One more thing to do.
  12. We are still learning how to utilize wisely.
  13. May need support, maintenance, and the resources thereof.
  14. Information may not translate to behavior without directed support.
  15. How to confirm validity of content.
  16. Info can be used in lawsuits.
  17. Is the time beneficial?
  18. Danger of noise. Hard to get to best information.
  19. Time to create.
  20. Hard to measure. Maybe we're fooling ourselves.
  21. Could be incorrect/bad information.
  22. Could be offensive information.
  23. Must bring people up-to-speed on technology.
  24. Can create cliques.
  25. Time suck–filling up on candy.
  26. Dangers of giving censors power.
  27. Do these media self-select different types of people, biasing information gathered?
  28. Time is our most limited resource. The key organizational-productivity leverage point.
  29. Often implemented without planning, no marketing, no preparation, etc.
  30. Sometimes systems have no purpose. So costs/time not parlayed to maximum effect.
  31. Unnatural groups may not work, may have difficulties.
  32. One or a few can take over.
  33. Example: General in military told story of how soldiers posted how to defuse an IED. Info was wrong. 2 died. Enemies can use information too.
  34. Many see this as the be-all end-all, creating big blind spots, overzealous implementation, poor planning, poor focus.
  35. Potential permanence of information and/or systems.
  36. Personal vs. work issues may arise.

Thanks to all the folks who contributed to my discussions. It was kind of hard to hear, but here are the names to thank: Nancy, Leslie, Terra, Pat, Sonya, Betsy, Michael, David, David, Ann, Joyce, Nancy, Chris, Chris, Richard, John, Susan, Paula, John.

eLearning Guild offering $400 early-bird discount!!

The eLearning Guild is offering a $400 early-bird discount if you register for their March Annual Gathering by December 19th. Check it out.

Note: I'll be presenting a workshop (with Roy Pollock) on Learning Measurement, and speaking several other times, so this conference is well worth your while. AND, by saving $400, you can easily afford our symposium.