I’m thrilled to announce that my Gold-Certification Workshop on Performance-Focused Smile Sheets is now open for registration, with access available in about a week on Tuesday May 14 (2019).

This certification workshop is the culmination of years of work and practice. First there was my work with clients on evaluation. Then there was the book. Then I gained extensive experience building and piloting smile sheets with a variety of organizations. I taught classroom and webinar workshops. I spoke at conferences and gave keynotes. And of course, I developed and launched LTEM (The Learning-Transfer Evaluation Model), which is revolutionizing the practice of workplace learning—and providing the first serious alternative to the Kirkpatrick-Katzell Four-Level Model.

Over the last year, I’ve been building an online, asynchronous workshop that was rigorous, comprehensive, and challenging enough to offer a certification. It’s now ready to go!

I’d love if you would enroll and join me and others in learning!

You can learn more about this Gold-Certification Workshop by clicking here.

 

I’ve had the distinct honor of being invited to speak at the Learning Technologies conference in London for three years in a row. This year, I talked about two learning innovations:

  • Performance-Focused Learner Surveys
  • LTEM (The Learning-Transfer Evaluation Model)

It was a hand-raising experience!

Most importantly, they have done a great job capturing my talk on YouTube.

Indeed, although I’ve made some recent improvements in the way I talk about these two learning innovations, the video does an excellent job of capturing some of the main points I’ve been making about the state of learning evaluation and two innovations that are tearing down some of the obstacles that have held us back from doing good evaluation.

Thanks to Stella Collins at Stellar Learning for organizing and facilitating my session!

Special thanks to the brilliant conference organizer and learning-industry influencer Robert Taylor for inviting and supporting me and my work.

Again, click here to see the video of my presentation at Learning Technologies London 2019.

While I was in London a few months ago, where I talked about learning evaluation, I was interviewed by the LearningNews about learning evaluation.

Some of what I said:

  • “Most of us have been doing the same damn thing we’ve always done [in learning evaluation]. On the other hand, there is a breaking of the logjam.”
  • “A lot of us are defaulting to happy sheets, and happy sheets that aren’t effective.”
  • “Do we in L&D have the skills to be able to do evaluation in the first place?…. My short answer is NO WAY!”
  • “We can’t upskill ourselves fast enough [in terms of learning evaluation].

It was a fun interview and LearningNews did a nice job in editing it. Special thanks to Rob Clarke for the interview, organizing, and video work (along with his great team)!!

Click here to see the interview.

I want to thank David Kelly and the eLearning Guild for awarding me the prestigious title of Guild Master.

Guild Masters including an amazing list of folks, including lots of research-to-practice legends like Ruth Clark, Julie Dirksen, Clark Quinn, Jane Bozarth, Karl Kapp, and others who utilize research-based recommendations in their work.

Delighted to be included!

 

 

I’d like to announce that the first certification workshop for my new Work-Learning Academy is almost ready to launch. The first course? Naturally, it’s a course on how to create effective learner surveys—on Performance-Focused Smile Sheets.

I’m thrilled—ecstatic really—because I’ve wanted to do something like this for years and years, but the elements weren’t quite available. I’ve always wanted to provide an online workshop, but the tools tended to push toward just making presentations. As a learning expert, I knew mere presentations—even if they include discussions and some minimal interactions like polling questions—just weren’t good enough to create real learning benefits. I’ve also always wanted a way to provide a meaningful credential—one that was actually worth something, one that went beyond giving people credit for attendance and completion. Finally, I figured out how to bring this all together

And note that LTEM (the Learning-Transfer Evaluation Model), helped me clarify my credentialing strategy. You can read about using LTEM for credentialing here, but, in short, our entry-level certification—our Gold Certification—requires learners to pass a rigorous LTEM Tier-5 assessment, demonstrating competence through realistic decision-making. Those interested in the next level credential—our Master Certification—will have to prove their competence at an LTEM Tier-6 designation. Further certification levels—our Artisan Certification and Research Certification—will require competence demonstrated at Tier-7 and/or Tier-8.

 

For over 20 years, I’ve been plying my research-to-practice craft through Work-Learning Research, Inc. I’m thrilled to announce that I’ll be certifying our first set of Gold Credential professionals within a few months. If you’d like to sign up to be notified when the credential workshop is available—or just learn more—follow this link:

Click here to go to our
Work-Learning Academy information page

My Year In Review 2018—Engineering the Future of Learning Evaluation

In 2018, I shattered my collarbone and lay wasting for several months, but still, I think I had one of my best years in terms of the contributions I was able to make. This will certainly sound like hubris, and surely it is, but I can’t help but think that 2018 may go down as one of the most important years in learning evaluation’s long history. At the end of this post, I will get to my failures and regrets, but first I’d like to share just how consequential this year was in my thinking and work in learning evaluation.

It started in January when I published a decisive piece of investigative journalism showing that Donald Kirkpatrick was NOT the originator of the four-level model; that another man, Raymond Katzell, has deserved that honor all along. In February, I published a new evaluation model, LTEM (The Learning-Transfer Evaluation Model)—intended to replace the weak and harmful Kirkpatrick-Katzell Four-Level Model. Already, doctoral students are studying LTEM and organizations around the world are using LTEM to build more effective learning-evaluation strategies.

Publishing these two groundbreaking efforts would have made a great year, but because I still have so much to learn about evaluation, I was very active in exploring our practices—looking for their strengths and weaknesses. I led two research efforts (one with the eLearning Guild and one with my own organization, Work-Learning Research). The Guild research surveyed people like you and your learning-professional colleagues on their general evaluation practices. The Work-Learning Research effort focused specifically on our experiences as practitioners in surveying our learners for their feedback.

Also in 2018, I compiled and published a list of 54 common mistakes that get made in learning evaluation. I wrote an article on how to think about our business stakeholders in learning evaluation. I wrote a post on one of the biggest lies in learning evaluation—how we fool ourselves into thinking that learner feedback gives us definitive data on learning transfer and organizational results. It does not! I created a replacement for the problematic Net Promoter Score. I shared my updated smile-sheet questions, improving those originally put forth in my award winning book, Performance-Focused Smile Sheets. You can access all these publications below.

In my 2018 keynotes, conference sessions, and workshops, I recounted our decades-long frustrations in learning evaluation. We are clearly not happy with what we’ve been able to do in terms of learning evaluation. There are two reasons for this. First, learning evaluation is very complex and difficult to accomplish—doubly so given our severe resource constraints in terms of both budget and time. Second, our learning-evaluation tools are mostly substandard—enabling us to create vanity metrics but not enabling us to capture data in ways that help us, as learning professionals, make our most important decisions.

In 2019, I will continue my work in learning evaluation. I still have so much to unravel. If you see a bit of wisdom related to learning evaluation, please let me know.

Will’s Top Fifteen Publications for 2018

Let me provide a quick review of the top things I wrote this year:

  1. LTEM (The Learning-Transfer Evaluation Model)
    Although published by me in 2018, the model and accompanying 34-page report originated in work begun in 2016 and through the generous and brilliant feedback I received from Julie Dirksen, Clark Quinn, Roy Pollock, Adam Neaman, Yvon Dalat, Emma Weber, Scott Weersing, Mark Jenkins, Ingrid Guerra-Lopez, Rob Brinkerhoff, Trudy Mandeville, and Mike Rustici—as well as from attendees in the 2017 ISPI Design-Thinking conference and the 2018 Learning Technologies conference in London. LTEM is designed to replace the Kirkpatrick-Katzell Four-Level Model originally formulated in the 1950s. You can learn about the new model by clicking here.
  2. Raymond Katzell NOT Donald Kirkpatrick
    Raymond Katzell originated the Four-Level Model. Although Donald Kirkpatrick embraced accolades for the Four-Level Model, it turns out that Raymond Katzell was the true originator. I did an exhaustive investigation and offered a balanced interpretation of the facts. You can read the original piece by clicking here. Interestingly, none of our trade associations have reported on this finding. Why is that? LOL
  3. When Training Pollutes. Our Responsibility to Lessen the Environmental Damage of Training
    I wrote an article and placed it on LinkedIn and as far as I can tell, very few of us really want to think about this. But you can get started by reading the article (by clicking here).
  4. Fifty-Four Mistakes in Learning Evaluation
    Of course we as an industry make mistakes in learning evaluation, but who knew we made so many? I began compiling the list because I’d seen a good number of poor practices and false narratives about what is important in learning evaluation, but by the time I’d gotten my full list I was a bit dumbstruck by the magnitude of problem. I’ve come to believe that we are still in the dark ages of learning evaluation and we need a renaissance. This article will give you some targets for improvements. Click here to read it.
  5. New Research on Learning Evaluation — Conducted with The eLearning Guild
    The eLearning Guild and Dr. Jane Bozarth (the Guild’s Director of Research) asked me to lead a research effort to determine what practitioners in the learning/elearning field are thinking and doing in terms of learning evaluation. In a major report released about a month ago, we reveal findings on how people feel about the learning measurement they are able to do, the support they get from their organizations, and their feelings about their current level of evaluation competence. You can read a blog post I wrote highlighting one result from the report—that a full 40% of us are unhappy with what we are able to do in terms of learning evaluation. You can access the full report here (if you’re a Guild member) and an executive summary. Also, stay tuned to my blog or signup for my newsletter to see future posts about our findings.
  6. Current Practices in Gathering Learner Feedback
    We at Work-Learning Research, Inc. conducted a survey focused on gathering learner feedback (i.e., smile sheets, reaction forms, learner surveys) that spanned 2017 and 2018. Since the publication of my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, I’ve spent a ton of time helping organizations build more effective learner surveys and gauging common practices in the workplace learning field. This research survey continued that work. To read my exhaustive report, click here.
  7. One of the Biggest Lies in Learning Evaluation — Asking Learners about Level 3 and 4 (LTEM Tiers 7 and 8)
    This is big! One of the biggest lies in learning evaluation. It’s a lie we like to tell ourselves and a lie our learning-evaluation vendors like to tell us. If we ask our learners questions that relate to their job performance or the organizational impact of our learning programs we are NOT measuring at Kirkpatrick-Katzell Level 3 or 4 (or at LTEM Tiers 7 and 8), we are measuring at Level 1 and LTEM Tier 3. You can read this refutation here.
  8. Who Will Rule Our Conferences? Truth or Bad-Faith Vendors?
    What do you want from the trade organizations in the learning field? Probably “accurate information” is high on your list. But what happens when the information you get is biased and untrustworthy? Could. Never. Happen. Right? Read this article to see how bias might creep in.
  9. Snake Oil. The Story of Clark Stanley as Preface to Clark Quinn’s Excellent Book
    This was one of my favorite pieces of writing in 2018. Did I ever mention that I love writing and would consider giving this all up for a career as a writer? You’ve all heard of “snake oil” but if you don’t know where the term originated, you really ought to read this piece.
  10. Dealing with the Emotional Readiness of Our Learners — My Ski Accident Reflections
    I had a bad accident on the ski slopes in February this year and I got thinking about how our learners might not always be emotionally ready to learn. I don’t have answers in this piece, just reflections, which you can read about here.
  11. The Backfire Effect. Not the Big Worry We Thought it was (for Those Who Would Debunk Learning Myths)
    This article is for those interested in debunking and persuasion. The Backfire Effect was the finding that trying to persuade someone to stop believing a falsehood, might actually make them more inclined to believe the falsehood. The good news is that new research showed that this worry might be overblown. You can read more about this here (if you dare to be persuaded).
  12. Updated Smile-Sheet Questions for 2018
    I published a set of learner-survey questions in my 2016 book, and have been working with clients to use these questions and variations on these questions for over two years since then. I’ve learned a thing or two and so I published some improvements early last year. You can see those improvements here. And note, for 2019, I’ll be making additional improvements—so stay tuned! Remember, you can sign up to be notified of my news here.
  13. Replacement for NPS (The Net Promoter Score)
    NPS is all the rage. Still! Unfortunately, it’s a terribly bad question to include on a learner survey. The good news is that now there is an alternative, which you can see here.
  14. Neon Elephant Award for 2018 to Clark Quinn
    Every year, I give an award for a great research-to-practice contribution in the workplace learning field. This year’s winner is Clark Quinn. See why he won and check out his excellent resources here.
  15. New Debunker Club Website
    The Debunker Club is a group of people who have committed to debunking myths in the learning field and/or sharing research-based information. In 2018, working with a great team of volunteers, we revamped the Debunker Club website to help build a community of debunkers. We now have over 800 members from around the world. You can learn more about why The Debunker Club exists by clicking here. Also, feel free to join us!

 

My Final Reflections on 2018

I’m blessed to be supported by smart passionate clients and by some of the smartest friends and colleagues in the learning field. My Work-Learning Research practice turned 20 years old in 2018. Being a consultant—especially one who focuses on research-to-practice in the workplace learning field—is still a challenging yet emotionally rewarding endeavor. In 2018, I turned my attention almost fully to learning evaluation. You can read about my two-path evaluation approach here. One of my research surveys totally flopped this year. It was focused on the interface between us (as learning professionals) and our organizations’ senior leadership. I wanted to know if what we thought senior leadership wanted was what they actually wanted. Unfortunately, neither I nor any of the respondents could entice a senior leader to comment. Not one! If you or your organization has access to senior managers, I’d love to partner with you on this! Let me know. Indeed, this doesn’t even have to be research. If your CEO would be willing to trade his/her time letting me ask a few questions in exchange for my time answering questions about learning, elearning, learning evaluation, etc., I’d be freakin’ delighted! I failed this year in working out a deal with another evaluation-focused organization to merge our efforts. I was bummed about this failure as the synergies would have been great. I also failed in 2018 to cure myself of the tendency to miss important emails. If you ever can’t get in touch with me, try, try again! Thanks and apologies! I had a blast in 2018 speaking and keynoting at conferences—both big and small conferences. From doing variations on the Learning-Research Quiz Show (a rollicking good time) to talking about innovations in learning evaluation to presenting workshops on my learning-evaluation methods and the LTEM model. Good stuff, if a ton of work. Oh! I did fail again in 2018 turning my workshops into online workshops. I hope to do better in 2019. I also failed in 2018 in finishing up a research review of the training transfer research. I’m like 95% done, but still haven’t had a chance to finish.

2018 broke my body, made me unavailable for a couple of months, but overall, it turned out to be a pretty damn good year. 2019 looks promising too as I have plans to continue working on learning evaluation. It’s kind of interesting that we are still in the dark ages of learning evaluation. We as an industry, and me as a person, have a ton more to learn about learning evaluation. I plan to continue the journey. Please feel free to reach out and let me know what I can learn from you and your organization. And of course, because I need to pay the rent, let me say that I’d be delighted if you wanted me to help you or your organization. You can reach me through the Work-Learning Research contact form.

Thanks for reading and being interested in my work!!!

Here is list of my public events for the next few months, including conference events, workshops, webinars, and research surveys.

 

OPEN NOW—Research Survey with the eLearning Guild

I’m partnering with the eLearning Guild to conduct a survey on learning-evaluation practices. I would greatly appreciate your time on this—and we estimate it will take you maybe five minutes.

https://www.surveygizmo.com/s3/4568935/Learning-Evaluation-Your-Goals-and-Concerns

 

VIDEO JUST RELEASED—Nigel Paine Interviews Will Thalheimer on LearningNowTV

LearningNowTV is a great resource broadcasting from the UK, and they’ve asked me to join them as a regular guest. I’m delighted.

Our first effort came out pretty good. See if you can see the minor hiccups, including me never saying hello, goodbye, or thank you. LOL

Good content though! I talk about LTEM, the Kirkpatrick-Katzell model, and more. Check it out:

https://learningnow.tv/watch/programme-46—30-august-2018/will-thalheimer-on-course-evaluation-kirkpatrick-and-the-ltem-model.html

 

ONLY A FEW DAYS LEFT TO REGISTER—Kansas City, ATD Conference, Thursday Sept 27

I’ll be delivering the closing keynote, but there is so much more to learn at this regional gem of a conference!

Title of my Keynote: Training Quiz Show: Secrets from the Learning Research (and the earth-shaking LTEM too!)

Last day to register is just two days away! Register here: https://tdkc.org/event-3031307

 

ATLANTA ISPI CHAPTER MEETING—Atlanta Georgia, Thursday Night, November 8th

The ISPI Atlanta chapter invited me to join them for two whirlwind sessions, this one on Thursday night and the following day for a morning workshop (see next item).

Title: Dr. Thalheimer’s Learning Medicine Show and Research Palooza

Registration Information Coming Soon!

 

ATLANTA ISPI WORKSHOP ON LTEM—Atlanta Georgia, Friday Morning, November 9th

This will be my first public workshop on LTEM, the Learning-Transfer Evaluation Model. I’ve spoken about LTEM several times over the last year, but this will be the first in-depth workshop. A half day NOT to be missed!

Registration Information Coming Soon!

 

ELEARNING GUILD COMPLIANCE CONFERENCE (ONLINE)—November 14

The eLearning Guild is hosting a fascinating conference on Compliance Training, and I’ll be talking about how to get valid data.

Title: Getting Valid Data on Your Compliance Training (Not as Easy As it Looks)! My session information.

Register Now: https://www.elearningguild.com/content/5358/compliance-training-summit-2018-home

There’s a whole host of great presenters for this two-day summit. Check it out!

 

ISPI BABS CHAPTER (ONLINE WEBINAR)—November 15

ISPI BABS CHAPTER, which includes geographically close-but-not-contiguous Bay Area and Boise State, always hosts interesting programs, including something called the Oyster-Barrel.

I’ll be presenting a webinar on LTEM (The Learning-Transfer Evaluation Model), which unlike oysters, is guaranteed to act like an aphrodisiac for learning professionals.

Title of Webinar: The Learning-Transfer Evaluation Model (LTEM): A Research-Inspired Alternative to the Four Levels.

Here’s a link to register: https://www.ispibabs.org/event-2934366

 

BERLIN GERMANY OEB CONFERENCE WORKSHOP—December 5 Afternoon

Half-day workshop title: Getting Radically Improved Data from Learner Feedback

Sign up for Workshops at: https://oeb.global/programme#workshops. Mine is Workshop A9.

 

BERLIN GERMANY OEB CONFERENCE—December 6 Spotlight Session

A quick 30-minute version of the The Learning Research Quiz Show.

 

 

 

Famous Song Lyrics:

Kansas City, Kansas City Here I Come!
They have a crazy way of learning there, you know you gotta get you some!

I’ll be keynoting a the ATD Kansas City Chapter’s Annual Conference in a few weeks. If you’re in the area, or want to visit please come join me and the local Kansas City area learning professionals.


Click here to sign up for the conference…

 

Thursday, September 27, 2018
  • 8:30 AM – 4:30 PM
  • Johnson County Community College
  • 12345 College Blvd, Overland Park, KS 66210

 

My research-and-consulting practice, Work-Learning Research, was 20 years old last Saturday. This has given me pause to reflect on where I’ve been and how learning research has involved in the past two decades.

Today, as I’m preparing a conference proposal for next year’s ISPI conference, I found an early proposal I put together for the Great Valley chapter of ISPI to speak at one of their monthly meetings back in 2002. I don’t remember whether they actually accepted my proposal, but here is an excerpt:

 

 

Interesting that even way back then, I had found and compiled research on retrieval practice, spacing, feedback, etc. from the scientific journals and the exhaustive labor of hundreds of academic researchers. I am still talking about these foundational learning principles even today—because they are fundamental and because research and practice continue to demonstrate their power. You can look at recent books and websites that are now celebrating these foundational learning factors (Make it Stick, Design for How People Learn, The Ingredients for Great Teaching, Learning Scientists website, etc.).

Feeling blessed today, as we here in the United States move into a weekend where we honor our workers, that I have been able to use my labor to advance these proven principles, uncovered first by brilliant academic researchers such as Bjork, Bahrick, Mayer, Ebbinghaus, Crowder, Sweller, van Merriënboer, Rothkopf, Runquist, Izawa, Smith, Roediger, Melton, Hintzman, Glenberg, Dempster, Estes, Eich, Ericsson, Davies, Garner, Chi, Godden, Baddeley, Hall, Hintzman, Herz, Karpicke, Butler, Kirschner, Clark, Kulhavy, Moreno, Pashler, Cepeda, and many others.

From these early beginnings, I created a listing of twelve foundational learning factors—factors that I have argued should be our first priority in creating great learning—reviewed here in this document.

Happy Labor Day everyone and special thanks to the researchers who continue to make my work possible—and enable learning professionals of all stripes to build increasingly effective learning!

If you’d like to leave a remembrance in regard to Work-Learning Research’s 20th anniversary, or just read my personal reflections about it, you can do that here.

 

On August 25th 1998, Work-Learning Research was officially born in Portland, Maine, in the United States of America. Please help me celebrate an eventful 20 years!!

In lieu of a big birthday-party bash, I’d like to offer some thanks, brag a little, and invite you to leave a comment below if my work has touched you in any beneficial ways.

If you want a history of the early years, that’s already written here.

I set out 20 years ago to help bridge the gap between scientific research and practice. I had some naive views about how easy that would be, but I’ve tried over the years to compile research from top-tier scientific journals on learning, memory, and instruction and translate what I find into practical recommendations for learning professionals—particularly for those in the workplace learning field. I haven’t done even one-tenth of what I thought I could do, but I see only a little harm in keeping at it.

Thanks!

I have a ton of people to thank for enabling me to persevere. First, my wife, who has been more than patient. Second, my daughter who, still in her mid-teens, brings me hope for the future. Also, my parents and family who have built a foundation of values and strength. A great deal of credit goes to my clients who, let’s face it, pay the bills and enable this operation to continue. Special thanks for the 227 people who sponsored my Kickstarter campaign to get my smile-sheet book published. Thanks also for the other research-to-practice professionals who are there with ideas, feedback, inspiration, and support. Thanks go out to all those who care about research-based work and evidence-based practice. I thank you for standing up for learning practices that work!

Brags

I’ve made a ton of mistakes as a entrepreneur/consultant, but I’m really proud of a few things, so permit me a moment of hubris to share what they are:

  1. Work-Learning Research has freakin’ survived 20 years!! As the legendary Red Sox radio announcer Joe Castiglione might say, “Can you believe it?”
  2. I have avoided selling out. While vendors regularly approach me asking for research or writing that will publicly praise their offerings, I demur.
  3. I published a book that added a fundamental innovation to the workplace learning field. Performance-Focused Smile Sheets will be, in my not-so-humble opinion, an historic text. I’m also proud that 227 people in our field stood up and contributed $13,614 to help me get the book published!
  4. I was talking about fundamental research-based concepts like retrieval practice and spacing back in the early 2000’s, over ten years before books like Brown, Roediger, and McDaniel (2014) popularized these concepts, and I continue to emphasize fundamental learning factors because they matter the most.
  5. I have developed a new Learning Evaluation model (LTEM) that enables us to abandon the problematic Kirkpatrick-Katzell Four-Level Model of Evaluation.
  6. I have developed a number of extremely useful models and frameworks, including the Learning Maximizers Model, the Learning Landscape Model, the SEDA Model, the Decisive Dozen, etc.
  7. I have pioneered methods to overcome the limitations of multiple-choice tests, specifically enabling multiple-choice tests to overcome its recognition-only problem.
  8. I have created a robust catalog of publications, blog posts, and videos that share research-based practical wisdom.
  9. I have, at least a little bit, encouraged people in our field to be more skeptical and more careful and to be less inclined to buy into some of the biggest myths in the learning field. I’m attempting now to reinvigorate the Debunker.Club to enable those who care about research-based practice to support each other.
  10. I have, in a small way (not as much as I wish I could) attempted to speak truth to power.
  11. I have, I hope at least a little bit, supported other research-to-practice advocates and thought leaders.
  12. I have had the honor of helping many clients and organizations, including notable engagements with The Navy Seals, the Defense Intelligence Agency, Bloomberg, The Centers for Disease Control and Prevention, Walgreens, ADP, Oxfam, Practising Law Institute, U.S. National Park Service, Society of Actuaries, the Kauffman Foundation, ISPI, the eLearning Guild, ATD, and Learning Technologies among many others.
  13. To make it a baker’s dozen, let me say I’m also proud that I’ve still got things I want to do…

Celebrate with Me!

While I would have loved to host a big party and invited you all, in lieu of that dream, I invite you to leave a comment.

Thank you for embracing me and my work for so many years!

Maybe it’s weird that I’m leading the celebration. Maybe it seems sad! Let me just say, f*ck that! The world doesn’t hand out accolades to most of us. We have to do our own work and celebrate where we can! I’m happy it’s Work-Learning Research’s 20 anniversary. I invite you to be happy with me!

I am truly grateful…

One more thing… the official anniversary is in a week, when I’ll be pleasantly lost in a family vacation… Apologies if I can’t respond quickly if you leave a note below!