My TiER1 Greatest Hits

, ,

Gratitude for TiER1 Performance. Today is my last day as a full-time employee of TiER1 and I’d like to share some of my most worthy public contributions over the last 2.5 years. See below.

In the learning industry, we’ve dealt with pandemic disruptions, new online technologies, and the advent of generative AI. These are heady times. For me, I’ve continued advocating for improved learning evaluation (as I myself continue to learn and discover new approaches). I’ve hungrily and humbly (because of all the complexity) been on a journey to learn and utilize the performance sciences. And I’ve been thinking about how to empower you and I as learning-and-performance professionals (as I write my next book).

My colleagues and clients at TiER1 have taught me much! I am grateful. I won’t name names because I’ll miss somebody, but lots of good folks have helped me learn.

Here are some of my TiER1 Greatest Hits, all of which were done in partnership or with substantial input and improvement from my TiER1 colleagues.

THE LEARNING TRENDS DIAGNOSTIC SURVEY. This annual survey of the L&D industry became a labor of love for me and a host of TiER1ers, with major contributions from Holly Hilton Bradbury, Walter Warwick, Will Phillips, Zachary Ryland, Laura Bachus, Ramsey Ford, Amanda Rapien, and Sarah Ehrnschwender (with significant help from others too). Also Kudos are due to Sharon Boller and her team for building a strong foundation from which we drew. Here are links to the past two years’ reports.

2022 Learning Trends Report. https://tier1performance.com/2022-learning-trends-report/

2023 Learning Trends Report. https://tier1performance.com/2023-learning-trends-report/

MAJOR ARTICLE ON THE WORK-PERFORMANCE FIELD. In this article, with contribution from Zachary Ryland and Jerry Hamburg, I outline the history of the workplace learning field, highlighting the major historical forces that shaped our field and the forces that are pushing us into the future.

https://tier1performance.com/the-work-performance-field/

KEY INSIGHTS FROM L&D THOUGHT LEADERS (from around the world). In this compendium, 48 thought leaders share their favorite writings, videos, etc.

https://tier1performance.com/key-ideas-insights-for-learning-performance-professionals/

AN ARTICLE ON HOW TO MAXIMIZE TRAINING RESULTS WITH ONE POWERFUL QUESTION.

https://tier1performance.com/maximize-your-training-results-with-one-powerful-question/

PERFORMANCE-FOCUSED LEARNER SURVEYS. Thanks to TiER1 for letting the world know about my book.

https://tier1performance.com/performance-focused-learner-surveys/

PERFORMANCE-ACTIVATION MODEL (the PA Model). This labor of love was co-developed with the amazing Jerry Hamburg. I’ve spoken about the PA Model at two conferences but haven’t written about it. It’s on the back-burner now as it seems too complex for popular consumption, but the thinking that went into the work stays with me still. I’ll share the image here for those who are curious.

The Business Case for the L&D Conference

, ,

The L&D Conference—a quixotic attempt to reinvent conferencing. Aligning conference activities with how we humans actually learn. It’s an online conference, taking place over six weeks. It enables attendees to learn in the flow of their work. So they can use what they’re learning, share with their teammates back in the office, try things out, get advice, stay motivated and inspired to apply their new learning. The conference is the brainchild of Matt Richter and I, and we are now joined by marketing guru Ashley Sinclair. We are so excited to share this with you!

Unfortunately, when people think of online conferences, they think mediocre webinars strung together over one or two or a few days. THIS AIN’T THAT! We would NEVER design a conference that ignored the fundamentals of human learning!

We think you’ll like:

  • our global conference
  • our world-class speakers
  • our research-inspired sessions
  • our rollicking debates
  • our panels
  • our discount house
  • our sponsorship opportunities

BUT IF YOU NEED MORE CONVINCING—or you think YOUR BOSS might need convincing, we can help.

Download this gorgeous document. It makes the business case for joining us in the first L&D Conference.

And actually, the document does more than that. It persuades, in glorious detail, why this will be the best, most effective, and boldest L&D conference ever.

 

Click this Sentence to See the Detailed Document

 

Click this Sentence to See the Less Detailed Webpage

 

Will Thalheimer Interviewed by Learnnovators

, , , , , , , , ,

The LEARNNOVATORS team (specifically Santhosh Kumar) asked if I would join them in their Crystal Balling with Learnnovators interview series, and I accepted! They have some really great people on the series, I recommend that you check it out!

The most impressive thing was that they must have studied my whole career history and read my publication list and watched my videos because they came up with a whole set of very pertinent and important questions. I was BLOWN AWAY—completely IMPRESSED! And, given their dedication, I spent a ton of time preparing and answering their questions.

It’s a two part series and here are the links:

Here are some of the quotes they pulled out and/or I’d like to highlight:

Learning is one of the most wondrous, complex, and important areas of human functioning.

The explosion of different learning technologies beyond authoring tools and LMSs is likely to create a wave of innovations in learning.

Data can be good, but also very very bad.

Learning Analytics is poised to cause problems as well. People are measuring all the wrong things. They are measuring what is easy to measure in learning, but not what is important.

We will be bamboozled by vendors who say they are using AI, but are not, or who are using just 1% AI and claiming that their product is AI-based.

Our senior managers don’t understand learning, they think it is easy, so they don’t support L&D like they should.

Because our L&D leaders live in a world where they are not understood, they do stupid stuff like pretending to align learning with business terminology and business-school vibes—forgetting to align first with learning.

We lie to our senior leaders when we show them our learning data—our smile sheets and our attendance data. We then manage toward these superstitious targets, causing a gross loss of effectiveness.

Learning is hard and learning that is focused on work is even harder because our learners have other priorities—so we shouldn’t beat ourselves up too much.

We know from the science of human cognition that when people encounter visual stimuli, their eyes move rapidly from one object to another and back again trying to comprehend what they see. I call this the “eye-path phenomenon.” So, because of this inherent human tendency, we as presenters—as learning designers too!—have to design our presentation slides to align with these eye-path movements.

Organizations now—and even more so in the near future—will use many tools in a Learning-Technology Stack. These will include (1) platforms that offer asynchronous cloud-based learning environments that enable and encourage better learning designs, (2) tools that enable realistic practice in decision-making, (3) tools that reinforce and remind learners, (4) spaced-learning tools, (5) habit-support tools, (6) insight-learning tools (those that enable creative ideation and innovation), et cetera

Learnnovators asked me what I hoped for the learning and development field. Here’s what I said:

Nobody is good at predicting the future, so I will share the vision I hope for. I hope we in learning and development continue to be passionate about helping other people learn and perform at their best. I hope we recognize that we have a responsibility not just to our organizations, but beyond business results to our learners, their coworkers/families/friends, to the community, society, and the environs. I hope we become brilliantly professionalized, having rigorous standards, a well-researched body of knowledge, higher salaries, and career paths beyond L&D. I hope we measure better, using our results to improve what we do. I hope we, more-and-more, take a small-S scientific approach to our practices, doing more A-B testing, compiling a database of meaningful results, building virtuous cycles of continuous improvement. I hope we develop better tools to make building better learning—and better performance—easier and more effective. And I hope we continue to feel good about our contributions to learning. Learning is at the heart of our humanity!

Ulrich Boser Book Now Updated in Paperback Edition

,

Today, Ulrich Boser released an updated paperback version of his book, Learn Better: Mastering the Skills for Success in Life, Business, and School… It is available on Amazon (make sure you get the paperback version).

Ulrich does good work and his book has been hailed by Walter Isaacson as “Alternately humorous, surprising, and profound,” and by Amazon Editors as one of the Best Science Books of the Year.

You can learn more about Ulrich’s work at his website.

 

Large Companies are Killing Their Own Innovation by Doing this One Stupid Thing

, , ,

CEO’s are calling for their companies to be more innovative in the ever-accelerating competitive landscape! Creativity is the key leverage point for innovation. Research I’ve compiled (from the science on creativity) shows that unique and valuable ideas are generated when people and teams look beyond their inner circle to those in their peripheral networks. GIVEN THIS, a smart company will seed themselves with outside influencers who are working with new ideas.

But what are a vast majority of big companies doing that kills their own creativity? They are making it difficult or virtually impossible for their front-line departments to hire small businesses and consultants. It’s allowed, but massive walls are being built! And these walls have exploded over the last five to ten years:

  1. Only fully vetted companies can be hired, requiring small lean companies to waste time in compliance—or turn away in frustration. Also causing large-company managers to favor the vetted companies, even if a small business or consultant would provide better value or more-pertinent products or services.
  2. Master Service Agreements are required (pushing small companies away due to time and legal fees).
  3. Astronomical amounts of insurance are required. Why the hell do consultants need $2 million in insurance, even when they are consulting on non-safety-related issues? Why do they need any insurance at all if they are not impacting critical safety factors?
  4. Companies can’t be hired unless they’ve been in business for 5 or 10 or 15 years, completely eliminating the most unique and innovative small businesses or consultants—those who recently set up shop.
  5. Minimum company revenues are required, often in the millions of dollars.

These barriers, of course, aren’t the only ones pushing large organizations away from small businesses or consultants. Small companies often can’t afford sales forces or marketing budgets so they are less likely to gain large companies’ share of attention. Small companies aren’t seen as safe bets because they don’t have a name, or their website is not as beautiful, or they haven’t yet worked with other big-name companies, or the don’t speak the corporate language. Given these surface characteristics, only the bravest, most visionary frontline managers will take the risk to make the creative hire. And even then, their companies are making it increasingly hard for them to follow through.

Don’t be fooled by the high-visibility anecdotes that show a CEO hiring a book author or someone featured in Wired, HBR, or on some podcast. Yes, CEO’s and senior managers can easily find ways to hire innovators, and the resulting top-down creativity infusion can be helpful. But it can be harmful as well!!!! Too many times senior managers are too far away from knowing what works and what’s needed on the front lines. They push things innocently not knowing that they are distracting the troops from what’s most important, or worse, pushing the frontline teams to do stupid stuff against their best judgment.

Even more troublesome with these anecdotes of top-down innovation is that they are too few and far between. There may be ten senior managers who can hire innovation seeds, but there are dozens or hundreds or thousands of folks who might be doing so but can’t.

A little digression: It’s the frontline managers who know what’s needed—or perhaps more importantly the “leveraging managers” if I can coin a term. These are the managers who are deeply experienced and wise in the work that is getting done, but high enough in the organization to see the business-case big picture. I will specifically exclude “bottle-cap managers” who have little or no experience in a work area, but were placed there because they have business experience. Research shows these kind of hires are particularly counterproductive in innovation.

Let me summarize.

I’m not selling anything here. I’m in the training, talent development, learning evaluation business as a consultant—I’m not an innovation consultant! I’m just sharing this out of my own frustration with these stupid counter-productive barriers that I and my friends in small businesses and consultancies have experienced. I also am venting here to provide a call to action for large organizations to wake the hell up to the harm you are inflicting on yourselves and on the economy in general. By not supporting the most innovative small companies and consultants, you are dumbing-down the workforce for years to come!

Alright! I suppose I should offer to help instead of just gripe! I have done extensive research on creativity. But I don’t have a workshop developed, the research is not yet in publishable form, and it’s not really what I’m focused on right now. I’m focused on innovating in learning evaluation (see my new learning-evaluation model and my new method for capturing valid and meaningful data from learners). These are two of the most important innovations in learning evaluation in the past few years!

However, a good friend of mine did, just last month, suggest that the world should see the research on creativity that I’ve compiled (thanks Mirjam!). Given the right organization, situation, and requirements—and the right amount of money—I might be willing to take a break from my learning-evaluation work and bring this research to your organization. Contact me to try and twist my arm!

I’m serious, I really don’t want to do this right now, but if I can capture funds to reinvest in my learning-evaluation innovations, I just might be persuaded. On the contact-me link, you can set up an appointment with me. I’d love to talk with you if you want to talk innovation or learning evaluation.

What Should We Call Ourselves? Let us Come to Consensus NOW!

, ,

A huge fiery debate rages in the learning field.

 

What do we call ourselves? Are we instructional designers, learning designers, learning experience designers, learning engineers, etc.? This is an important question, of course, because words matter. But it is also a big freakin’ waste of time, so today, I’m going to end the debate! From now on we will call ourselves by one name. We will never debate this again. We will spend our valuable time on more important matters. You will thank me later! Probably after I am dead.

How do I know the name I propose is the best name? I just know. And you will know it too when you hear the simple brilliance of it.

How do I know the name I propose is the best name? Because Jim Kirkpatrick and I are in almost complete agreement on this, and, well, we have a rocky history.

How do I know the name I propose is the best name? Because it’s NOT the new stylish name everybody’s now printing on their business cards and sharing on LinkedIn. That name is a disaster, as I will explain.

The Most Popular Contenders

I will now list each of the major contenders for what we should call ourselves and then thoroughly eviscerate each one.

Instructional Designer

This is the traditional moniker—used for decades. I have called myself an instructional designer and felt good about it. The term has the benefit of being widely known in our field but it has severe deficiencies. First, if you’re at a party and you tell people you’re an instructional designer, they’re likely to hear “structural designer” or “something-something designer” and think you’re an engineer or a new-age guru who has inhaled too much incense. Second, our job is NOT to create instruction, but to help people learn. Third, our job is NOT ONLY to create instruction to help people learn, but to also create, nurture, or enable contexts that help people learn. Instructional designer is traditional, but not precise. It sends the wrong message. We should discard it.

Learning Designer

This is not bad. It’s my second choice. But it suffers from being too vanilla, too plain, too much lacking in energy. More problematic is that it conveys the notion that we can control learning. We cannot design learning! We can only create or influence situations and materials and messages that enable learning and mathemagenic processes—that is, cognitive processes that give rise to learning. We must discard this label too.

Learning Engineer

This seems reasonable at first glance. We might think our job is to engineer learning—to take the science and technology of learning and use it to blueprint learning interventions. But this is NOT our job. Again, we don’t control learning. We can’t control learning. We can just enable it. Yes! The same argument against “designing learning” can be used against “engineering learning.” We must also reject the learning engineering label because there are a bunch of crazed technology evangelists running around advocating for learning engineering who think that big data and artificial intelligence is going to solve all the problems of the learning profession. While it is true that data will help support learning efforts, we are more likely to make a mess of this by focusing on what is easy to measure and not on what is important and difficult to measure. We must reject this label too!

Learning Experience Designer

This new label is the HOT new label in our field, but it’s a disastrous turn backward! Is that who we are—designers of experiences? Look, I get it. It seems good on the surface. It overcomes the problem of control. If we design experiences, we rightly admit that we are not able to control learning but can only enable it through learning experiences. That’s good as far as it goes. But is that all there is? NO DAMMIT! It’s a freakin’ cop-out, probably generated and supported by learning-technology platform vendors to help sell their wares! What the hell are we thinking? Isn’t it our responsibility to do more than design experiences? We’re supposed to do everything we can to use learning as a tool to create benefits. We want to help people perform better! We want to help organizations get better results! We want to create benefits that ripple through our learners’ lives and through networks of humanity. Is it okay to just create experiences and be happy with that? If you think so, I wish to hell you’d get out of the learning profession and cast your lack of passion and your incompetence into a field that doesn’t matter as much as learning! Yes! This is that serious!

As learning professionals we need to create experiences, but we also need to influence or create the conditions where our learners are motivated and resourced and supported in applying their learning. We need to utilize learning factors that enable remembering. We need to create knowledge repositories and prompting mechanisms like job aids and performance support. We need to work to create organizational cultures and habits of work that enable learning. We need to support creative thinking so people have insights that they otherwise wouldn’t have. We also must create learning-evaluation systems that give us feedback so we can create cycles of continuous improvement. If we’re just creating experiences, we are in the darkest and most dangerous depths of denial. We must reject this label and immediately erase the term “Learning Experience Designer” from our email signatures, business cards, and LinkedIn profiles!

The Best Moniker for us as Learning Professionals

First, let me say that there are many roles for us learning professionals. I’ve been talking about the overarching design/development role, but there are also trainers, instructors, teachers, professors, lecturers, facilitators, graphic designers, elearning developers, evaluators, database managers, technologists, programmers, LMS technicians, supervisors, team leaders, et cetera, et cetera, et cetera. Acknowledged!!! Now let me continue. Thanks!

A month ago, Mirjam Neelen reached out to me because she is writing a book on how to use the science of learning in our role as learning professionals. She’s doing this with another brilliant research-to-practice advocate, the learning researcher Paul Kirschner, following from their blog, 3-Star Learning. Anyway, Mirjam asked me what recommendation I might have for what we call ourselves. It was a good question, and I gave her my answer.

I gave her THE answer. I’m not sure she agreed and she and Paul and their publisher probably have to negotiate a bit, but regardless, I came away from my discussions with Mirjam convinced that the learning god had spoken to me and asked me to share the good word with you. I will now end this debate. The label we should use instead of the others is Learning Architect. This is who we are! This is who we should be!

Let’s think about what architects do—architects in the traditional sense. They study human nature and human needs, as well as the science and technology of construction, and use that knowledge/wisdom to create buildings that enable us human beings to live well. Architects blueprint the plans—practical plans—for how to build the building and then they support the people who actually construct the buildings to ensure that the building’s features will work as well as possible. After the building is finished, the people in the buildings lead their lives under the influence of the building’s design features. The best architects then assess the outcomes of those design features and suggest modifications and improvements to meet the goals and needs of the inhabitants.

We aspire to be like architects. We don’t control learning, but we’d like to influence it. We’d like to motivate our learners to engage in learning and to apply what they’ve learned. We’d like to support our learners in remembering. We’d like to help them overcome obstacles. We’d like to put structures in place to enable a culture of learning, to give learners support and resources, to keep learners focused on applying what they’ve learned. We’d like to support teams and supervisors in their roles of enabling learning. We’d like to measure learning to get feedback on learning so that we can improve learning and troubleshoot if our learners are having problems using what we’ve created or applying what they’ve learned.

We are learning architects so let’s start calling ourselves by that name!

But Isn’t “Architect” a Protected Name?

Christy Tucker (thanks Christy!) raised an important concern in the comments below, and her concern was echoed by Sean Rea and Brett Christensen. The term “architect” is a protected term, which you can read about on Wikipedia. Architects rightly want to protect their professional reputation and keep their fees high, protected from competition from people with less education, experience, and competence.

But, to my non-legal mind, this is completely irrelevant to our discussion. When we add an adjective, the name is a different name. It’s not legal to call yourself a doctor if you’re not a doctor, but it’s okay to call yourself the computer doctor, the window doctor, the cakemix doctor, the toilet doctor, or the LMS doctor.

While the term “architect” is protected, putting an adjective in front of the name changes everything. A search of LinkedIn for “data architects” lists 57,624 of them. A search of “software architect” finds 172,998. There are 3,110 “performance architects,” 24 “justice architects,” and 178 “sustainability architects.”

Already on LinkedIn, 2,396 people call themselves “learning architects.”

Searching DuckDuckGo, some of the top results were consultants calling themselves learning architects from the UK, New Zealand, Australia. LinkedIn says there are almost 10,000 learning architecture jobs in the United States.

This is a non-issue. First, adding the adjective changes the name legally. Second, even if it didn’t, there is no way that architect credentialing bodies are going to take legal action against the hundreds of thousands of people using the word “architect” with an adjective. I say this, of course, not as a lawyer—and you should not rely on my advice as legal advice.

But still, this has every appearance of being a non-issue and we learning professionals should not be so meek as to shy away from using the term learning architect.

I was listening to a podcast last week that interviewed Jim Kirkpatrick. I like to listen to what Jim and Wendy have to say because many people I speak with in my work doing learning evaluation are influenced by what they say and write. As you probably know, I think the Kirkpatrick-Katzell Four-Level Model causes more harm then good, but I like to listen and learn things from the Kirkpatrick’s even though I never hear them sharing ideas that are critical of their models and teachings. Yes! I’m offering constructive criticism! Anyway, I was listening to the podcast and agreeing with most of what Jim was saying when he mentioned that what we ought to call ourselves is, wait for it, wait for it, wait for it: “Learning-and-Performance Architects!” Did I mention that I just love Jim Kirkpatrick! Jim and I are in complete agreement on this. I’ll quibble in that the name Learning-and-Performance Architect is too long, but I agree with the sentiment that we ought to see performance as part of our responsibility.

So I did some internet searching this week for the term “Learning Architect.” I found a job at IBM with that title, estimated by Glassdoor to pay between $104,000 and $146,000, and I think I’m going to apply for that job as this consulting thing is kind of difficult these days, especially having to write incisive witty profound historic blog posts for no money and no fame.

I also found a podcast by the eLearning Coach Connie Malamed on her excellent podcast where she reviews a book by the brilliant and provocative Clive Shepherd with the title, The New Learning Architect. It was published in 2011 and now has an updated 2016 edition. Interestingly, in a post from just this year in 2019, Clive is much less demonstrative about advocating for the term Learning Architect, and casually mentions that Learning Solutions Designer is a possibility before rejecting it because of the acronym LSD. I will reject it because designing solutions may give some the idea that we are designing things, when we need to design more than tangible objects.

In searching the internet, I also found three consultants or group of consultants calling themselves learning architects. I also searched LinkedIn and found that the amazing Tom Kuhlmann has been Vice President of Community at Articulate for 12 years but added the title of Chief Learning Architect four years and eight months ago. I know Tom’s great because of our personal conversations in London and because he’s always sharing news of my good works to the Articulate community (you are, right? Tom?), but most importantly because on Tom’s LinkedIn page one of the world’s top entrepreneurs offered a testimonial that Tom improved his visual presentations by 12.9472%. You can’t make this stuff up, not even if you’re a learning experience designer high on LSD!

Clearly, this Learning Architect idea is not a new thing! But I have it on good authority that now here today, May 24, 2019, we are all learning architects!

Here are two visual representations I sent to Mirjam to help convey the breadth and depth of what a Learning Architect should do:

 

I offer these to encourage reflection and discussion. They were admittedly a rather quick creation, so certainly, they must have blind spots.

Feel free to discuss below or elsewhere the ideas discussed in this article.

And go out and be the best learning architect you can be!

I have it on good authority that you will be…

 

 

 

Evidence-Based Information to Inform Our Politics — Maybe History!?!!

,
Read more

Bob Mager’s Was Only Half Right about Learning Objectives

, , , ,

This is NOT a post about Bob Mager. It is something else entirely.

In probably the best video I will ever create, I made the case that learning professionals and learners should NOT receive the same set of learning objectives.

The rationale is this: Because objectives are designed to guide behavior, how could one statement possibly guide the behaviors of two separate audiences? Sometimes maybe! But not always!

Arguments for the Infallibility of an Instructional-Design Hero

Recently, I’ve heard it argued that Bob Mager, in his classic text, “Preparing Instructional Objectives,” urged us to create instructional objectives only for us as learning professionals, that he never intended that instructional objectives be presented to learners. This is a testable assertion, which is great! We can agree that Mager gave us some good advice on how to craft objectives for ourselves as learning professionals. But did Mager also, perhaps, suggest that objectives could be presented to learners?

Here are several word-for-word quotes from Mager’s book:

Page 16: Heading: “Goal Posts for Students”

Page 16: “Clearly defined objectives also can be used to provide students with the means to organize their own time and efforts toward accomplishment of those objectives.”

Page 17: “With clear objectives, it is possible to organize the instruction itself so that instructors and students alike can focus their efforts on bridging the gap…”

Page 19: Chapter Summary. “Objectives are useful for providing: … Tools for guiding student efforts…”

Page 43: “Objectives in the hands of students prevent the students from having to guess at how they might best organize their time and effort.”

So Mager clearly started the confusion! But Mager wrote at a time before research on cognition enabled greater insight.

Forget Mager’s contribution. The big problem is that the most common practice seems to still be efforts to create a set of learning objectives to use for both learners and learning practitioners.

Scolded

I was even scolded for not knowing the difference between an instructional objective (for learning professionals) and a learning objective (for learners). Of course, these revisionist definitions are not true and are not helpful. They are fake news, concocted perhaps by a person who thinks or was taught that our instructional-design heroes are perfect and their work is sacrosanct. The truth is that these terms have been used interchangeably. For example, in a research study by my mentor and academic advisor, Ernie Rothkopf, he and his research partner used the term instructional objectives to refer to objectives presented to learners.

Rothkopf, E. Z., & Kaplan, R. (1972). An exploration of the effect of density and specificity of instructional objectives on learning from text. Journal of Educational Psychology, 6, 295-302.

My Main Points

  • We need at least two types of objectives (although I’ve argued for more)—one to guide the design, development, and evaluation of learning; one to guide learners as they are learning. I’ve called these “focusing objectives,” because the research shows that they guide attention toward objective-relevant content.
  • When we make arguments, we ought to at least skim the sources to see if we know what we’re talking about.
  • We ought to stop with hero worship. All of us do some good things and some bad things. Even the best of us.
  • Hero worship in the learning field is particularly problematic because learning is so complex and we all still have so much to learn. All of us attempting to make recommendations are likely to be wrong some of the time.
  • It is ironic that our schools of instructional design teach graduate students to memorize facts and hold up heroes as infallible immortals—when instead they ought to be educating these future citizens how progress gets made over long periods of time by a large collective of people. They also ought to be teaching students to understand at a deeper level, not just a knowledge level. But truly, we can’t blame the schools of instructional design. After all, they started with canonically-correct instructional objectives (focused on low-level knowledge because they are easier to create).

Finally, let me say that in the video I praise Bob Mager’s work on learning objectives for us learning professionals. This post is not about Mager.

 

2018 in Review: Engineering the Future of Learning Evaluation

, , ,

My Year In Review 2018—Engineering the Future of Learning Evaluation

In 2018, I shattered my collarbone and lay wasting for several months, but still, I think I had one of my best years in terms of the contributions I was able to make. This will certainly sound like hubris, and surely it is, but I can’t help but think that 2018 may go down as one of the most important years in learning evaluation’s long history. At the end of this post, I will get to my failures and regrets, but first I’d like to share just how consequential this year was in my thinking and work in learning evaluation.

It started in January when I published a decisive piece of investigative journalism showing that Donald Kirkpatrick was NOT the originator of the four-level model; that another man, Raymond Katzell, has deserved that honor all along. In February, I published a new evaluation model, LTEM (The Learning-Transfer Evaluation Model)—intended to replace the weak and harmful Kirkpatrick-Katzell Four-Level Model. Already, doctoral students are studying LTEM and organizations around the world are using LTEM to build more effective learning-evaluation strategies.

Publishing these two groundbreaking efforts would have made a great year, but because I still have so much to learn about evaluation, I was very active in exploring our practices—looking for their strengths and weaknesses. I led two research efforts (one with the eLearning Guild and one with my own organization, Work-Learning Research). The Guild research surveyed people like you and your learning-professional colleagues on their general evaluation practices. The Work-Learning Research effort focused specifically on our experiences as practitioners in surveying our learners for their feedback.

Also in 2018, I compiled and published a list of 54 common mistakes that get made in learning evaluation. I wrote an article on how to think about our business stakeholders in learning evaluation. I wrote a post on one of the biggest lies in learning evaluation—how we fool ourselves into thinking that learner feedback gives us definitive data on learning transfer and organizational results. It does not! I created a replacement for the problematic Net Promoter Score. I shared my updated smile-sheet questions, improving those originally put forth in my award winning book, Performance-Focused Smile Sheets. You can access all these publications below.

In my 2018 keynotes, conference sessions, and workshops, I recounted our decades-long frustrations in learning evaluation. We are clearly not happy with what we’ve been able to do in terms of learning evaluation. There are two reasons for this. First, learning evaluation is very complex and difficult to accomplish—doubly so given our severe resource constraints in terms of both budget and time. Second, our learning-evaluation tools are mostly substandard—enabling us to create vanity metrics but not enabling us to capture data in ways that help us, as learning professionals, make our most important decisions.

In 2019, I will continue my work in learning evaluation. I still have so much to unravel. If you see a bit of wisdom related to learning evaluation, please let me know.

Will’s Top Fifteen Publications for 2018

Let me provide a quick review of the top things I wrote this year:

  1. LTEM (The Learning-Transfer Evaluation Model)
    Although published by me in 2018, the model and accompanying 34-page report originated in work begun in 2016 and through the generous and brilliant feedback I received from Julie Dirksen, Clark Quinn, Roy Pollock, Adam Neaman, Yvon Dalat, Emma Weber, Scott Weersing, Mark Jenkins, Ingrid Guerra-Lopez, Rob Brinkerhoff, Trudy Mandeville, and Mike Rustici—as well as from attendees in the 2017 ISPI Design-Thinking conference and the 2018 Learning Technologies conference in London. LTEM is designed to replace the Kirkpatrick-Katzell Four-Level Model originally formulated in the 1950s. You can learn about the new model by clicking here.
  2. Raymond Katzell NOT Donald Kirkpatrick
    Raymond Katzell originated the Four-Level Model. Although Donald Kirkpatrick embraced accolades for the Four-Level Model, it turns out that Raymond Katzell was the true originator. I did an exhaustive investigation and offered a balanced interpretation of the facts. You can read the original piece by clicking here. Interestingly, none of our trade associations have reported on this finding. Why is that? LOL
  3. When Training Pollutes. Our Responsibility to Lessen the Environmental Damage of Training
    I wrote an article and placed it on LinkedIn and as far as I can tell, very few of us really want to think about this. But you can get started by reading the article (by clicking here).
  4. Fifty-Four Mistakes in Learning Evaluation
    Of course we as an industry make mistakes in learning evaluation, but who knew we made so many? I began compiling the list because I’d seen a good number of poor practices and false narratives about what is important in learning evaluation, but by the time I’d gotten my full list I was a bit dumbstruck by the magnitude of problem. I’ve come to believe that we are still in the dark ages of learning evaluation and we need a renaissance. This article will give you some targets for improvements. Click here to read it.
  5. New Research on Learning Evaluation — Conducted with The eLearning Guild
    The eLearning Guild and Dr. Jane Bozarth (the Guild’s Director of Research) asked me to lead a research effort to determine what practitioners in the learning/elearning field are thinking and doing in terms of learning evaluation. In a major report released about a month ago, we reveal findings on how people feel about the learning measurement they are able to do, the support they get from their organizations, and their feelings about their current level of evaluation competence. You can read a blog post I wrote highlighting one result from the report—that a full 40% of us are unhappy with what we are able to do in terms of learning evaluation. You can access the full report here (if you’re a Guild member) and an executive summary. Also, stay tuned to my blog or signup for my newsletter to see future posts about our findings.
  6. Current Practices in Gathering Learner Feedback
    We at Work-Learning Research, Inc. conducted a survey focused on gathering learner feedback (i.e., smile sheets, reaction forms, learner surveys) that spanned 2017 and 2018. Since the publication of my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, I’ve spent a ton of time helping organizations build more effective learner surveys and gauging common practices in the workplace learning field. This research survey continued that work. To read my exhaustive report, click here.
  7. One of the Biggest Lies in Learning Evaluation — Asking Learners about Level 3 and 4 (LTEM Tiers 7 and 8)
    This is big! One of the biggest lies in learning evaluation. It’s a lie we like to tell ourselves and a lie our learning-evaluation vendors like to tell us. If we ask our learners questions that relate to their job performance or the organizational impact of our learning programs we are NOT measuring at Kirkpatrick-Katzell Level 3 or 4 (or at LTEM Tiers 7 and 8), we are measuring at Level 1 and LTEM Tier 3. You can read this refutation here.
  8. Who Will Rule Our Conferences? Truth or Bad-Faith Vendors?
    What do you want from the trade organizations in the learning field? Probably “accurate information” is high on your list. But what happens when the information you get is biased and untrustworthy? Could. Never. Happen. Right? Read this article to see how bias might creep in.
  9. Snake Oil. The Story of Clark Stanley as Preface to Clark Quinn’s Excellent Book
    This was one of my favorite pieces of writing in 2018. Did I ever mention that I love writing and would consider giving this all up for a career as a writer? You’ve all heard of “snake oil” but if you don’t know where the term originated, you really ought to read this piece.
  10. Dealing with the Emotional Readiness of Our Learners — My Ski Accident Reflections
    I had a bad accident on the ski slopes in February this year and I got thinking about how our learners might not always be emotionally ready to learn. I don’t have answers in this piece, just reflections, which you can read about here.
  11. The Backfire Effect. Not the Big Worry We Thought it was (for Those Who Would Debunk Learning Myths)
    This article is for those interested in debunking and persuasion. The Backfire Effect was the finding that trying to persuade someone to stop believing a falsehood, might actually make them more inclined to believe the falsehood. The good news is that new research showed that this worry might be overblown. You can read more about this here (if you dare to be persuaded).
  12. Updated Smile-Sheet Questions for 2018
    I published a set of learner-survey questions in my 2016 book, and have been working with clients to use these questions and variations on these questions for over two years since then. I’ve learned a thing or two and so I published some improvements early last year. You can see those improvements here. And note, for 2019, I’ll be making additional improvements—so stay tuned! Remember, you can sign up to be notified of my news here.
  13. Replacement for NPS (The Net Promoter Score)
    NPS is all the rage. Still! Unfortunately, it’s a terribly bad question to include on a learner survey. The good news is that now there is an alternative, which you can see here.
  14. Neon Elephant Award for 2018 to Clark Quinn
    Every year, I give an award for a great research-to-practice contribution in the workplace learning field. This year’s winner is Clark Quinn. See why he won and check out his excellent resources here.
  15. New Debunker Club Website
    The Debunker Club is a group of people who have committed to debunking myths in the learning field and/or sharing research-based information. In 2018, working with a great team of volunteers, we revamped the Debunker Club website to help build a community of debunkers. We now have over 800 members from around the world. You can learn more about why The Debunker Club exists by clicking here. Also, feel free to join us!

 

My Final Reflections on 2018

I’m blessed to be supported by smart passionate clients and by some of the smartest friends and colleagues in the learning field. My Work-Learning Research practice turned 20 years old in 2018. Being a consultant—especially one who focuses on research-to-practice in the workplace learning field—is still a challenging yet emotionally rewarding endeavor. In 2018, I turned my attention almost fully to learning evaluation. You can read about my two-path evaluation approach here. One of my research surveys totally flopped this year. It was focused on the interface between us (as learning professionals) and our organizations’ senior leadership. I wanted to know if what we thought senior leadership wanted was what they actually wanted. Unfortunately, neither I nor any of the respondents could entice a senior leader to comment. Not one! If you or your organization has access to senior managers, I’d love to partner with you on this! Let me know. Indeed, this doesn’t even have to be research. If your CEO would be willing to trade his/her time letting me ask a few questions in exchange for my time answering questions about learning, elearning, learning evaluation, etc., I’d be freakin’ delighted! I failed this year in working out a deal with another evaluation-focused organization to merge our efforts. I was bummed about this failure as the synergies would have been great. I also failed in 2018 to cure myself of the tendency to miss important emails. If you ever can’t get in touch with me, try, try again! Thanks and apologies! I had a blast in 2018 speaking and keynoting at conferences—both big and small conferences. From doing variations on the Learning-Research Quiz Show (a rollicking good time) to talking about innovations in learning evaluation to presenting workshops on my learning-evaluation methods and the LTEM model. Good stuff, if a ton of work. Oh! I did fail again in 2018 turning my workshops into online workshops. I hope to do better in 2019. I also failed in 2018 in finishing up a research review of the training transfer research. I’m like 95% done, but still haven’t had a chance to finish.

2018 broke my body, made me unavailable for a couple of months, but overall, it turned out to be a pretty damn good year. 2019 looks promising too as I have plans to continue working on learning evaluation. It’s kind of interesting that we are still in the dark ages of learning evaluation. We as an industry, and me as a person, have a ton more to learn about learning evaluation. I plan to continue the journey. Please feel free to reach out and let me know what I can learn from you and your organization. And of course, because I need to pay the rent, let me say that I’d be delighted if you wanted me to help you or your organization. You can reach me through the Work-Learning Research contact form.

Thanks for reading and being interested in my work!!!

Happy Birthday Work-Learning Research!! 20 Years Old

, , ,

On August 25th 1998, Work-Learning Research was officially born in Portland, Maine, in the United States of America. Please help me celebrate an eventful 20 years!!

In lieu of a big birthday-party bash, I’d like to offer some thanks, brag a little, and invite you to leave a comment below if my work has touched you in any beneficial ways.

If you want a history of the early years, that’s already written here.

I set out 20 years ago to help bridge the gap between scientific research and practice. I had some naive views about how easy that would be, but I’ve tried over the years to compile research from top-tier scientific journals on learning, memory, and instruction and translate what I find into practical recommendations for learning professionals—particularly for those in the workplace learning field. I haven’t done even one-tenth of what I thought I could do, but I see only a little harm in keeping at it.

Thanks!

I have a ton of people to thank for enabling me to persevere. First, my wife, who has been more than patient. Second, my daughter who, still in her mid-teens, brings me hope for the future. Also, my parents and family who have built a foundation of values and strength. A great deal of credit goes to my clients who, let’s face it, pay the bills and enable this operation to continue. Special thanks for the 227 people who sponsored my Kickstarter campaign to get my smile-sheet book published. Thanks also for the other research-to-practice professionals who are there with ideas, feedback, inspiration, and support. Thanks go out to all those who care about research-based work and evidence-based practice. I thank you for standing up for learning practices that work!

Brags

I’ve made a ton of mistakes as a entrepreneur/consultant, but I’m really proud of a few things, so permit me a moment of hubris to share what they are:

  1. Work-Learning Research has freakin’ survived 20 years!! As the legendary Red Sox radio announcer Joe Castiglione might say, “Can you believe it?”
  2. I have avoided selling out. While vendors regularly approach me asking for research or writing that will publicly praise their offerings, I demur.
  3. I published a book that added a fundamental innovation to the workplace learning field. Performance-Focused Smile Sheets will be, in my not-so-humble opinion, an historic text. I’m also proud that 227 people in our field stood up and contributed $13,614 to help me get the book published!
  4. I was talking about fundamental research-based concepts like retrieval practice and spacing back in the early 2000’s, over ten years before books like Brown, Roediger, and McDaniel (2014) popularized these concepts, and I continue to emphasize fundamental learning factors because they matter the most.
  5. I have developed a new Learning Evaluation model (LTEM) that enables us to abandon the problematic Kirkpatrick-Katzell Four-Level Model of Evaluation.
  6. I have developed a number of extremely useful models and frameworks, including the Learning Maximizers Model, the Learning Landscape Model, the SEDA Model, the Decisive Dozen, etc.
  7. I have pioneered methods to overcome the limitations of multiple-choice tests, specifically enabling multiple-choice tests to overcome its recognition-only problem.
  8. I have created a robust catalog of publications, blog posts, and videos that share research-based practical wisdom.
  9. I have, at least a little bit, encouraged people in our field to be more skeptical and more careful and to be less inclined to buy into some of the biggest myths in the learning field. I’m attempting now to reinvigorate the Debunker.Club to enable those who care about research-based practice to support each other.
  10. I have, in a small way (not as much as I wish I could) attempted to speak truth to power.
  11. I have, I hope at least a little bit, supported other research-to-practice advocates and thought leaders.
  12. I have had the honor of helping many clients and organizations, including notable engagements with The Navy Seals, the Defense Intelligence Agency, Bloomberg, The Centers for Disease Control and Prevention, Walgreens, ADP, Oxfam, Practising Law Institute, U.S. National Park Service, Society of Actuaries, the Kauffman Foundation, ISPI, the eLearning Guild, ATD, and Learning Technologies among many others.
  13. To make it a baker’s dozen, let me say I’m also proud that I’ve still got things I want to do…

Celebrate with Me!

While I would have loved to host a big party and invited you all, in lieu of that dream, I invite you to leave a comment.

Thank you for embracing me and my work for so many years!

Maybe it’s weird that I’m leading the celebration. Maybe it seems sad! Let me just say, f*ck that! The world doesn’t hand out accolades to most of us. We have to do our own work and celebrate where we can! I’m happy it’s Work-Learning Research’s 20 anniversary. I invite you to be happy with me!

I am truly grateful…

One more thing… the official anniversary is in a week, when I’ll be pleasantly lost in a family vacation… Apologies if I can’t respond quickly if you leave a note below!