Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem.

Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim.

Read more

About two years ago, four enterprising learning researchers reviewed the research on training and development and published their findings in a top-tier refereed scientific journal. They did a really nice job!

Unfortunately, a vast majority of professionals in the workplace learning-and-performance field have never read the research review, nor have they even heard about it.

As a guy whose consulting practice is premised on the idea that good learning research can be translated into practical wisdom for instructional designers, trainers, elearning developers, chief learning officers and other learning executives, I have been curious to see to what extent this seminal research review has been utilized by other learning professionals. So, for the last year and a half or so, I’ve been asking the audiences I encounter in my keynotes and other conference presentations whether they have encountered this research review.

Often I use the image below to ask the question:

Click here to see original research article…


What would be your guess as to the percentage of folks in our industry who have read this?






Sadly, in almost all of the audiences I’ve encountered, less than 5% of the learning professionals have read this research review.

Indeed, usually more than 95% of workplace learning professionals have “never heard of it” even two years after it was published!!!


And the slur this dumps on our industry’s most potent institutions should be self-evident. And I, too, must take blame for not being more successful in getting these issues heard.

A Review of the Review

People who are subscribed to my email newsletter (you can sign up here), have already been privy to this review many months ago.

I hope the following review will be helpful, and remember, when you’re gathering knowledge to help you do your work, make sure you’re gathering it from sources who are mindful of the scientific research. There is a reason that civilization progresses through its scientific efforts–science provides a structured process of insight generation and testing, creating a self-improving knowledge-generation process that maximizes innovation while minimizing bias.


Quotes from the Research Review:

“It has long been recognized that traditional,
stand-up lectures are an inefficient and
unengaging strategy for imparting
new knowledge and skills.” (p. 86)


“Training costs across organizations remain
relatively constant as training shifts from
face-to-face to technology-based methods.” (p. 87)


“Even when trainees master new knowledge and
skills in training, a number of contextual factors
determine whether that learning is applied
back on the job…” (p. 90)


“Transfer is directly related to opportunities
to practice—opportunities provided either by
the direct supervisor or the organization
as a whole.” (p. 90)


“The Kirkpatrick framework has a number of
theoretical and practical shortcomings…” (p. 91)


I, Will Thalheimer, am a research translator. I study research from peer-reviewed scientific journals on learning, memory, and instruction and attempt to distill whatever practical wisdom might lurk in the dark cacophony of the research catacomb. It’s hard work—and I love it—and the best part is that it gives me some research-based wisdom to share with my consulting clients. It helps me not sound like a know-nothing. Working to bridge the research-practice gap also enables me to talk with trainers, instructional designers, elearning developers, chief learning officers, and other learning executives about their experiences using research-based concepts.


It is from this perspective that I have a sad, and perhaps horrifying, story to tell. In 2012—an excellent research review on training was published in a top-tier journal. Unbelievably, most training practitioners have never heard of this research review. I know because when I speak at conferences and chapters in our field I often ask how many people have read the article. Typically, less than 5% of experienced training practitioners have! Less than 1 in 20 people in our field have read a very important review article.


What the hell are we doing wrong? Why does everyone know what a MOOC is, but hardly anyone has looked at a key research article?


You can access the article by clicking here. You can also read my review of some of the article’s key points as I lay them out below.


Is This Research Any Good?

Not all research is created equal. Some is better than others. Some is crap. Too much “research” in the learning-and-performance industry is crap so it’s important to first acknowledge the quality of the research review.

The research review by Eduardo Salas, Scott Tannenbaum, Kurt Kraiger, and Kimberly Smith-Jentsch from November 2012 was published in the highly-regarded peer-reviewed scientific journal, Psychological Science in the Public Interest, published by the Association for Psychological Science, one of the most respected social-science professional organizations in the world. The research review not only reviews research, but also utilizes meta-analytic techniques to distill findings from multiple research studies. In short, it’s high-quality research.


The rest of this article will highlight key messages from the research review.


Training & Development Gets Results!

The research review by Salas, Tannenbaum, Kraiger, and Smith-Jentsch shows that training and development is positively associated with organizational effectiveness. This is especially important in today’s economy because the need for innovation is greater and more accelerated—and innovation comes from the knowledge and creativity of our human resources. As the researchers say, “At the organizational level, companies need employees who are both ready to perform today’s jobs and able to learn and adjust to changing demands. For employees, that involves developing both job-specific and more generalizable skills; for companies, it means taking actions to ensure that employees are motivated to learn.” (p. 77). Companies spend a ton of money every year on training—in the United States the estimate is $135 billion—so it’s first important to know whether this investment produces positive outcomes. The bottom line: Yes, training does produce benefits.


To Design Training, It Is Essential to Conduct a Training Needs Analysis

“The first step in any training development effort ought to be a training needs analysis (TNA)—conducting a proper diagnosis of what needs to be trained, for whom, and within what type of organizational system. The outcomes of this step are (a) expected learning outcomes, (b) guidance for training design and delivery, (c) ideas for training evaluation, and (d) information about the organizational factors that will likely facilitate or hinder training effectiveness. It is, however, important to recognize that training is not always the ideal solution to address performance deficiencies, and a well-conducted TNA can also help determine whether a non-training solution is a better alternative.” (p. 80-81) “In sum, TNA is a must. It is the first and probably the most important step toward the design and delivery of any training.” (p. 83) “The research shows that employees are often not able to articulate what training they really need” (p. 81) so just asking them what they need to learn is not usually an effective strategy.


Learning Isn’t Always Required—Some Information can be Looked Up When Needed

When doing a training-needs analysis and designing training, it is imperative to separate information that is “need-to-know” from that which is “need-to-access.” Since learners forget easily, it’s better to use training time to teach the need-to-know information and prepare people on how to access the need-to-access information.


Do NOT Offer Training if It is NOT Relevant to Trainees

In addition to being an obvious waste of time and resources, training courses that are not specifically relevant to trainees can hurt motivation for training in general. “Organizations are advised, when possible, to not only select employees who are likely to be motivated to learn when training is provided but to foster high motivation to learn by supporting training and offering valuable training programs.” (p. 79) This suggests that every one of the courses on our LMS should have relevance and value.


It’s about Training Transfer—Not Just about Learning!

“Transfer refers to the extent to which learning during training is subsequently applied on the job or affects later job performance.” (p. 77) “Transfer is critical because without it, an organization is less likely to receive any tangible benefits from its training investments.” (p. 77-78) To ensure transfer, we have to utilize proven scientific research-based principles in our instructional designs. Relying on our intuitions is not enough—because they may steer us wrong.


We must go Beyond Training!

“What happens in training is not the only thing that matters—a focus on what happens before and after training can be as important. Steps should be taken to ensure that trainees perceive support from the organization, are motivated to learn the material, and anticipate the opportunity to use their skills once on (or back on) the job.” (p. 79)


Training can be Designed for Individuals or for Teams

“Today, training is not limited to building individual skills—training can be used to improve teams as well.” (p. 79)


Management and Leadership Training Works

“Research evidence suggests that management and leadership development efforts work.” (p. 80) “Management and leadership development typically incorporate a variety of both formal and informal learning activities, including traditional training, one-on-one mentoring, coaching, action learning, and feedback.” (p. 80)


Forgetting Must Be Minimized, Remembering Must Be Supported

One meta-analysis found that one year after training, “trainees [had] lost over 90% of what they learned.” (p. 84) “It helps to schedule training close in time to when trainees will be able to apply what they have learned so that continued use of the trained skill will help avert skill atrophy. In other words, trainees need the chance to ‘use it before they lose it.’ Similarly, when skill decay is inevitable (e.g., for infrequently utilized skills or knowledge) it can help to schedule refresher training.” (p. 84)


Common Mistakes in Training Design Should Be Avoided

“Recent reports suggest that information and demonstrations (i.e., workbooks, lectures, and videos) remain the strategies of choice in industry. And this is a problem [because] we know from the body of research that learning occurs through the practice and feedback components.” (p. 86) “It has long been recognized that traditional, stand-up lectures are an inefficient and unengaging strategy for imparting new knowledge and skills.” (p. 86) Researchers have “noted that trainee errors are typically avoided in training, but because errors often occur on the job, there is value in training people to cope with errors both strategically and on an emotional level.” (p. 86) “Unfortunately, systematic training needs analysis, including task analysis, is often skipped or replaced by rudimentary questions.” (p. 81)


Effective Training Requires At Least Four Components

“We suggest incorporating four concepts into training: information, demonstration, practice, and feedback.” (p. 86) Information must be presented clearly and in a way that enables the learners to fully understand the concepts and skills being taught. Skill demonstrations should provide clarity to enable comprehension. Realistic practice should be provided to enable full comprehension and long-term remembering. Proving feedback after decision-making and skill practice should be used to correct misconceptions and improve the potency of later practice efforts.

The bottom line is that more realistic practice is needed. Indeed, the most effective training utilizes relatively more practice and feedback than is typically provided. “The demonstration component is most effective when both positive and negative models are shown rather than positive models only.” (p. 87)

Will’s Note: While these four concepts are extremely valuable, personally I think they are insufficient. See my research review on the Decisive Dozen for my alternative.


E-Learning Can Be Effective, But It May Not Lower the Cost of Training

“Both traditional forms of training and technology-based training can work, but both can fail as well. (p. 87) While the common wisdom argues that e-learning is less costly, recent “survey data suggest that training costs across organizations remain relatively constant as training shifts from face-to-face to technology-based methods.” (p. 87) This doesn’t mean that e-learning can’t offer a cost savings, but it does mean that most organizations so far haven’t realized cost savings. “Well-designed technology-based training can be quite effective, but not all training needs are best addressed with that approach. Thus, we advise that organizations use technology-based training wisely—choose the right media and incorporate effective instructional design principles.” (p. 87)


Well-Designed Simulations Provide Potent Learning and Practice

“When properly constructed, simulations and games enable exploration and experimentation in realistic scenarios. Properly constructed simulations also incorporate a number of other research-supported learning aids, in particular practice, scaffolding or context-sensitive support, and feedback. Well-designed simulation enhances learning, improves performance, and helps minimize errors; it is also particularly valuable when training dangerous tasks. (p. 88)


To Get On-the-Job Improvement, Training Requires After-Training Support

“The extent to which trainees perceive the posttraining environment (including the supervisor) as supportive of the skills covered in training had a significant effect on whether those skills are practiced and maintained.” (p. 88) “Even when trainees master new knowledge and skills in training, a number of contextual factors determine whether that learning is applied back on the job: opportunities to perform; social, peer, and supervisory support; and organizational policies.” (p. 90) A trainee’s supervisor is particularly important in this regard. As repeated from above, researchers have “discovered that transfer is directly related to opportunities to practice—opportunities provided either by the direct supervisor or the organization as a whole.” (p. 90)


On-the-Job Learning can be Leveraged with Coaching and Support

“Learning on the job is more complex than just following someone or seeing what one does. The experience has to be guided. Researchers reported that team leaders are a key to learning on the job. These leaders can greatly influence performance and retention. In fact, we know that leaders can be trained to be better coaches…Organizations should therefore provide tools, training, and support to help team leaders to coach employees and use work assignments to reinforce training and to enable trainees to continue their development.” (p. 90)


Trainees’ Supervisors Can Make or Break Training Success

Researchers have “found that one misdirected comment by a team leader can wipe out the full effects of a training program.” (p. 83) “What organizations ought to do is provide leaders with information they need to (a) guide trainees to the right training, (b) clarify trainees’ expectations, (c) prepare trainees, and (d) reinforce learning…” (p. 83) Supervisors can increase trainees’ motivation to engage in the learning process. (p. 85) “After trainees have completed training, supervisors should be positive about training, remove obstacles, and ensure ample opportunity for trainees to apply what they have learned and receive feedback.” (p. 90) “Transfer is directly related to opportunities to practice—opportunities provided either by the direct supervisor or the organization.” (p. 90)


Will’s Note: I’m a big believer in the power of supervisors to enable learning. I’ll be speaking on this in an upcoming ASTD webinar.


Basing Our Evaluations on the Kirkpatrick 4 Levels is Insufficient!!!

“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings. [It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… Although the Kirkpatrick hierarchy has clear limitations, using it for training evaluation does allow organizations to compare their efforts to those of others in the same industry. The authors recommendations for improving training evaluation fit into two categories. First, [instead of only using the Kirkpatrick framework] “organizations should begin training evaluation efforts by clearly specifying one or more purposes for the evaluation and should then link all subsequent decisions of what and how to measure to the stated purposes.” (p. 91) Second, the authors recommend that training evaluations should “use precise affective, cognitive, and/or behavioral measures that reflect the intended learning outcomes.” (p. 91)


This is a devastating critique that should give us all pause. Of course it is not the first such critique, nor will it have to be the last I’m afraid. The worst part about the Kirkpatrick model is that it controls the way we think about learning measurement. It doesn’t allow us to see alternatives.


Leadership is Needed for Successful Training and Development

“Human resources executives, learning officers, and business leaders can influence the effectiveness of training in their organizations and the extent to which their company’s investments in training produce desired results. Collectively, the decisions these leaders make and the signals they send about training can either facilitate or hinder training effectiveness…Training is best viewed as an investment in an organization’s human capital, rather than as a cost of doing business. Underinvesting can leave an organization at a competitive disadvantage. But the adjectives “informed” and “active” are the key to good investing. When we use the word “informed,” we mean being knowledgeable enough about training research and science to make educated decisions. Without such knowledge, it is easy to fall prey to what looks and sounds cool—the latest training fad or technology.”  (p. 92)

Thank you!

I’d like to thank all my clients over the years for hiring me as a consultant, learning auditor, workshop provider, and speaker–and thus enabling me to continue in the critical work of translating research into practical recommendations.

If you think I might be able to help your organization, please feel free to contact me directly by emailing me at “info at worklearning dot com” or calling me at 617-718-0767.


It's not exactly subscription learning–at least not the way MIT is talking about it yet–but the Massachusetts Institute of Technology is considering breaking their courses into modules.

Read the article in the Chronicle of Higher Education.

Subscription Learning is the idea of providing short–usually less than 10 minute–nuggets of learning content or learning interaction. Often, we think of this as an information-delivery platform. In other words, we think of it as training that is spread over time in tiny packets of training content.

But we really ought to think beyond this limited “training” perspective! Subscription Learning can be so much more.

Just this week, I spoke with Marty Rosenheck of Cognitive Advisors, a cognitive-science inspired consultancy and learning-development shop. Cognitive Advisors has recently released TREK, a “Learning Experience Manager” that allows organizations to go beyond training and support people as they learn in the workplace.

From the Cognitive Advisors website, here’s what TREK does:

  • Manages, tracks and reports on the full range of learning
  • Is built from the ground-up with the Experience API (Tin Can) to capture learning outside the LMS
  • Offers cloud-based software designed for today’s mobile workforce
  • Aligns all learning experiences to competencies
  • Supports personalized learning paths to tailor learning for each learner

This technology wasn’t possible just a few years ago, and the future of this product category, which Marty has coined the “Learning Experience Manager,” seems exceedingly bright. The future looks promising because it fulfills a real need, providing support for on-the-job learning.

Indeed, one of the problems with the last decade’s flirtation with informal learning (and on-the-job learning) is that it happens willy nilly without adequate support. People can learn the wrong things, they can learn too slowly, they can get frustrated and give up, they can fail to learn. Just as learning researchers have found that discovery learning is not generally effective–unless it is “guided,” informal learning is often not as effective and efficient as it ought to be.

Real-world Example

Here’s a real-world example of how TREK has been used:

Professionals in the water-quality-assessment field have to get up to speed quickly to do their jobs. And, they have to do their jobs right. There’s no tolerance for poor water-quality testing. The problem is compounded because there’s so much to learn. New folks often feel they’re gazing through muddy waters until they’ve got lots of experience under their belts–and they have clarity about what to do when. They rely on experienced people to answer questions, provide guidance, and monitor their progress.

The down side of this is that it’s very expensive to keep sending experienced people out with the new folks, and its very inefficient and frustrating when inexperienced people have to keep calling their supervisors and mentors from the field.

Enter TREK. The Water Quality Association, working with Cognitive Advisors, piloted a program to provide their members with a structured on-the-job learning path, enabling learners to learn on the job–while being coached by their immediate supervisors.

TREK worked using employee’s smartphones’ sensors (camera, audio and video recorder, and GPS). Employees captured evidence of their critical actions at each step in their learning path. This evidence was submitted through TREK to each person’s designated manager-coach. As each step was completed, managers were notified and were prompted to review their direct reports’ submissions.

Managers provided brief feedback–either written or in a recorded audio nugget–and this feedback was presented to the learners. Managers weren’t left on their own to flounder in their coaching activities. They were provided with coaching guides, checklists, and success criteria within the TREK interface on their smartphones.

Interestingly, what TREK did in this case was to provide support to both learners and the learners’ managers. For both groups, this improved the effectiveness and efficiency of the learning/coaching experience.

Not Just a Technology

As Marty explained to me in discussing this new product category–the reason Learning Experience Management technology is possible now is that several forces have come together. First and foremost, a new database specification has been developed (Experience API, also known as Tin Can) that enables experiences to be collected and categorized–where once we could only capture training-related information. Second, mobile technology is now ubiquitous in cell phones. Third, cloud computing has become a norm, enabling continuous connectivity between learners and others. Finally, learning analytics, social media, and badging have added to the user experience.

But even given these technological advances, the thing that makes TREK and its performance-support coaching possible is that it aligns with human cognition. The key to providing a great learning experience is to ensure that the learning content is captured and presented in ways that can maximize learning.

Marty illustrated the key link in the process, as he described how he leads customers through a cognitive task analysis process he’s perfected over the years. He calls it knowledge harvesting. It reminded me very much of my days building simulations and working with SME’s to extract knowledge. It’s incredibly valuable, but you don’t want to skimp on the process.

Why is This Subscription Learning

TREK provides a subscription learning experience because by focusing on individual workplace tasks, it is providing small chunks of learning that is spaced over time.

The diagram Cognitive Advisors uses to explain their cognitive apprenticeship model highlights a number of small nuggets on the learning-path trajectory.


Folks to Watch

Marty Rosenheck and his business partner Colleen Enghauser are folks to watch–as is their company, Cognitive Advisors. Their dedication to creating learning technology aligned with the learning research is inspiring!



As of today, the Learning Styles Challenge payout is rising from $1000 to $5000! That is, if any person or group creates a real-world learning intervention that takes learning styles into account–and proves that such an intervention produces better learning results than a non-learning-styles intervention, they’ll be awarded $5,000!

Special thanks to the new set of underwriters, each willing to put $1000 in jeopardy to help get the word out to the field:

Learning Styles Challenge Rules

We’re still using the original rules, as established back in 2006. Read them here.

What is Implied in This Debunking

The basic finding in the research is that learning interventions that take into account learning styles do no better than learning interventions that do not take learning styles into account. This does not mean that people do not have differences in the way they learn. It just means that designing with learning styles in mind is unlikely to produce benefits–and thus the extra costs are not likely to be a good investment.

Interestingly, there are learning differences that do matter! For example, if we really want to get benefits from individual differences, we should consider the knowledge and skill level of our learners.

What Can You Do to Spread the Word

Thanks to multiple efforts by many people over the years to lessen the irrational exuberance of the learning-styles proliferators, fewer and fewer folks in the learning field are falling prey to the learning-styles myth. But the work is not done yet. This issue still needs your help!

Here’s some ideas for how you can help:

  • Spread the word through social media! Blogs, Twitter, LinkedIn, Facebook!
  • Share this information with your work colleagues, fellow students, etc.
  • Gently challenge those who proselytize learning styles.
  • Share the research cited below.

History of the Learning Styles Challenge

It has been exactly eight years since I wrote in a blog post:

I will give $1000 (US dollars) to the first person or group who can prove that taking learning styles into account in designing instruction can produce meaningful learning benefits.

Eight years is a long time. Since that time, over one billion babies have been born, 72 billion metric tons of carbon pollution have been produced, and the U.S. Congress has completely stopped functioning.

However, not once in these past eight years has any person or group collected on the Learning Styles challenge. Not once!

Research on Learning Styles

However, since 2006, more and more people have discovered that learning styles are unlikely to be an effective way to design instruction.

First, there was the stunning research review in the top-tier scientific journal, Psychological Science in the Public Interest:

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105-119.

The authors wrote the following:

We conclude therefore, that at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice. Thus, limited education resources would better be devoted to adopting other educational practices that have a strong evidence base, of which there are an increasing number. However, given the lack of methodologically sound studies of learning styles, it would be an error to conclude that all possible versions of learning styles have been tested and found wanting; many have simply not been tested at all. (p. 105)

To read more about what they wrote, click here.

Two years later, two of the authors reiterated their findings in a separate–and nicely written–article for the Association for the Study of Medical Education. You can access that article at: http://uweb.cas.usf.edu/~drohrer/pdfs/Rohrer&Pashler2012MedEd.pdf. Here’s the research citation:

Rohrer, D., & Pashler, H. (2012). Learning styles: Where’s the evidence? Medical Education, 46(7), 634-635.

A researcher who had once advocated for learning styles did an about face after he did some additional research:

Cook, D. A. (2012). Revisiting cognitive and learning styles in computer-assisted instruction: Not so useful after all. Academic Medicine, 87(6), 778-784.
Of course, not everyone is willing to give up on learning styles. For example, Furnham (2012) wrote:
The application of, and research into, learning styles and approaches is clearly alive and well. (p. 77).
Furnham, A. (2012). Learning styles and approaches to learning. In K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, & M. Zeidner (Eds.), APA handbooks in psychology. APA educational psychology handbook, Vol. 2. Individual differences and cultural and contextual factors (pp. 59-81). doi:10.1037/13274-003

A quick cursory look–today–through the PsycINFO database shows that scientific published articles on learning styles are still being published.

Learning Styles in the Workplace Learning Field

Guy Wallace, performance analyst and instructional architect, has been doing a great job keeping the workplace learning field up on the learning-styles debate. Check out his article in eLearn Magazine and his blog post update.

You’ll note from Guy’s blog post that many prominent thought leaders in the field have been suspicious of learning styles for many years.




Last week I launched the website LearningAudit.com to promote the practice of learning audits.


It is my passionate belief that our learning interventions would be tremendously improved if we took a research-based systematic approach to reviewing them. LearningAudit.com is dedicated to the proposition that we can all do this.

On the site there is the research-to-practice report, "How to Conduct a Learning Audit" and a job aid to support the learning-audit process.

The Course Review Template is a research-based job aid that helps us review our training courses–whether classroom or elearning–to get a sense of the effectiveness their designs.

Representative Comments:

Great provocation. We’re a training provider and scored a “typical” client assignment (20-21). Lots of food for thought for conversations with our clients about what they can do to improve the value they get from our training and follow-up!

The CRT does an excellent job at evaluating using the most important measures.

This template made me deeply aware of all of the variables that we often do not even consider as we work to improve our training courses. The number of questions, and their wording, made it difficult to pad the score.

Updated for 2014

The Course Review Template was just updated and is now available in its new incarnation. One change is that it’s now utilizing a creative commons copyright, making it easier for you to use in your organization.

Indeed, you’re welcome to make changes in the PDF or in the following Word format.

Help Support the Further Development of the CRT

After you’ve used the Course Review Template, complete the feedback form to record your results and send me feedback. Thanks! (UPDATE 2017: We’re no longer collecting feedback through a survey format.)