At a recent industry conference, a speaker, offering their expertise on learning evaluation, said this:

“As a discipline, we must look at the metrics that really matter… not to us but to the business we serve.”

Unfortunately, this is one of the most counterproductive memes in learning evaluation. It is counterproductive because it throws our profession under the bus. In this telling, we have no professional principles, no standards, no foundational ethics. We are servants, cleaning the floors the way we are instructed to clean them, even if we know a better way.

Year after year we hear from so-called industry thought leaders that our primary responsibility is to the organizations that pay us. This is a dangerous half truth. Of course we owe our organizations some fealty and of course we want to keep our jobs, but we also have professional obligations that go beyond this simple “tell-me-what-to-do” calculus.

This monomaniacal focus on measuring learning in terms of business outcomes reminds me of the management meme from the 1980s and 90s, that suggested that the goal of a business organization is to increase stakeholder value. This single-bottom-line focus has come under blistering attack for its tendency to skew business operations toward short-term results while ignoring long-term business results and for producing outcomes that harm employees, hurt customers, and destroy the environment.

If we give our business stakeholders the metrics they say that matter to them, but fail to capture the metrics that matter to our success as learning professionals in creating effective learning, then we not only fail ourselves and our learners but we fail our organization as well.

Evaluation What For?

To truly understand learning evaluation, we have to ask ourselves why we’re evaluating learning in the first place! We have to work backwards from the answer to this question.

Why does anyone evaluate? We evaluate to help us make better decisions and take better actions. It’s really that simple! So as learning professionals, we need information to help us make our most important decisions. We should evaluate to support these decisions!

What are our most important decisions? Here’s a few:

  • Which part of the content taught, if any, is relevant and helpful to supporting employees in doing their work? Which parts should be modified or discarded?
  • Which aspects of our learning designs are helpful in supporting comprehension, remembering, and motivation to learn? Which aspects should be modified or discarded?
  • Which after-training supports are helpful in enabling learning to be transferred and utilized by employees in their work? Which supports should be kept? Which need to be modified or discarded?

What are our organizational stakeholders’ most important decisions about learning? Here are a few:

  • Are our learning and development efforts creating optimal learning results? What additional support and resources should the organization supply that might improve learning results? What savings can be found in terms of support and resources—and are these savings worth the lost benefits?
  • Is the leadership of the learning and development function producing a cycle of continuous improvement, generating improved learning outcomes or generating learning outcomes optimized given their resource constraints? If not, can they be influenced to be better or should they be replaced?
  • Is the leadership of the learning and development function creating and utilizing evaluation metrics that enable the learning and development team to get valid feedback about the design factors that are most important in creating our learning results? If not, can they be influenced to use better metrics or should they be replaced?

Two Goals for Learning Evaluation

When we think of learning evaluation, we should have two goals. First, we should create learning-evaluation metrics that enable us to make our most important decisions regarding content, design components (i.e., focused at least on comprehension, remembering, motivation to apply learning), and after-training support. Second, we should do enough in our learning evaluations to gain sufficient credibility with our business stakeholders to continue our good work. Focusing only on the second of these is a recipe for disaster. 

Vanity Metrics

In the business start-up world there is a notion called “vanity metrics,” for example see warnings by Eric Ries, the originator of the lean startup movement. Vanity metrics are metrics that seem to be important, but that are not important. They are metrics that often make us look good even if the underlying data is not really meaningful.

Most calls to provide our business stakeholders with the metrics that matter to them result in beautiful visualizations and data dashboards that focus on vanity metrics. Ubiquitous vanity metrics in learning include the number of trainees trained, the cost per training, the estimates of learners for the value of the learning, complicated benefit/cost analyses of that utilize phantom measures of benefits, etc. By focusing only or primarily on these metrics we don’t have data to improve our learning designs, we don’t have data that enables us create cycles of improvement, we don’t have data that enables us to hold ourselves accountable.

 

 

15th December 2018

Neon Elephant Award Announcement

Dr. Will Thalheimer, President of Work-Learning Research, Inc., announces the winner of the 2018 Neon Elephant Award, given to Clark Quinn for writing the book Millennials, Goldfish & Other Training Misconceptions: Debunking Learning Myths and Superstitions, and for his many years advocating for research-based practices in the workplace learning field.

Click here to learn more about the Neon Elephant Award…

 

2018 Award Winner – Clark Quinn, PhD

Clark Quinn, PhD, is an internationally-recognized consultant and thought-leader in learning technology and organizational learning. Dr. Quinn holds a doctorate in Cognitive Psychology from the University of California at San Diego. Since 2001, Clark has been consulting, researching, writing, and speaking through his consulting practice, Quinnovation (website). Clark has been at the forefront of some of the most important trends in workplace learning, including his early advocacy for mobile learning, his work with the Internet Time Group advocating for a greater emphasis on workplace learning, and his collaboration on the Serious eLearning Manifesto to bring research-based wisdom to elearning design. With the publication of his new book, Clark again shows leadership—now in the cause of debunking learning myths and misconceptions.

Clark is the author of numerous books, focusing not only on debunking learning myths, but also on the practice of learning and development and mobile learning. The following are representative:

In addition to his lifetime of work, Clark is honored for his new book on debunking the learning myths, Millennials, Goldfish & Other Training Misconceptions: Debunking Learning Myths and Superstitions.

Millennials, Goldfish & Other Training Misconceptions provides a quick overview of some of the most popular learning myths, misconceptions, and mistakes. The book is designed as a quick reference for practitioners—to help trainers, instructional designers, and elearning developers avoid wasting their efforts and their organizations’ resources in using faulty concepts. As I wrote in the book’s preface, “Clark Quinn has compiled, for the first time, the myths, misconceptions, and confusions that imbue the workplace learning field with faulty decision making and ineffective learning practices.”

When we think about how much time and money has been wasted by learning myths, when we consider the damage done to learners and organizations, when we acknowledge the harm done to the reputation of the learning profession, we can see how important it is to have a quick reference like Clark has provided.

Clark’s passion for good learning is always evident. From his strategic work with clients, to his practical recommendations around learning technology, to his polemic hyperbole in the revolution book, to his longstanding energy in critiquing industry frailties and praising great work, to his eLearning Guild participatory leadership, to his editorial board contributions at eLearn Magazine, and to his excellent new book; Clark is a kinetic force in the workplace learning field. For his research-inspired recommendations, his tenacity in persevering as a thought-leader consultant, and for his ability to collaborate and share his wisdom, we in the learning field owe Clark Quinn our grateful thanks!

 

 

Click here to learn more about the Neon Elephant Award…