Posts

What do our most popular blog posts say about our field—the learning field?

A few months ago (in the last half of 2018), I reached out to bloggers in the learning field to find out. This blog post includes the numbers, wit, and wisdom from these bloggers. In addition to me, there are 18 other bloggers who generously shared their most popular blog posts.

I’m thrilled with the cross section of bloggers who are included here. We’ve got mega-bloggers (people who get over 25,000 people coming to their homepage each year, we’ve got medium bloggers, and we’ve got folks with small but passionate audiences. We’ve got some of the biggest names in the learning field. We’ve got folks focused on workplace learning and folks focused on education. We’ve got about an even split between men and women. Most of all, we have a group of folks bold and generous enough to share the reality of their blogs.

My Observations From the Results

I highly encourage you to peruse the contributions below. Each blogger shares his/her most popular blog post—the post that gets the most yearly visits—and reflects on what makes it so popular. They also share their feelings about why they’re blogging in the first place.

Some of the most popular posts are short. Some are long. Some are personal, even intimate. Some recount research with cold steely precision. Some have a negative slant, raging against poor practices. Some have a positive slant, reveling in the wonder of learning and service to others.

So what makes for a popular blog post in the learning field? Well, we have a relatively small and certainly not-fully representative sample, but the most important thing seems to be providing people with information that is perceived as valuable and/or unique. Some characteristics that seem especially important:

  • Answering important questions—questions that often get asked.
  • Providing definitive or research-based information.
  • Debunking myths or arguing against common traditional practices.
  • Providing a list of information.
  • Providing links where readers can learn more.
  • Introducing a creative or unique concept or idea.
  • Focusing on a topic of current popular interest.
  • There is a visual element to the blog post.
  • The topic is one likely to be assigned by university professors.
  • Timeless topics, because over time they engender lots of links.
  • Topics that help people do their work better.

These are some of the lesson learned from my reflections. You will probably see other things in the list of most popular blog posts. Please add your observations and conclusions in the comments below.

The bloggers are listed in random order below.

What the Blog Posts Say about the Learning Field

Our sample of bloggers offer thoughtful reflections on the practice of learning and development. It seems readers are hungry for useful, validated, and unique information—as long as it is presented in ways that are pithy, straightforward, and powerful.

To me, as a research-to-practice guy, I’m encouraged by the interest in evidence-based information and the number of people searching for information on learning myths. I’m also delighted that none of the most popular blog posts are advocating silly or harmful fads. This could be a result of the kinds of people likely to respond to a call to action from me, someone known for a certain perspective and approach. On the other hand, I did ask people to share the call-to-action widely and we did get quite a broad range of bloggers.

The most popular blog posts also suggest that blog readers in the learning field are looking for information that they can use right away—that add value and help them do their work.

Our readers also want unique perspectives. Not the same old thing. Some of the most popular posts talked about humble leadership, training ghettos, being human on social media, and digital body language—topics so compelling that readers can’t help but engage.

What do you think the list of most popular blog posts say about the learning field? Respond below! I’d love to hear multiple perspectives on this.

I got interested in the popularity of different blog posts because one of my blog posts is ridiculously more popular than my blog itself, regularly getting more than three times more traffic than my home page!! I’ll give you my stats first, and then share each blogger’s responses one at a time—shared in a random order.

Will Thalheimer

What is the name of your blog? (NOT YOUR URL)

Will’s Blog (formerly Will At Work Learning)

Blog Address:

https://www.worklearning.com/wills-blog/

What do you try to do in your blog? What’s your focus, goal, or slant?

My blog is designed to share research-based practical wisdom with the workplace learning field and news and research of importance to those who follow my work. Focuses more and more on learning evaluation.

What is the URL of your most visited blog post?

https://www.worklearning.com/2006/05/01/people_remember/

What is the title of this blog post?

People remember 10%, 20%…Oh Really?

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

34,135

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

9,994

What is the gist or main message of THIS blog post?

The post debunks the myth that people remember 10% of what they read, 20% of what they hear, 30% of what they see, etc. These numbers have often been associated with the learning pyramid and Edgar Dale’s Cone. This was my original blog post from 2006 and has been updated over the years.

Why do you think this post is so popular with your readers?

The myth is so widely shared and so many people have taken responsibility to debunk this myth on their own blogs and websites that lots of traffic gets pointed to this post as it was among the first warnings posted on the internet.

Mike Taylor

What is the name of your blog? (NOT YOUR URL)

Ask. Learn. Share.

Blog Address:

https://mike-taylor.org/

What do you try to do in your blog? What’s your focus, goal, or slant?

Share useful things from the intersection of learning, design, and technology.

What is the URL of your most visited blog post?

https://mike-taylor.org/2011/06/10/21-questions-to-ask-before-designing-any-training-program/

What is the title of this blog post?

21 Questions to ask before Designing Any Training Program

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

2,244

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

7,254

What is the gist or main message of THIS blog post?

This page is just a collection of the most recent posts.

Why do you think this post is so popular with your readers?

I think this is a very key foundational idea for how to get started on any learning-related project.

Andrew Jacobs

What is the name of your blog? (NOT YOUR URL)

Lost and Desperate.

Blog Address:

https://lostanddesperate.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

Writing about learning in a way which helps me reflect and tries to move the industry forward.

What is the URL of your most visited blog post?

https://lostanddesperate.com/2014/03/14/50-big-ideas-to-change-l-and-d/

What is the title of this blog post?

50 BIG IDEAS TO CHANGE L AND D

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

151

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

1,097

What is the gist or main message of THIS blog post?

There are lots of ways you can develop your L&D offer that just need you to think a bit bigger.

Why do you think this post is so popular with your readers?

It’s aspirational, it’s practical, it’s a bit evangelical.

Jo Cook

What is the name of your blog? (NOT YOUR URL)

Lightbulb Moment Blog

Blog Address:

https://lightbulbmoment.info

What do you try to do in your blog? What’s your focus, goal, or slant?

Virtual classroom and webinars, but also learning and development more broadly.

What is the URL of your most visited blog post?

https://lightbulbmoment.info/2018/04/04/what-is-digital-body-language/

What is the title of this blog post?

What is Digital Body Language?

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

704

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

6,189

What is the gist or main message of THIS blog post?

That there is an online equivalent of body language for when delivering virtual classrooms.

Why do you think this post is so popular with your readers?

It’s a new concept on an area where people don’t have much experience or confidence.

Ryan Tracey

What is the name of your blog? (NOT YOUR URL)

E-Learning Provocateur

Blog Address:

https://ryan2point0.wordpress.com

What do you try to do in your blog? What’s your focus, goal, or slant?

To provoke deeper thinking about digital learning in the corporate sector.

What is the URL of your most visited blog post?

https://ryan2point0.wordpress.com/2010/01/12/taxonomy-of-learning-theories/

What is the title of this blog post?

Taxonomy of Learning Theories

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

1,113

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

8,019

What is the gist or main message of THIS blog post?

A brief overview of, and a proposed taxonomy for, key theories that apply to workplace learning.

Why do you think this post is so popular with your readers?

Because practitioners are confused by different theories and are uncertain as to how they apply to their role.

Wilfred Rubens

What is the name of your blog? (NOT YOUR URL)

WilfredRubens.com over leren en ICT

Blog Address:

http://www.te-learning.nl/blog/

What do you try to do in your blog? What’s your focus, goal, or slant?

I am very interested in how technology can enhance and facilitate learning. I use my blog to share information about technology enhanced learning.

What is the URL of your most visited blog post?

http://www.te-learning.nl/blog/voor-en-nadelen-gebruik-sociale-media-door-jongeren/

What is the title of this blog post?

Voor- en nadelen gebruik sociale media door jongeren

Translated into English by Google: Pros and cons of the use social media by young people

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

16,515

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

14,227

What is the gist or main message of THIS blog post?

Overview of pro’s and con’s of the use of social media by young people. People who are currently arguing in favour of ‘ban the media’ are gradually putting themselves outside social reality. It is better to focus on sensible use, using the possibilities of social media. That is more effective than fighting against windmills.

Why do you think this post is so popular with your readers?

Actual for more than ten years. Content is appealing. Food for thought. Well indexed by search engines.

Julie Drybrough

What is the name of your blog? (NOT YOUR URL)

fuchsia blue blog

Blog Address:

https://fuchsiablueblog.wordpress.com

What do you try to do in your blog? What’s your focus, goal, or slant?

I write about organisational learning/ change/ culture as I see them. I write vignettes about my consultancy work and observation pieces about our profession/ field. I try to get folk thinking more deeply or oddly about what they do.

What is the URL of your most visited blog post?

https://fuchsiablueblog.wordpress.com/2016/03/05/the-power-of-humble-leadership/

What is the title of this blog post?

The Power of Humble Leadership

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

208

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

2,852

What is the gist or main message of THIS blog post?

Intro to fuchsia blue – about me etc

Why do you think this post is so popular with your readers?

It’s the front-page when you hit the site, generally

Neil Von Heupt

What is the name of your blog? (NOT YOUR URL)

Divergent Learning

Blog Address:

https://divergentlearning.wordpress.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

I try to write things that will add value to those who might read them.

What is the URL of your most visited blog post?

https://divergentlearning.wordpress.com/2017/08/25/a-human-social-media-experiment/

What is the title of this blog post?

A human social media experiment

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

99

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

163

What is the gist or main message of THIS blog post?

Being human on social media

Why do you think this post is so popular with your readers?

It had two things going for it – a trip down memory lane via childhood books, and a curated list of L&D reading material. And it was quite visual, so I guess that’s three!

Brett Christensen

What is the name of your blog? (NOT YOUR URL)

Workplace Performance

Blog Address:

https://workplaceperformanceblog.wordpress.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

Promote the science of performance improvement.

What is the URL of your most visited blog post?

https://workplaceperformanceblog.wordpress.com/2016/05/26/needs-assessment-or-needs-analysis/

What is the title of this blog post?

Needs Assessment or Needs Analysis

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

521

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

3,577

What is the gist or main message of THIS blog post?

There is a difference between needs assessment and needs analysis.

Why do you think this post is so popular with your readers?

Because it’s a common question and I hope I am writing in a way that connects with everyone.

Dennis Callahan

What is the name of your blog? (NOT YOUR URL)

Learnstreaming

Blog Address:

http://learnstreaming.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

Sharing my thoughts on workplace learning.

What is the URL of your most visited blog post?

http://learnstreaming.com/learning-means-believing-in-yourself/

What is the title of this blog post?

Learning Means Believing in Yourself

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

4,440

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

Not Sure

What is the gist or main message of THIS blog post?

To learn, you need to have confidence in yourself, You can do it!

Why do you think this post is so popular with your readers?

With so much change and uncertainty, you are always the constant. Trust and believe in yourself and you can grow.

Mirjam Neelen and Paul Kirschner

What is the name of your blog? (NOT YOUR URL)

3-Star Learning Experiences

Blog Address:

https://3starlearningexperiences.wordpress.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

Our blog aims to present learning professionals with evidence-informed ideas on how to make both the instructional and the learning experience more effective, efficient, and enjoyable. We discuss fads & fallacies, we try to find nuance, and we provide our readers with concrete ideas on how to design 3-star learning experiences based on the evidence out there!

What is the URL of your most visited blog post?

https://3starlearningexperiences.wordpress.com/2018/06/05/no-feedback-no-learning/

What is the title of this blog post?

No Feedback, No Learning

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

5,761

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

13,583

What is the gist or main message of THIS blog post?

This blog discusses why feedback is critical for learning as well as different types of feedback for learning and how they hurt or support learning.

Why do you think this post is so popular with your readers?

I think people realize that feedback is one of the most, if not the most important tools for supporting learning. Giving effective feedback has also been found to be one of the most powerful educational interventions to improve learning. Effective feedback positively affects learning outcomes and motivation to learn, and can help build accurate schema.

Christy Tucker

What is the name of your blog? (NOT YOUR URL)

Experiencing E-Learning

Blog Address:

https://christytucker.wordpress.com

What do you try to do in your blog? What’s your focus, goal, or slant?

Mostly instructional design for elearning, especially scenarios for workplace learning.

What is the URL of your most visited blog post?

https://christytucker.wordpress.com/2007/05/26/what-does-an-instructional-designer-do/

What is the title of this blog post?

No Feedback, No Learning

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

32,246

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

7,141

What is the gist or main message of THIS blog post?

It’s my explanation of what instructional designers do, as a response to all the times I have had to explain my work to others

Why do you think this post is so popular with your readers?

It answers a question people genuinely have (what is instructional design). It used to rank better in search engine results, but has dropped because the post is now over 10 years old.

Donald H Taylor

What is the name of your blog? (NOT YOUR URL)

Donald H Taylor

Blog Address:

http://donaldhtaylor.co.uk/

What do you try to do in your blog? What’s your focus, goal, or slant?

Share my thoughts on learning and performance at work.

What is the URL of your most visited blog post?

http://donaldhtaylor.co.uk/are-you-in-the-training-ghetto/

What is the title of this blog post?

Are you in the Training Ghetto?

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

Not Sure

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

Not Sure

What is the gist or main message of THIS blog post?

Many L&D departments run the risk of being sidelined as they fail to adapt to change at the same speed as the rest of the business.

Why do you think this post is so popular with your readers?

People recognise the issues involved, and can place themselves on the 2×2 grid I provide.

Matt Guyan

What is the name of your blog? (NOT YOUR URL)

Learn. Show. Repeat.

Blog Address:

http://www.mattguyan.com/

What do you try to do in your blog? What’s your focus, goal, or slant?

To share what I’m learning as well as my thoughts about eLearning.

What is the URL of your most visited blog post?

http://www.mattguyan.com/letter-to-an-elearning-creator/

What is the title of this blog post?

Letter to an eLearning Creator

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

4,760

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

2,615

What is the gist or main message of THIS blog post?

I was venting my frustration at many of the things that are wrong with eLearning!

Why do you think this post is so popular with your readers?

People seemed to relate to it either as a developer or user.

Guy Wallace

What is the name of your blog? (NOT YOUR URL)

Pursuing Performance

Blog Address:

https://eppic.biz/

What do you try to do in your blog? What’s your focus, goal, or slant?

Share on the topics and tasks of performance-based ISD and Performance Improvement, and pay it forward, as my many mentors had done for me.

What is the URL of your most visited blog post?

https://eppic.biz/2012/03/26/the-big-5-in-human-personality-assessments-canoe/

What is the title of this blog post?

The Big 5 in Human Personality Assessments: CANOE

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

12,815

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

43,428

What is the gist or main message of THIS blog post?

Sharing a valid approach to personality assessments: The Big Five factors are openness, conscientiousness, extraversion, agreeableness, and neuroticism (common acronyms are OCEAN, NEOAC, or CANOE).

Why do you think this post is so popular with your readers?

Those that become aware of the shortcoming of MBTI, DiSC, etc. in some uses within HR need something valid in its place.

Connie Malamed

What is the name of your blog? (NOT YOUR URL)

The eLearning Coach

Blog Address:

http://theelearningcoach.com

What do you try to do in your blog? What’s your focus, goal, or slant?

My focus is on designing learning experiences in the workplace to support adult learners and to help them improve their performance at work. I try to cover topics related to this, such as instructional design, cognitive psychology, visual design and multimedia, technology-based learning, etc.

What is the URL of your most visited blog post?

http://theelearningcoach.com/learning/10-definitions-learning/

What is the title of this blog post?

10 Definitions of Learning

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

90,730

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

25,541

What is the gist or main message of THIS blog post?

We all know that the human brain is immensely complex and still somewhat of a mystery. It follows then, that learning—a primary function of the brain—is understood in many different ways. Here are ten ways that learning can be described.

Why do you think this post is so popular with your readers?

I think the traffic is coming from any type of learning professional (IDers, teachers, professors, trainers) who are interested in the variety of ways to think about learning. They are trying to figure out what learning is or they have a philosophical interest. It could also be coming from students who need a definition for a paper they are writing.

Christopher Pappas

What is the name of your blog? (NOT YOUR URL)

eLearning Industry

Blog Address:

https://elearningindustry.com

What do you try to do in your blog? What’s your focus, goal, or slant?

The best collection of eLearning articles, eLearning concepts, eLearning software, and eLearning resources.

What is the URL of your most visited blog post?

https://elearningindustry.com/the-20-best-learning-management-systems

What is the title of this blog post?

The 20 Best Learning Management Systems (2018 Update)

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

151,345

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

288,334

What is the gist or main message of THIS blog post?

Choosing the right Learning Management System for the deployment of your eLearning courses might seem a daunting task. Care to find out about the best Learning Management Systems the eLearning industry has to offer? In this article, I’ll present a list of the 20 best Learning Management Systems for all needs and budgets.

Why do you think this post is so popular with your readers?

There are many organization that are looking to find or replace their Learning Management System. They are interested to read what eLearning Industry has to offer.

Michelle Ockers

What is the name of your blog? (NOT YOUR URL)

It doesn’t have a name.

Blog Address:

http://michelleockers.com/blog/

What do you try to do in your blog? What’s your focus, goal, or slant?

(1) Thought leadership – opinions on role and future of L&D. Increasingly targeted at business leaders. (2) Daily dispatches – Narrating my work (term from Austin Kleon) http://michelleockers.com/daily-dispatches/.

What is the URL of your most visited blog post?

http://michelleockers.com/mo-blog/how-we-modernised-our-learning-and-development-model-mindset-and-capabilities/

What is the title of this blog post?

How We Modernised our Learning and Development Model, Mindset and Capabilities

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

Not Sure

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

Not Sure

What is the gist or main message of THIS blog post?

Modernisation of L&D practices, mindset and capability.

Why do you think this post is so popular with your readers?

Case study. Tips.

Tracy Schiffmann

What is the name of your blog? (NOT YOUR URL)

Brain-Based and Trauma-Informed Teaching

Blog Address:

http://www.tracyschiffmann.com

What do you try to do in your blog? What’s your focus, goal, or slant?

Support teachers and trainers in working more effectively with trauma-impacted adult learners.

What is the URL of your most visited blog post?

http://tracy-schiffmann.squarespace.com/blog/2016/12/17/10-trauma-informed-classroom-strategies-for-navigating-behavior-emergencies

What is the title of this blog post?

10 Trauma-Informed Classroom Strategies for Navigating Classroom Behavior Emergencies

How many page views FOR THIS PARTICULAR BLOG POST in the last 12 MONTHS?

429

How many page views FOR YOUR BLOG’S HOME PAGE in the last 12 MONTHS?

66

What is the gist or main message of THIS blog post?

I introduce 10 practical strategies for responding to challenging classroom behavior that may be a result of trauma-impact. I then share a true story from my teaching experience and demonstrate how the strategies were used to respond to the upset adult learner.

Why do you think this post is so popular with your readers?

I have many readers who teach trauma-impacted adult learners in prisons, jails, community-based organizations, and community colleges. They see the impact of trauma on both behavior and on their student’s ability to learn. Institutionalized racism, natural disasters, bullying, childhood abuse, and even the current political climate are traumatizing.

Just today I wrote an article on Training and Climate Change and what, if anything, we workplace learning professionals can do about it.

See and comment on LinkedIn where I published the article. Click to go there now.

Respondents

Over 200 learning professionals responded to Work-Learning Research’s 2017-2018 survey on current practices in gathering learner feedback, and today I will reveal the results. The survey ran from November 29th, 2017 to September 16th, 2018. The sample of respondents was drawn from Work-Learning Research’s mailing list and through extensive calls for participation in a variety of social media. Because of this sampling methodology, the survey results are likely skewed toward professionals who care and/or pay attention to research-based practice recommendations more than the workplace learning field as a whole. They are also likely more interested and experienced in learning evaluation as well.

Feel free to share this link with others.

Goal of the Research

The goal of the research was to determine what people are doing in the way of evaluating their learning interventions through the practice of asking learners for their perspectives.

Questions the Research Hoped to Answer

  1. Are smile sheets (learner-feedback questions) still the most common method of doing learning evaluation?
  2. How does their use compare with other methods? Are other methods growing in prominence/use?
  3. How satisfied are learning professionals with their organizations’ learner-feedback methods?
  4. To what extent are organizations looking for alternatives to their current learner-feedback methods?
  5. What kinds of questions are used on smile sheets? Has Thalheimer’s new approach, performance-focused questioning, gained any traction?
  6. What do learning professionals think their current smile sheets are good at measuring (Satisfaction, Reputation, Effectiveness, Nothing)?
  7. What tools are organizations using to gather learner feedback?
  8. How useful are current learner-feedback questions in helping guide improvements in learning design and delivery?
  9. How widely are the target metrics of LTEM (The Learning-Transfer Evaluation Model) currently being measured?

A summary of the findings indexed to these questions can be found at the end of this post.

Situating the Practice of Gathering Learner Feedback

When we gather feedback from learners, we are using a Tier 3 methodology on the LTEM (Learning-Transfer Evaluation Model) or Level 1 on the Kirkpatrick-Katzell Four-Level Model of Training Evaluation.

Demographic Background of Respondents

Respondents came from a wide range of organizations, including small, midsize, and large organizations.

Respondents play a wide range of roles in the learning field.

Most respondents live in the United States and Canada, but there was some significant representation from many predominantly English-speaking countries.

Learner-Feedback Findings

About 67% of respondents report that learners are asked about their perceptions on more than half of their organization’s learning programs, including elearning. Only about 22% report that they survey learners on less than half of their learning programs. This finding is consistent with past findings—surveying learners is the most common form of learning evaluation and is widely practiced.

The two most common question types in use are Likert-like questions and numeric-scale questions. I have argued against their use* and I am pleased that Performance-Focused Smile Sheet questions have been utilized by so many so quickly. Of course, this sample of respondents is comprised of folks on my mailing list so this result surely doesn’t represent current practice in the field as a whole. Not yet! LOL.

*Likert-like questions and numeric-scale questions are problematic for several reasons. First, because they offer fuzzy response choices, learners have a difficult time deciding between them and this likely makes their responses less precise. Second, such fuzziness may inflate bias as there are not concrete anchors to minimize biasing effects of the question stems. Third, Likert-like options and numeric scales likely deflate learner responding because learners are habituated to such scales and because they may be skeptical that data from such scales will actually be useful. Finally, Likert-like options and numeric scales produce indistinct results—averages all in the same range. Such results are difficult to assess, failing to support decision-making—the whole purpose for evaluation in the first place. To learn more, check out Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form (book website here).

The most common tools used to gather feedback from learners were paper surveys and SurveyMonkey. Questions delivered from within an LMS were the next highest. High-end evaluation systems like Metrics that Matter were not highly represented in our respondents.

Our respondents did not rate their learner-feedback efforts as very effective. Their learner surveys were seen as most effective in gauging learner satisfaction. Only about 33% of respondents thought their learner surveys gave them insights on the effectiveness of the learning.

Only about 15% of respondents found their data very useful in providing them feedback about how to improve their learning interventions.

Respondents report that their organizations are somewhat open to alternatives to their current learner-feedback approaches, but overall they are not actively looking for alternatives.

Most respondents report that their organizations are at least “modestly happy” with their learner-feedback assessments. Yet only 22% reported being “generally happy” with them. Combining this finding with the one above showing that lots of organizations are open to alternatives, it seems that organizational satisfaction with current learner-feedback approaches is soft.

We asked respondents about their organizations’ attempts to measure the following:

  • Learner Attendance
  • Whether Learner is Paying Attention
  • Learner Perceptions of the Learning (eg, Smile Sheets, Learner Feedback)
  • Amount or Quality of Learner Participation
  • Learner Knowledge of the Content
  • Learner Ability to Make Realistic Decisions
  • Learner Ability to Complete Realistic Tasks
  • Learner Performance on the Job (or in another future performance situation)
  • Impact of Learning on the Learner
  • Impact of Learning on the Organization
  • Impact of Learning on Coworkers, Family, Friends of the Learner
  • Impact of Learning on the Community or Society
  • Impact of Learning on the Environment

These evaluation targets are encouraged in LTEM (The Learning-Transfer Evaluation Model).

Results are difficult to show—because our question was very complicated (admittedly too complicated)—but I will summarize the findings below.

As you can see, learner attendance and learner perceptions (smile sheets) were the most commonly measured factors, with learner knowledge a distant third. The least common measures involved the impact of the learning on the environment, community/society, and the learner’s coworkers/family/friends.

The flip side—methods rarely utilized in respondents’ organizations—shows pretty much the same thing.

Note that the question above, because it was too complicated, probably produced some spurious results, even if the trends at the extremes are probably indicative of the whole range. In other words, it’s likely that attendance and smile sheets are the most utilized and measures of impact on the environment, community/society, and learners’ coworkers/family/friends are the least utilized.

Questions Answered Based on Our Sample

  1. Are smile sheets (learner-feedback questions) still the most common method of doing learning evaluation?

    Yes! Smile sheets are clearly the most popular evaluation method, along with measuring attendance (if we include that as a metric).

  2. How does their use compare with other methods? Are other methods growing in prominence/use?

    Except for Attendance, nothing else comes close. The next most common method is measuring knowledge. Remarkably, given the known importance of decision-making (Tier 5 in LTEM) and task competence (Tier 6 in LTEM), these are used in evaluation at a relatively low level. Similar low levels are found in measuring work performance (Tier 7 in LTEM) and organizational results (part of Tier 8 in LTEM). We’ve known about these relatively low levels from many previous research surveys.

    Hardly any measurement is being done on the impact of learning on learner or his/her coworkers/family/friends, the impact of the learning on the community/society/environment, or on learner participation/attention.

  3. How satisfied are learning professionals with their organizations’ learner-feedback methods?

    Learning professionals are moderately satisfied.

  4. To what extent are organizations looking for alternatives to their current learner-feedback methods?

    Organizations are open to alternatives, with some actively seeking alternatives and some not looking.

  5. What kinds of questions are used on smile sheets? Has Thalheimer’s new approach, performance-focused questioning, gained any traction?

    Likert-like options and numeric scales are the most commonly used. Thalheimer’s performance-focused smile-sheet method has gained traction in this sample of respondents—people likely more in the know about Thalheimer’s approach than the industry at large.

  6. What do learning professionals think their current smile sheets are good at measuring (Satisfaction, Reputation, Effectiveness, Nothing)?

    Learning professionals think their current smile sheets are fairly good at measuring the satisfaction of learners. A full one-third of respondents feel that their current approaches are not valid enough to provide them with meaningful insights about the learning interventions.

  7. What tools are organizations using to gather learner feedback?

    The two most common methods for collecting learner feedback are paper surveys and SurveyMonkey. Questions from LMSs are the next most widely used. Sophisticated evaluation tools are not much in use in our respondent sample.

  8. How useful are current learner-feedback questions in helping guide improvements in learning design and delivery?

    This may be the most important question we might ask, given that evaluation is supposed to aid us in maintaining our successes and improving on our deficiencies. Only 15% of respondents found learner feedback “very helpful” in helping them improve their learning. Many found the feedback “somewhat helpful” but a full one-third found the feedback “not very useful” in enabling them to improve learning.

  9. How widely are the target metrics of LTEM (The Learning-Transfer Evaluation Model) currently being measured?

    As described in Question 2 above, many of the targets of LTEM are not being adequately measured at this point in time (November 2017 to September 2018, during the time immediately before and after LTEM was introduced). This indicates that LTEM is poised to help organizations uncover evaluation targets that can be helpful in setting goals for learning improvements.

Lessons to be Drawn

The results of this survey reinforce what we’ve known for years. In the workplace learning industry, we default to learner-feedback questions (smile sheets) as our most common learning-evaluation method. This is a big freakin’ problem for two reasons. First, our learner-feedback methods are inadequate. We often use poor survey methodologies and ones particularly unsuited to learner feedback, including the use of fuzzy Likert-like options and numeric scales. Second, even if we used the most advanced learner-feedback methods, we still would not be doing enough to gain insights into the strengths and weaknesses of our learning interventions.

Evaluation is meant to provide us with data we can use to make our most critical decisions. We need to know, for example, whether our learning designs are supporting learner comprehension, learner motivation to apply what they’ve learned, learner ability to remember what they’ve learned, and the supports available to help learners transfer their learning to their work. We typically don’t know these things. As a result, we don’t make design decisions we ought to. We don’t make improvements in the learning methods we use or the way we deploy learning. The research captured here should be seen as a wake up call.

The good news from this research is that learning professionals are often aware and sensitized to the deficiencies of their learning-evaluation methods. This seems like a good omen. When improved methods are introduced, they will seek to encourage their use.

LTEM, the new learning-evaluation model (which I developed with the help of some of the smartest folks in the workplace learning field) is targeting some of the most critical learning metrics—metrics that have too often been ignored. It is too new to be certain of its impact, but it seems like a promising tool.

Why I have turned my Attention to Evaluation (and why you should too!)

For 20 years, I’ve focused on compiling scientific research on learning in the belief that research-based information—when combined with a deep knowledge of practice—can drastically improve learning results. I still believe that wholeheartedly! What I’ve also come to understand is that we as learning professionals must get valid feedback on our everyday efforts. It’s simply our responsibility to do so.

We have to create learning interventions based on the best blend of practical wisdom and research-based guidance. We have to measure key indices that tell us how our learning interventions are doing. We have to find out what their strengths are and what their weaknesses are. Then we have to analyze and assess and make decisions about what to keep and what to improve. Then we have to make improvements and again measure our results and continue the cycle—working always toward continuous improvement.

Here’s a quick-and-dirty outline of the recommended cycle for using learning to improve work performance. “Quick-and-dirty” means I might be missing something!

  1. Learn about and/or work to uncover performance-improvement needs.
  2. If you determine that learning can help, continue. Otherwise, build or suggest alternative methods to get to improved work performance.
  3. Deeply understand the work-performance context.
  4. Sketch out a very rough draft for your learning intervention.
  5. Specify your evaluation goals—the metrics you will use to measure your intervention’s strengths and weaknesses.
  6. Sketch out a rough draft for your learning intervention.
  7. Specify your learning objectives (notice that evaluation goals come first!).
  8. Review the learning research and consider your practical constraints (two separate efforts subsequently brought together).
  9. Sketch out a reasonably good draft for your learning intervention.
  10. Build your learning intervention and your learning evaluation instruments (Iteratively testing and improving).
  11. Deploy your “ready-to-go” learning intervention.
  12. Measure your results using the previously determined evaluation instruments, which were based on your previously determined evaluation objectives.
  13. Analyze your results.
  14. Determine what to keep and what to improve.
  15. Make improvements.
  16. Repeat (maybe not every step, but at least from Step 6 onward)

And here is a shorter version:

  1. Know the learning research
  2. Understand your project needs.
  3. Outline your evaluation objectives—the metrics you will use.
  4. Design your learning.
  5. Deploy your learning and your measurement.
  6. Analyze your results.
  7. Make Improvements
  8. Repeat.

More Later Maybe

The results shared here are the result from all respondents. If I get the time, I’d like to look at subsets of respondents. For example, I’d like to look at how learning executives and managers might differ from learning practitioners. Let me know how interested you would be in these results.

Also, I will be conducting other surveys on learning-evaluation practices, so stay tuned. We have been too long frustrated with our evaluation practices and more work needs to be done in understanding the forces that keep us from doing what we want to do. We could also use more and better learning-evaluation tools because the truth is that learning evaluation is still a nascent field.

Finally, because I learn a ton by working with clients who challenge themselves to do more effective interventions, please get in touch with me if you’d like a partner in thinking things through and trying new methods to build more effective evaluation practices. Also, please let me know how you’ve used LTEM (The Learning-Transfer Evaluation Model).

Some links to make this happen:

Appreciations

As always, I am grateful to all the people I learn from, including clients, researchers, thought leaders, conference attendees, and more… Thanks also to all who acknowledge and share my work! It means a lot!

I use a toothbrush that has a design that research shows maximizes the benefits of brushing. It spins, and spinning is better than oscillations. It also has a timer, telling me when I’ve brushed for two minutes. Ever since a hockey stick broke up my mouth when I was twenty, I’ve been sensitive about the health of my teeth.

But what the heck does this have to so with learning and development? Well, let’s see.

Maybe my toothbrush is a performance-support exemplar. Maybe no training is needed. I didn’t read any instructions. I just used it. The design is intuitive. There’s an obvious button that turns it on, an obvious place to put toothpaste (on the bristles), and it’s obvious that the bristles should be placed against the teeth. So, the tool itself seems like it needs no training.

But I’m not so sure. Let’s do a thought experiment. If I give a spinning toothbrush to a person who’s never brushed their teeth, would they use it correctly? Would they use it at all? Doubtful!

What is needed to encourage or enable good tooth-brushing?

  • People probably need something to compel them to brush, perhaps knowledge that brushing prevents dental calamities like tooth decay, gum disease, bad breath—and may even prevent cognitive decline as in Alzheimer’s. Training may help motivate action.
  • People will probably be more likely to brush if they know other people are brushing. Tons of behavioral economics studies have shown that people are very attuned to social comparisons. Again, training may help motivate action. Interestingly, people may be more likely to brush with a spinning toothbrush if others around them are also brushing with spinning toothbrushes. Training coworkers (or in this case other family members) may also help motivate action.
  • People will probably brush more effectively if they know to brush all their teeth, and to brush near their gums as well—not just the biting surfaces of their teeth. Training may provide this critical knowledge.
  • People will probably brush more effectively if they are set up—probably if they set themselves up—to be triggered by environmental cues. For example, tooth-brushing is often most effectively triggered when people brush right after breakfast and right before they go to bed. Training people to set up situation-action triggering may increase later follow through.
  • People will probably brush more effectively if they know that they should brush for two minutes or so rather than just brushing quickly. Training may provide this critical knowledge. Note, of course, that the toothbrush’s two-minute timer may act to support this behavior. Training and performance support can work together to enable effective behavior.
  • People will be more likely to use an effective toothbrush if the cost of the toothbrush is reasonable given the benefits. The costs of people’s tools will affect their use.
  • People will be more likely to use a toothbrush if the design is intuitive and easy to use. The design of tools will affect their use.

I’m probably missing some things in the list above, but it should suffice to show the complex interplay between our workplace tools/practices/solutions and training and prompting mechanisms (i.e., performance support and the like).

But what insights, or dare we say wisdom, can we glean from these reflections? How about these for starters:

  • We could provide excellent training, but if our tools/practices/solutions are poorly designed they won’t get used.
  • We could provide excellent training, but if our tools/practices/solutions are too expensive they won’t get used.
  • Let’s not forget the importance of prior knowledge. Most of us know the basics of tooth brushing. It would waste time, and be boring, to repeat that in a training. The key is to know, to really know, not just guess, what our learners know—and compare that to what they really need to know.
  • Even when we seem to have a perfectly intuitive, well-designed tool/practice/solution let’s not assume that no training is needed. There might be knowledge or motivational gaps that need to be bridged (yes, the pun was intended! SMILE). There might be situation-action triggering sets that can be set up. There might be reminders that would be useful to maintain motivation and compel correct technique.
  • Learning should not be separated from design of tools/practices/solutions. We can support better designs by reminding the designers and developers of these objects/procedures that training can’t fix a bad design. Better yet, we can work hand in hand involved in prototyping the tool/training bundle to enable the most pertinent feedback during the design process itself.
  • Training isn’t just about knowledge, it’s also about motivation.
  • Motivation isn’t just the responsibility of training. Motivation is an affordance of the tools/practices/solutions themselves, it is borne in the social environment, it is subject to organizational influence, particularly through managers and peers.
  • Training shouldn’t be thought of as a one-time event. Reminders may be valuable as well, particularly around the motivational aspects (for simple tasks), and to support remembering (for tasks that are easily forgotten or misunderstood).

One final note. We might also train people to use the time when they are engaged in automated tasks—tooth-brushing for example—to reflect on important aspects of their lives, gaining from the learning that might occur or the thoughts that may enable future learning. And adding a little fun into mundane tasks. Smile for the tiny nooks and crannies of our lives that may illuminate our thinking!

 

 

Let’s find out by asking them!

And, let’s ask ourselves (workplace learning professionals) what we think senior leaders will tell us.

NOTE: This may take some effort on our part. Please complete the survey yourself and ask senior leaders at your organization (if your organization is 1000 people or more) to complete the survey.

 

The Survey Below is for both Senior Organizational Leaders AND for Workplace Learning Professionals.

We will branch you to a separate set of questions!

Answer the survey questions below, or you need it, here is a link to the survey.

 



Send me an email if you want to talk more about learning evaluation...

This contact form is deactivated because you refused to accept Google reCaptcha service which is necessary to validate any messages sent by the form.

One of the most common questions I get when I speak about the Performance-Focused Smile-Sheet approach (see the book’s website at SmileSheets.com) is “What can be done to get higher response rates from my smile sheets?”

Of course, people also refer to smile sheets as evals, level 1’s, happy sheets, hot or warm evaluation, response forms, reaction forms, etc. They also refer to both paper-and-pencil forms and online surveys. Indeed, as smile sheets go online, more and more people are finding that online surveys get a much lower response rate than in-classroom paper surveys.

Before I give you my list for how to get a higher response rate, let me blow this up a bit. The thing is, while we want high response rates, there’s something much more important than response rates. We also want response relevance and precision. We want the questions to relate to learning effectiveness, not just learning reputation and learner satisfaction. We also want the learners to be able to answer the questions knowledgeably and give our questions their full attention.

If we have bad questions — one’s that use Likert-like or numeric scales for example — it won’t matter that we have high response rates. In this post, I’m NOT going to focus on how to write better questions. Instead, I’m just tackling how we can motivate our learners to give our questions more of their full attention, thus increasing the precision of their responding while also increasing our response rates as well.

How to get Better Responses and Higher Response Rates

  1. Ask with enthusiasm, while also explaining the benefits.
  2. Have a trusted person make the request (often an instructor who our learners have bonded with).
  3. Mention the coming smile sheet early in the learning (and more than once) so that learners know it is an integral part of the learning, not just an add-on.
  4. While mentioning the smile sheet, let folks know what you’ve learned from previous smile sheets and what you’ve changed based on the feedback.
  5. Tell learners what you’ll do with the data, and how you’ll let them know the results of their feedback.
  6. Highlight the benefits to the instructor, to the instructional designers, and to the organization. Those who ask can mention how they’ve benefited in the past from smile sheet results.
  7. Acknowledge the effort that they — your learners — will be making, maybe even commiserating with them that you know how hard it can be to give their full attention when it’s the end of the day or when they are back to work.
  8. Put the time devoted to the survey in perspective, for example, “We spent 7 hours today in learning, that’s 420 minutes, and now we’re asking you for 10 more minutes.”
  9. Ensure your learners that the data will be confidential, that the data is aggregated so that an individual’s responses are never shared.
  10. Let your learners know the percentage of people like them who typically complete the survey (caveat: if it’s relatively high).
  11. Use more distinctive answer choices. Avoid Likert-like answer choices and numerical scales — because learners instinctively know they aren’t that useful.
  12. Ask more meaningful questions. Use questions that learners can answer with confidence. Ask questions that focus on meaningful information. Avoid obviously biased questions — as these may alienate your learners.

How to get Better Responses and Higher Response Rates on DELAYED SMILE SHEETS

Sometimes, we’ll want to survey our learners well after a learning event, for example three to five weeks later. Delayed smile sheets are perfectly positioned to find out more about how the learning is relevant to the actual work or to our learners’ post-learning application efforts. Unfortunately, prompting action — that is getting learners to engage our delayed smile sheets — can be particularly difficult when asking for this favor well after learning. Still, there are some things we can do — in addition to the list above — that can make a difference.

  1. Tell learners what you learned from the end-of-learning smile sheet they previously completed.
  2. Ask the instructor who bonded with them to send the request (instead of an unknown person from the learning unit).
  3. Send multiple requests, preferably using a mechanism that only sends these requests to those who still need to complete the survey.
  4. Have the course officially end sometime AFTER the delayed smile sheet is completed, even if that is largely just a perception. Note that multiple-event learning experiences lend themselves to this approach, whereas single-event learning experiences do not.
  5. Share with your learners a small portion of the preliminary data from the delayed smile sheet. “Already, 46% of your fellow learners have completed the survey, with some intriguing tentative results. Indeed, it looks like the most relevant topic as rated by your fellow learners is… and the least relevant is…”
  6. Share with them the names or job titles of some of the people who have completed the survey already.
  7. Share with them the percentage of folks from his/her unit who have responded already or share a comparison across units.

What about INCENTIVES?

When I ask audiences for their ideas for improving responses and increasing response rates, they often mention some sort of incentive, usually based on some sort of lottery or raffle. “If you complete the survey, your name will be submitted to have chance to win the latest tech gadget, a book, time off, lunch with an executive, etc.”

I’m a skeptic. I’m open to being wrong, but I’m still skeptical about the cost/benefit calculation. Certainly for some audiences an incentive will increase rates of completion. Also, for some audiences, the harms that come with incentives may be worth it.

What harms you might ask? When we provide an external incentive, we might be sending a message to some learners that we know the task has no redeeming value or is tedious or difficult. People who see their own motivation as caused by the external incentive are potentially less likely to seriously engage our questions, producing bad data. We’re also not just having an effect on the current smile sheet. When we incentivize people today, they may be less willing next time to engage in answering our questions. They may also be pushed into believing that smile sheets are difficult, worthless, or worse.

Ideally, we’d like our learners to want to provide us with data, to see answering our questions as a worthy and helpful exercise, one that is valuable to them, to us, and to our organization. Incentives push against this vision.

 

I must be in a bad mood — or maybe I’ve been unlucky in clinking on links — but this graphic is horrifying. Indeed, it’s so obviously flawed that I’m not even going to point out it’s most glaring problem. You decide!

One more editorial comment before the big reveal:  Why, why, why is the gloriously noble and important field of learning besieged by such crap!!!!

 

 

.

.

.

.

.

.

.

.

.

.

Why is the goal of a learning-focused game, “fun?”

A 2003 meta-analysis found that fitness training was likely to improve cognitive functioning in older adults.

I'm reprising this because it is one of Psychological Science's most cited articles as recently as September 1, 2016.

Fitness and Aging

The researchers examined 18 scientific studies and 197 separate effect sizes. They categorized measures of cognitive functioning into four categories as depicted above in the graph, including:

  • Executive functioning (the ability to plan, schedule, and generally engage in high-level decision-making).
  • Controlled processing (the ability to engage in simple decision-making).
  • Visuospatial processing (the ability to transform visual or spatial information).
  • Speed processing (the ability to make quick reactions).

As you can see in the graph above, overall the groups that exercised outperformed those who didn't.

 

Some Details:

  • Results were stronger for people 66-80 than for those 55-65 (judged by effect size), although all groups showed significant benefits from exercise.
  • Exercise for less than 30 minutes produced very little benefit compared to exercise for 30-60 minutes.
  • Females seemed to get more benefits from exercising, but the way comparisons were made makes this conclusion somewhat sketchy.
  • Those who engaged in both weight-training and cardio-training had slightly better results than those who did cardio alone.

 

Research Citation:

Colcombe, S., & Kramer, A. F. (2003). Fitness effects on the cognitive function of older adults: A meta-analytic study. Psychological Science, 14, 125-130.

 

More Information

Check out a 2009 blog post I wrote on aging and cognition, and test your knowledge with the quiz!!

Dr. Karl Kapp is one of the learning field’s best research-to-practice gurus! Legendary for his generosity and indefatigable energy, it is my pleasure to interview him for his wisdom on games, gamification, and their intersection.

His books on games and gamification are wonderful. You can click on the images below to view them on Amazon.com.

 

 

The following is a master class on games and learning:

 

Will (Question 1):

Karl, you’ve written a definitive exploration of Gamification in your book, The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education. As I read your book I was struck by your insistence that Gamification “is not the superficial addition of points, rewards, and badges to learning experiences.” What the heck are you talking about? Everybody knows that gamification is all about leaderboards, or so the marketplace would make us believe… [WINK, WINK] What are you getting at in your repeated warning that gamification is more complex than we might think?

Karl:

If you examine why people play games, the reasons are many, but often players talk about the sense of mastery, the enjoyment of overcoming a challenge, the thrill of winning and the joy of exploring the environment. They talk about how they moved from one level to another or how they encountered a “boss level” and defeated the boss after several attempts or how they strategized a certain way to accomplish the goal of winning the game. Or they describe how they allocated resources so they could defeat a difficult opponent. Rarely, if ever, do people who play games talk about the thrill of earning a point or the joy of being number seven on the leaderboard or the excitement of earning a badge just for showing up.

The elements of points, badges and leaderboards (PBLs) are the least exciting and enticing elements of playing games. So there is no way we should lead with those items when gamifying instruction. Sure PBLs play a role in making a game more understandable or in determining how far away a player is from the “best” but they do little to internally motivate players by themselves. Reliance solely on the PBL elements of games to drive learner engagement is not sustainable and not even what makes games motivational or engaging. It’s the wrong approach to learning and motivation. It’s superficial; it’s not deep enough to have lasting meaning.

Instead, we need to look at the more intrinsically motivating and deeper elements of games such as: challenge, mystery, story, constructive feedback (meaningful consequences) strategy, socialization and other elements that make games inherently engaging. We miss a large opportunity when we limit our “game thinking” to points, badges and leaderboards. We need to expand our thinking to include elements that truly engage a player and draw them into a game. These are the things that make games fun and frustrating and worth our investment in time.

 

Will (Question 2):

You wrote that “too many elements of reality and the game ceases to be engaging,”—and I’m probably taking this out of context—but I wonder if that is true in all cases? For example, I can imagine a realistic flight simulator for fighter pilots that creates an almost perfect replica of the cockpit, g-forces, and more, that would be highly engaging… On the other hand, my 13-year-daughter got me hooked on Tanki, an online tank shot-em-up game, and there are very few elements of reality in the game—and I, unfortunately, find it very engaging. Is it different for novices and experts? Are the recommendations for perceptual fidelity different for different topic areas, different learning goals, et cetera?

Karl:

A while ago, I read a fake advertisement for a military game. It was a parody. The fake game description described how the “ultra-realistic” military game would be hours of fun because it was just like actually being in the military. The description told the player that he or she would have hours of fun walking to the mess hall, maintaining equipment, getting gasoline for the jeep, washing boots, patrolling and zigging and cleaning latrines. Now none of these things are really fun, in fact, they are boring but they are part of the life of being in the military. Military games don’t include these mundane activities. Instead, you are always battling an enemy or strategizing what to do next. The actions that a military force performs 95% of the time are not included in the game because they are too boring.

 If games where 100% realistic, they would not be fun. So, instead, games are an abstraction of reality because they focus on things within reality that can be made engaging or interesting. If a game reflected reality 100%, there would be boring game play. Now certainly, games can be designed to “improve” reality and make it more fun. In the game, The Sims, you wake up, get dressed and go to work which all seems pretty mundane. However, these realistic activities in The Sims are an abstraction of the tasks you actually perform. The layer of abstraction makes the game more exciting, engaging and fun. But in either the military game case or The Sims, too much reality is not fun.

The flight simulator needs to be 100% realistic because it’s not really a game (although people do play it as a game) but the real purpose of a simulation is training and perfection of skills. A flight simulator can be fun for some people to “play” but in a 100% realistic simulator, if you don’t know what you are doing, it’s boring because you keep crashing. For someone who doesn’t know how to fly, like me. If you made a World War II air battle game which had 100% realistic controls for my airplane, it wouldn’t be fun. In game design, we need to balance elements of reality with the learning goal and the element of engagement.

For some people, a simulator can be highly engaging because the learner is performing the task she would do on the job. So there needs to be a balance in games and simulations to have the right amount of reality for the goals you are trying to achieve.

 

Will (Question 3):

In developing a learning game, what should come first, the game or the goals (of learning)?

Karl:

Learning goals must come first and must remain at the forefront of the game design process. Too often I see the mistake of a design team becoming too focused on games elements and losing site of the learning goals. In our field, we are paid to help people learn, not to entertain them. Learning first.

Having said that, you can’t ignore or treat the game elements as second class citizens, you can’t bolt-on a points system and think you have now developed a fun game—you haven’t. The best process involves simultaneously integrating game mechanics and learning elements. It’s tricky and not a lot of instructional designers have experience or training in this area but it’s critical to have integration of game and learning elements, the two need to be designed together. Neither can be an afterthought.

 

Will (Question 4):

Later we’ll talk about the research you’ve uncovered about the effectiveness of games. As I peruse the literature on games, the focus is mostly on the potential benefits of games. But what about drawbacks? I, for one, “waste” a ton of time playing games. Opportunity costs are certainly one issue, but maybe there are other drawbacks as well, including addiction to the endorphins and adrenaline; a heightened state of engagement during gaming that may make other aspects of living – or learning – seem less interesting, engaging. What about learning bad ideas, being desensitized to violence, sexual predation, or other anti-social behaviors? Are there downsides to games? And, in your opinion, has the research to date done enough to examine negative consequences of games?

Karl:

Yes, games can have horrible, anti-social content. They can also have wonderful, pro-social content. In fact, a growing area of game research focuses on possible pro-social aspects of games. The answer really is the content. A “game” per-say is neither pro- or anti-social like any other instructional medium. Look at speeches. Stalin gave speeches filled with horrible content and Martin Luther King, Jr. gave speeches filled with inspiring content. Yet we never seem to ask the question “Are speeches inherently good or bad?”

Games, like other instructional media, have caveats that good instructional designers need to factor when deciding if a game is the right instructional intervention. Certainly time is a big factor. It takes time to both develop a game and to play a game. So this is a huge downside. You need to weigh the impact you think the game will have on learner retention or knowledge versus another instructional intervention. Although, I can tell you there are at least two meta-analysis studies that indicate that games are more effective for learning than traditional, lecture-based instruction. But the point is not to blindly choose game over lecture or discussion. The decision regarding the right instructional design needs to be thoughtful. Knowing the caveats should factor into the final design decision.

Another caveat is that games should not be “stand-alone.” It’s far better for a learning game to be included as part of a larger curriculum rather than developed without any sense of how it fits into the larger pictures. Designers need to make sure they don’t lose site of the learning objective. If you are considering deploying a game within your organization, you have to make sure it’s appropriate for your culture. Another big factor to consider is how the losers are handled in the game. If a person is not successful at a game, what are the repercussions? What if she gets mad and shuts down? What if he walks away halfway through the experience because he is so frustrated? These types of contingencies need to be considered when developing a game. So, yes, there are downsides to games as there are downsides to other types of instruction. Our job, as instructional designers, is to understand as many downsides and upsides as possible for many different design possibilities and make an informed, evidence-based decision.

 

Will (Question 5):

As you found in your research review, feedback is a critical element in gaming. I’ve anointed “feedback” as one of the most important learning factors in my Decisive Dozen – as feedback is critical in all learning. The feedback research doesn’t seem definitive in recommending immediate versus delayed feedback, but the wisdom I take from the research suggests that delayed feedback is beneficial in supporting long-term remembering, whereas immediate feedback is beneficial in helping people “get” or comprehend key learning points or contingencies. In some sense, learners have to build correct mental models before they can (or should) reinforce those understandings through repetitions, reinforcement, and retrieval practice.

Am I right that most games provide immediate feedback? If not, when is immediate feedback common in games, when is delayed feedback common? What missed opportunities are there in feedback design?

Karl:

You are right; most games provide immediate, corrective feedback. You know right-away if you are performing the right action and, if not, the consequences of performing the wrong action. A number of games also provide delayed feedback in the form of after-action reviews. These are often seen in games using branching. At the end of the game, the player is given a description of choices she made versus the correct choices. So, delayed feedback is common in some types of games. In terms of what is missing in terms of feedback, I think that most learning games do a poor job of layering feedback. In well-designed video games, at the first level of help, a player can receive a vague clue. If this doesn’t work or too much time passes, the game provides a more explicit clue and finally, if that doesn’t work, the player receives step-by-step instructions. Most learning games are too blunt. They tend to give the player the answer right away rather than layers choices or escalating the help. I think that is a huge missed opportunity.

 

Will (Question 6):

By the way, your book does a really nice job in describing the complexity and subtlety of feedback, including Robin Hunicke’s formulation for what makes feedback “juicy.” What subtleties around feedback do most of us instructional designers or instructors tend to miss?

Karl:

Our feedback in learning games and even elearning modules is just too blunt. We need more subtlety. Hunicke describes the need for feedback to have many different attributes including the need for the feedback to be tactile and coherent. She refers to tactile feedback as creating an experience where the player can feel the feedback as it is occurring on screen so that it’s not forced or unnatural within the game play. Instructional designers typically don’t create feedback the player or learner feels, typically, they create feedback that is “in your face” such as “Nice job!” or “Sorry, try again.” She describes coherent feedback as feedback that stays within the context of the game. It is congruent with on screen actions and activities as well as with the storyline unfolding as the interactions occur. Our learning games seem to fail at including both of these elements in our feedback. In general, our field needs to focus on feedback that is more naturally occurring and within the flow of the learning.

 

Will (Question 7):

Do learners have to enjoy the game to learn from it? What are the benefits of game pleasure? Are there drawbacks at all?

Karl:

Actually, research by Tracy Sitzmann indicates (2011) that a learner doesn’t have to indicate that he or she was “entertained” to learn from a serious game. So fun should not be the standard by which we measure the success of game. Instead, she found that what makes a game effective for learning is the level of engagement. Engagement should be the goal when designing a learning game. However, there are a number of studies that indicate that games are motivational. Although, one meta-analysis on games indicated that motivation was not a factor. So, I am not sure if pleasure is a necessary factor for learning. Instead, I tend to focus more on building engagement and having learners make meaningful decisions and less on learner enjoyment and fun. This tends to run counter to why most people want a learning game but the reason we should want learning games is to encourage engagement and higher order thinking and not to simply make boring learning fun. Engagement, mastery and tough decision making might not always be fun but, as you indicated in your questions about simulations, it can be engaging and learning results from engagement and then understanding the consequences of actions taken during that engagement.

 

Will (Question 8):

As I was perusing research on games, one of my surprises was that games seemed to be used for health-behavior change at least as much as learning. What they heck’s going on?

Karl:

Games are great tools for promoting health. We all know that we should focus on health and wellness but we often let other life elements get in the way. Making staying healthy a game provides, in many cases that little bit of extra motivation to make you stay on course. I think games for health work so well because they capitalize on our already existing knowledge that we need to stay healthy and then provide tracking of progress, earning of points and other incentives to help us give that extra boost that makes us take the extra 100 steps needed to get our 10,000 for the day. Ironically, I find games used in many life and death situations.

 

Will (Question 9):

In your book you have a whole chapter devoted to research on games. I really like your review. Of course, with all the recent research, maybe we’ve learned even more. Indeed, I just did a search of PsycINFO (a database of scientific research in the social sciences). When I searched for “games” in the title, I found 110 articles in peer-reviewed journals in this year (2016) alone. That’s a ton of research on games!!

Let’s start with the finding in your book that the research methodology of much of the research is not very rigorous. You found that concern from more than one reviewer. Is that still true today (in 2016)? If the research base is not yet solid, what does that mean for us as practitioners? Should we trust the research results or should we be highly skeptical — OR, where in-between these extremes should we be?

Karl:

The short answer, as with any body of research, is to be skeptical but not paralyzed. Waiting for the definitive decision on games is a continually evolving process. Research results are rarely a definitive answer; they only give us guidance. I am sure you remember when “research” indicated that eggs were horrible for you and then “research” revealed that eggs were the ultimate health food. We need to know that research evolves and is not static. And, we need to keep in mind that some research indicated that smoking had health benefits so I am always somewhat skeptical. Having said that, I don’t let skepticism stop me from doing something. If the research seems to be pointing in a direction but I don’t have all the answers, I’ll still “try it out” to see for myself.

That said the research on games, even research done today, could be much more rigorous. There are many flaws which include small sample sizes, no universal definition of games and too much focus on comparing the outcomes of games with the outcomes of traditional instruction. One would think that argument would be pretty much over but decade after decade we continue to compare “traditional instruction” with radio, television, games and now mobile devices. After decades of research the findings are almost always the same. Good design, regardless of the delivery medium, is the most crucial aspect for learning. Where the research really needs to go, and it’s starting to go in that direction, is toward comparing elements of games to see which elements lead to the most effective and deep learning outcomes. So, for example, is the use of a narrative more effective in a learning game than the use of a leaderboard or is the use of characters more critical for learning than the use of a strategy-based design? I think the blanket comparisons are bad and, in many cases, misleading. For example, Tic-Tac-Toe is a game but so is Assassin’s Creed IV. So to say that all games teach pattern recognition because Tic-Tac-Toe teaches pattern recognition is not good. As Clark Aldrich stated years ago, the research community needs some sort of taxonomy to help identify different genres of games and then research into the learning impact of those genres.

So, I am always skeptical of game research and try to carefully describe limitations of the research I conduct and to carefully review research that has been conducted by others. I tend to like meta-analysis studies which are one method of looking at the body of research in the field and then drawing conclusions but even those aren’t perfect as you have arguments about what studies were included and what studies were not included.

At this point I think we have some general guidelines about the use of games in learning. We know that games are most effective in a curriculum when they are introduced and described to the learners, then the learners play the game and then there is a debrief. I would like to focus more on what we know from the research on games and how to implement games effectively rather than the continuous, and in my opinion, pointless comparison of games to traditional instruction. Let’s just focus on what works when games do provide positive learning outcomes.

 

Will (Question 10):

A recent review of serious games (Tsekleves, Cosmas, & Aggoun, 2014, 2016) concluded that their benefits were still not fully supported. “Despite the increased use of computer games and serious games in education, there are still very few empirical studies with conclusive results on the effectiveness of serious games in education.” This seems a bit strong given other findings from recent meta-analyses, for example the moderate effect sizes found in a meta-analysis from Wouters, van Nimwegen, van Oostendorp, & van der Spek (2013).

Can you give us a sense of the research? Are serious games generally better, sometimes better, or rarely better than conventional instruction? Or, are they better in some circumstance, for some learners, for some topics – rather than others? How should us practitioners think about the research findings?

Karl:

Wouters et al. (2013) found that games are more effective than traditional instruction as did Stizmann (2011). But, as you indicated, other meta-analysis studies have not come to that conclusion. So, again, I think the real issue is that the term “games” is way too broad for easy comparisons and we need to focus more on the elements of games and how the individual elements intermingle and combine to cause learning to occur. One major problem with research in the field of games is that to conduct effective and definitive research we often want to isolate one variable and then keep all other variables that same. That processes is extremely difficult to do with games. New research methods might need to be invented to effectively discover how game variables interact with one another. I even saw an article that declared that all games are situational learning and should be studied in that context rather than in an experimental context. I don’t know the answer but there are few simple solutions to game-based research and definitive declarations of the effectiveness of games.

However, having said all that, here are some things we do know from the research related to using games for learning:

  • Games should be embedded in instructional programs. The best learning outcomes from using a game in the classroom occur when a three-step embedding process is followed. The teacher should first introduce the game and explain its learning objectives to the students. Then the students play the game. Finally, after the game is played, the teacher and students should debrief one another on what was learned and how the events of the game support the instructional objectives. This process helps ensure that learning occurs from playing the game (Hays, 2005; Sitzmann, 2011).
  • Ensure game objectives align with curriculum objectives. Ke (2009) found that the learning outcomes achieved through computer games depend largely on how educators align learning (i.e., learning subject areas and learning purposes), learner characteristics, and game-based pedagogy with the design of an instructional game. In other words, if the game objectives match the curriculum objectives, disjunctions are avoided between the game design and curricular goals (Schifter, 2013). The more closely aligned curriculum goals and game goals, the more likely the learning outcomes of the game will match the desired learning outcomes of the student.
  • Games need to include instructional support. In games without instructional support such as elaborative feedback, pedagogical agents, and multimodal information presentations (Hays, 2005; Ke, 2009; Wouters et al., 2013)., students tend to learn how to play the game rather than learn domain-specific knowledge embedded in the game. Instructional support that helps learners understand how to use the game increases the effectiveness of the game by enabling learners to focus on its content rather than its operational rules.
  • Games do not need to be perceived as being “entertaining” to be educationally effective. Although we may hope that Maria finds the game entertaining, research indicates that a student does not need to perceive a game as entertaining to still receive learning benefits. In a meta-analysis of 65 game studies, Sitzmann (2011) found that although “most simulation game models and review articles propose that the entertainment value of the instruction is a key feature that influences instructional effectiveness, entertainment is not a prerequisite for learning,” that entertainment value did not impact learning (see also Garris et al., 2002; Tennyson & Jorczak, 2008; Wilson et al., 2009). Furthermore, what is entertaining to one student may not be entertaining to another. The fundamental criterion in selecting or creating a game should be the learner’s actively engagement with the content rather than simply being entertained (Dondling, 2007; Sitzmann, 2011).

 

Will (Question 11):

If the research results are still tentative, or are only strong in certain areas, how should we as learning designers think about serious games? Is there overall advice you would recommend?

Karl:

First of all, I’d like to point to the research that exists indicating that lectures are not as effective for learning as some believe. So practitioners, faculty members and others have defaulted to lectures and held them up as the “holy grail” of learning experiences when the literature clearly doesn’t back up the use of lectures as the best method for teaching higher level thinking skills. If one wants to be skeptical of learning designs, start with the lecture.

Second, I think the guidelines outlined above are a good start. We are literally learning more all the time so keep checking to see the latest. I try to publish research on my blog (karlkapp.com) and at the ATD Science of Learning blog and, of course, the Will at Work blog for all things learning research are good places to look.

Third, we need to take more chances. Don’t be paralyzed waiting for research to tell you what to do. Try something, if you fail, try something else. Sure you can spend your career creating safe PowerPoint-based slide shows where you hit next to continue but that doesn’t really move your career or the field forward. Take what is known from reading books and from vetted and trusted internet sources and make professionally informed decisions.

 

Will (Question 12):

Finally, if we decide to go ahead and develop or purchase a serious game, what are the five most important things people should know?

Karl:

  1. First clearly define your goals. Why are you designing or purchasing a serious game and what do you expect as the outcome? After the learners play the game what should they be able to do? How should they think? What result do you desire? Without a clearly defined outcome, you will run into problems.
  2. Determine how the game fits into your overall learning curriculum. Games should not be stand-alone; they really should be an integral part of a larger instructional plan. Determine where the serious game fits into the bigger picture.
  3. Consider your corporate culture. So cultures will allow a fanciful game with zombies or strange characters and some will not. Know what your culture will tolerate in terms of game look and feel and then work within those parameters.
  4. If the game is electronic, get your information technology (IT) folks involved early. You’ll need to look at things like download speed, access, browser compatibility and a host of other technical issues that you need to consider.
  5. Think carefully and deeply before you decide to develop a game internally. Developing good, effective serious games is tough. It’s not a two-week project. Partner with a vendor to obtain the desired result.
  6. (A bonus) Don’t neglect the power of card games or board games for teaching. If you have the opportunity to bring learners together, consider low-tech game solutions. Sometimes those are the most impactful.

 

Will (Question 13):

One of your key pieces of advice is for folks to play games to learn about their power and potential. What kind of games should we choose to play? How should we prioritize our game playing? What kind of games should we avoid because they’ll just be a waste of time or might give us bad ideas about games for learning?

Karl:

I think you should play all types of games. First, pick different types of games from a delivery perspective so pick some card games, board games, causal games on your smartphone and video games on a game console. Mix it up. Then play different types of games such as role-play games, cooperative games, matching games, racing games, games where you collect items (like Pokémon Go). The trick is to not just play games that you like but to play a variety of games. You want to build a “vocabulary” of game knowledge. Once you’ve built a vocabulary, you will have a formidable knowledge base on which to draw when you want to create a new learning game.

Also, you can’t just play the games. You need to play and critically evaluate the games. Pay attention to what is engaging about the game, what is confusing, how the rules are crafted, what game mechanics are being employed, etc.? Play games with a critical eye. Of course, you will run the danger of ruining the fun of games because you will dissect any game you are playing to determine what about the game is good and what is bad but, that’s ok, you need that skill to help you design games. You want to think like a game designer because when you create a serious game, you are a game designer. Therefore, the greater the variety of game you the play and dissect, the better game designer you will become.

 

Will (Question 14):

If folks are interested, where can they get your book?

Karl:

Amazon.com is a great place to purchase my book or at the ATD web site. Also, if people have access to Lynda.com, I have several courses on Lynda including “The Gamification of Learning”. And I have a new book coming out in January co-authored by my friend Sharon Boller called “Play to Learn” where we walk readers through the entire serious game design process from conceptualization to implementation. We are really excited about that book because we think it will be very helpful for people who want to create learning games.

 

You can click on the images below to view Karl’s Gamification books on Amazon.com.

 

 

 

 

Research

Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528.

Tsekleves, E., Cosmas, J., & Aggoun, A. (2016). Benefits, barriers and guideline recommendations for the implementation of serious games in education for stakeholders and policymakers. British Journal of Educational Technology, 47(1), 164-183. Available at: http://onlinelibrary.wiley.com/doi/10.1111/bjet.12223/pdf

Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249-265. http://dx.doi.org/10.1037/a0031311

I was speaking to a CLO (Chief Learning Officer) today about research-to-practice stuff, and he made a statement that blew my mind. It was brilliant!

First, a little background.

He’s been a CLO at two organizations with over a decade of tenure in learning-executive positions. He knows what he’s talking about.

He talked persuasively about how many learning and development units are stuck in the old-world view of learning as delivering training (and training alone). In such learning cultures, L&D folks are order takers. A more enlightened approach includes training, but it also involves performance-consulting, performance-facilitation, knowledge management, improved evaluations, etc., with a key goal of helping to create a partnership between L&D and its organizational stakeholders.

Okay, that’s not news. We know this is the right thing to do.

Here’s what struck me as illuminating. He said that what’s really needed to make the leap from the old way to the new way, you need a large capital-like investment to get you started, to bring in committed change agents, to overcome inertia, to have resources and tools to let you leverage research-based methods. Without a large resource infusion, you get incremental change, but it’s just not enough to change the culture so you end up fighting long exhausting battles with only some success.

But what CEO’s are going to be so enlightened as to pony up the dough?

You’d think they might. After all, making huge investments upfront is what successful businesses do, what successful entrepreneurs do, what successful athletes do, etc.

We in L&D have to figure out a way to convince our CEO’s to make the investment. Otherwise, we’ll likely remain in the purgatory of order taking henceforth.