The ISPI Conference-Session Feedback Form was designed with the following ideas in mind:

Why do the questions ask what they do?

  1. ISPI is devoted to performance-improvement so the form should focus on factors known to support workplace performance improvement, including such factors as:
    • Comprehension of concepts taught.
    • Likelihood of achieving expected work results.
    • Reinforcement through practice–going beyond lecture.
    • Likelihood of sharing concepts with others.
    • Validity of the concepts taught as supported by research and evidence.
  2. Conference attendees have a reasonable expectation to receive sessions that they feel are valuable, as evidenced by the following:
    • Level of engagement they experienced.
    • Importance of the concepts taught.
  3. Open-ended questions often provide the best feedback.


Why are the questions designed the way they are?

  1. The form was designed to fit on one page, to make it useable in a conference context.
  2. The answer options attempt to provide — as much a possible in a small space — learners with options that provide enough granularity so they can make clear distinctions when deciding between options.
  3. Such answer-option specificity has the added advantage of providing clearer feedback to presenters and conference organizers about the success of the session.
  4. Answer options are specifically designed NOT to use numbers, nor scaled adjectives (e.g., “strongly agree,” “agree,” etc.) — as both approaches make it difficult for learners to make precise decisions and for stakeholders to understand the meaning of the results.


How was the form designed?

  1. Will Thalheimer was asked by ISPI leadership to design the form, based on his work on performance-focused smile sheets and his upcoming book. This web page sits on his blog.
  2. After Will’s initial input, feedback was solicited and improvements were made to the form. Special thanks to Bonnie Beresford for providing important feedback and partnering in piloting this form and another form.
  3. Both forms will be piloted in 2015 and additional improvements will be considered.


We’d love your feedback!

The form has been modified to enable feedback on the form directly. However, if you’re interested in making additional comments, please use the form below to provide feedback. Thank you!!!


The "Learning Benefits of Questions" is a research-to-practice report on how to use questions to boost learning results. First published back in 2003, and partially funded by Questionmark (to whom I am still grateful), the Learning Benefits of Questions was inspired by fundamental learning research, provided a practical perspective, and even provided a diagnostic to help readers determine how well they understood questions for learning.

Still getting requests and seeing people refer to this research-to-practice report, I've decided to reissue the report, with a few minor improvements. You can download the report using the following link:

Download Learning Benefits of Questions 2014 v2.0

You can also see our other papers, articles, and job aids at the Work-Learning Research catalog.

I just read a vendor blog post that lists the pros and cons of gamification.

PLEASE, let us be smarter than this!

Gamification is NOT a THING !!!!!!!!!!!!!!!!!!!!!!!

There can be NO pros and cons to gamification…

Gamification is a label for dozens of specific factors, each of which can be used or not used, or used alone or in concert with other gamification learning factors.

Here is a small list of gamification factors (just off the top of my head):

  1. competing against a standard
  2. competing against others
  3. being given some sort of non-tangible "award" for perseverance
  4. being given some sort of non-tangible "award" for some level of success
  5. being given some sort of non-tangible "award" randomly as you "play"
  6. working on a team
  7. escaping a threat
  8. working toward a specific goal
  9. performing with a time constraint
  10. et cetera (ad infinitum?)

[Hey, if anybody has published a list of gamification factors, let me know and I'll post it.]

Seriously, when we oversimplify, we not only show our ignorance of the magical complexity of human learning and cognition, we also hurt our own thinking and problem solving and those of every person with whom we are communicating.

Sure, some vendor wants to sell gamification. I get that. But what is really being said is: some vendor wants to sell gamification to the most vulnerable within our profession (newbies, etc.) and even to the less vulnerable and even to the best-and-brightest who may have a temporary brain freeze from such miscommunication.

No, no, no! Sorry! That's too cynical, right! Probably the vendor honestly thinks that their list of pros and cons is being helpful. Probably the vendor doesn't really understand that all of us must look more deeply than our industry's surface ripples.

How to end this blog post? Hmmm. This is difficult. I'm not sure. Okay, I got it.

Final advice:

  1. Evaluate the labels used in the learning industry.
  2. Seek the constituent factors.
  3. Get data on their causal effects.
  4. Hire learning experts from time to time to reality check your learning designs.
  5. Give yourself a gold star for reading this blog post to the end.
  6. You have reached WAWL Level 2, performing better than 92.4% of your colleagues!
  7. To get to WAWL Level 3, do a Google Search of Gamification, find a list of Gamification Factors, and send me the link.
  8. To get to WAWL Level 4, create your own list, reflect on what you discover, post it somewhere, and send me the link.
  9. May the forces of the Neo Elepha keep you safe on your journey. WHO-LA!


New research shows that while people tend to avoid saying hello to strangers, when they do say hello, they are happier for it. Read this nice article in the New York Times which describes the research.

I add this article to this blog on learning, because it reminded me that we in the learning field often put too much trust on our learners. The bottom line is that learners often don't know how best to learn.

This means that we shouldn't willy-nilly design our learning to enable our learners to do anything they want to. Sometimes this can be beneficial because it gives learners a sense of autonomy and it can help them get information they need–but more often than not, it enables learners to make bad decisions about their own learning.

The article also reminded me about research on creativity which shows that domain spanners–people who spend time interacting with others outside their main area of interest are often more creative because they draw from other spheres of thought. It's interesting that we are shy talking to strangers, even though doing so will not only make us happier, but it may enrich us ways we can't imagine.

To put it simply, subscription learning is a radical new paradigm in elearning. It enables us to think about elearning in new ways–in ways that challenge our old mental models of elearning.

Elearning is still a relatively young field, having its start in the 1960’s during the advent of the computer age and gradually gaining a critical mass after the internet became a mass phenomenon. Because it’s a young field, we are still learning how to think about elearning. With each new paradigm, we think more deeply, more fully about what elearning is–and can be. Below is my categorization of the most important elearning paradigms as of 2014.

eLearning Paradigms 2014

  • Content Presenter (enables content to be presented to learners)
  • Comprehension Tester (enables learners’ knowledge to be tested–and feedback provided)
  • Practice Provider (enables learners’ decision-making to be tested–and feedback provided)
  • Performance Supporter (enables performers to be prompted toward action)
  • Reminder (enables learners or performers to be reminded to learn and/or take action)
  • Social Augmentation Provider (enables learners to learn from and with each other)
  • Gamification Provider (provides motivational incentives and behavioral prompts to action)
  • Mobile Learning Provider (provides learning and/or performance support through mobile technology)
  • Data Utilizer (enables data collection and data-based interventions)
  • Video Provider (enables video to be utilized in various ways)
  • Learning Organizer (provides organizational structure around learning opportunities)
  • Personalizer (enables content or prompting to be individualized or tailored)
  • Learning-Delivery Augmenter (enables easy delivery of content or prompting)
  • Context-Based Triggerer (enables content or prompting to be delivered depending on context)
  • Cost Saver (enables learning to be delivered at a lower cost)

I’m sure that I’m missing some elearning paradigms. You might have noticed that I’m only listing elearning memes that have a positive connotation. I am not mentioning such things as boring, trivial, poorly-designed. Also, some of the list may not be true, or may not always be true. For example, I’ve recently read research that shows that elearning is not often a cost saver. The bottom line, however, is that the list above represents a good number of the ways in which we tend to think about elearning.

Here’s the thing: The paradigms listed above represent the dominant mental models we use when we think about elearning. As Thomas Kuhn wrote many years ago, paradigms are a double-edge sword. On the one hand, they help us think. On the other hand, they put boundaries on what we think. For us in the learning field, we get both benefits and costs from our elearning paradigms. They help us consider ways that we might design or utilize elearning. On the darker side, they constrict our thinking. One of the reasons we created the eLearningManifesto was to get the field to think beyond some of its weaker paradigms.

Our Thinking on eLearning is Still Evolving 500px

The Subscription-Learning Meme

I am pushing the idea of subscription learning not just because it is aligned with the learning research on the spacing effect, but also because it gives us a completely new way of thinking about elearning. It opens elearning to new possibilities.

Where we often think of elearning content delivery as requiring relatively long events of 30 minutes or more, subscription learning lets us think of much shorter events spaced over time. Where we often think of performance support as being delivered through a single-focus system at a time of known need, subscription learning can prompt a series of thoughts or actions even when learners don’t know they need to know. Where we think that learners have to seek their own learning nuggets, subscription learning can push learning to learners to better support learning as a process.

For folks going to the ISPI Conference in Indianapolis next week, here are my session titles.

For details, specific timing, and slides, click here.


Speed Mentoring

Special Session
The Serious eLearning Manifesto:
An Effort to Help eLearning Live Up to its Promise

General Session (RTP)
The Decisive Dozen–Research-Supported Learning Factors

Pit Stop & Refuel: Roundtable & Lunch Discussion 
Insight Learning: Helping Employees Have Creative Insights

In the Decisive Dozen–the list of the twelve most important learning factors–Content Validity is Rule #1. If you're training your learners on bad content, you're doing more harm than good.

The TSA has apparently broken this rule to the tune of One Billion Dollars, spending money to train TSA agents to read body language–when there is no scientific evidence that people can actually detect liars.

This has to be one of the most costly training errors in the history of training and development! Congratulations TSA officials…

If you want to test your own skill at detecting liars, see this nice NY Times interactive.

I'm teaching the following half-day workshop at the upcoming ISPI conference.

This workshop goes for $150, which is an absolutely incredible bargain…

Click here to learn more about the workshop or the conference.


Radically New Research-Inspired Smile Sheet Design
Will Thalheimer, Work-Learning Research, Inc.
Workshop Code: WUQ

Research shows that traditional smile sheets are not related to learning results. Given this, we could just ignore smile sheets. However, because most organizations want to use smile sheets anyway, shouldn’t we try to make them better? In this workshop, Thalheimer will show how to create a “performance-focused smile sheet,” a radical improvement in smile sheet design–one that has been inspired by his years of research-based practice. The new design involves two meta-improvements. First, it is a smile sheet design that seeks information about key performance factors; and second, it is a smile sheet design that creates outputs that are more meaningful than data from scale responses. Performance-focused smile sheets will never be a one-source learning-feedback mechanism, but they provide a significant improvement over the traditional smile sheet.

Hello ISPI Attendees!!

Thanks for your interest!!

My public sessions are detailed below, and the slides are available as a download for my session on the Decisive Dozen. Other handouts will be conveniently delivered during the sessions.

If you’d like to set up a private meeting to discuss your organization’s issues while we’re at the conference, feel free to email me at: will.thalheimer@work-learning (dot) com or just grab me at the conference…



Speed Mentoring

Sunday April 13, 2 to 5PM

I will be one of many mentors available for quick mentoring sessions.
Looks like a great group of folks. Should be fun!!




Special Session
The Serious eLearning Manifesto:
An Effort to Help eLearning Live Up to its Promise

Tuesday April 15 – 10:00-11:15 am

Handouts Handed Out at Session

The eLearning Manifesto was recently released on March 13, 2014. An effort by a small group of elearning thought-leaders, the Manifesto aims to help elearning live up to its promise by challenging members of the elearning industry to take personal responsibility for building more effective elearning. The Manifesto is free and available at It offers 22 principles as guidance for elearning developers, elearning buyers, and elearning advocates. Many of its principles come right out of the ISPI playbook. The Manifesto was designed to get a conversation going–and that is the aim of this session. Facilitated by one of the Manifesto authors, we will review the actual Manifesto, discuss its origins, debate its strengths and limitations, and consider how to make it practically useful.

Participants will be able to:

• Describe the rationale behind the need for improved elearning.
• Articulate the values inherent in the Serious eLearning Manifesto.
• Consider the Manifesto’s strengths and limitations.
• Plan for using the Manifesto to drive improved elearning



General Session (RTP)
The Decisive Dozen–Research-Supported Learning Factors

Wednesday April 16 – 10:00-11:15 am

Download Slides:  Download Thalheimer — Decisive Dozen — ISPI 2014

For decades, the workplace learning and performance field has been swamped with fads and misconceptions that harm learners and depress learning results. Research can provide salvation, but only if it is made accessible and easy to use. After 15 years of research, Dr. Will Thalheimer has developed a dozen learning factors that–if implemented–can improve learning results dramatically. These “decisive dozen” will be detailed in a forthcoming book and have been published in a research review.

Participants will be able to:

• Radically improve their learning designs and learning results.
• Conceptualize learning in terms of comprehension, remembering, and application.
• Access a research review of 12 key learning factors.



Pit Stop & Refuel: Roundtable & Lunch Discussion 
Insight Learning: Helping Employees Have Creative Insights

Wednesday April 16 – 11:30 am-1:30 pm

Handouts Handed Out at Session

Most learning interventions target “transfer learning,” learning that is transferred from a person (or instructional program) to one or more learners. But people learn when they have insights as well.“Insight learning” can be targeted by workplace learning professionals to enable employees to have more and higher-quality creative insights.


ALL Half-Day Workshops CANCELLED, Including Mine:

Preconference Workshop
Radically New Research-Inspired Smile Sheet Design
Workshop Code: WUQ

Sorry it was cancelled! If interested in this workshop delivered to your company or available to you online, contact me at: will.thalheimer@work-learning (dot) com



Does Subscription Learning have legs?

It's too early to tell, but the eLearning Guild is buying Google Ads on "subscription learning."

Capture of Guild Advertisement