Update January 2018: To see my latest recommendations for smile-sheet question design, go to this web page.


The Original Post:

The response to the book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, has been tremendous! Since February, when it was published, I’ve received hundreds of thank you’s from folks the world over who are thrilled to have a new tool — and to finally have a way to get meaningful data from learner surveys. At the ATD conference where I spoke recently, the book sold out it was so popular! If you want to buy the book, the best place is still SmileSheets.com, the book’s website.

Since publication, I’ve begun a research effort to learn how companies are utilizing the new smile-sheet approach — and to learn what’s working, what the roadblocks are, and what new questions they’ve developed. As I said in the book in the chapter that offers 26 candidate questions, I hope that folks tailor questions, improve them, and develop new questions. This is happening, and I couldn’t be more thrilled. If your company is interested in being part of my research efforts, please contact me by clicking here. Likewise, if you’ve got new questions to offer, let me know as well.

Avoiding Issues of Traditional Smile Sheets

Traditional smile sheets tend to focus on learners’ satisfaction and learners’ assessments of the value of the learning experience. Scientific research shows us that such learner surveys are not likely to be correlated with learning results. Performance-focused smile sheets offer several process improvements:

  1. Avoid Likert-like scales and numerical scales which create a garbage-in garbage-out problem, which don’t offer clear delineations between answer choices, which don’t support respondent decision-making, and which open responding to bias.
  2. Instead, utilize concrete answer choices, giving respondents more granularity, and enabling much more meaningful results.
  3. In addition to, or instead of, focusing on factors related to satisfaction and perceived value; target factors that are related to learning effectiveness.

New Example Questions

As new questions come to my attention, I’ll share them here on my blog and elsewhere. You can sign up for my email newsletter if you want to increase the likelihood that you’ll see new smile-sheet questions (and for other learning-research related information as well).

Please keep in mind that there are no perfect assessment items, no perfect learning metrics, and no perfect smile-sheet questions. I’ve been making improvements to my own workshop smile sheets for years, and every time I update them, I find improvements to make. If you see something you don’t like in the questions below, that’s wonderful! When evaluating an assessment item, it’s useful to ask whether the item (1) is targeting something important and (2) is it better than other items that we could use or that we’ve used in the past.

Question Example — A Question for Learners’ Managers

My first example comes from a world renowned data and media company. They decided to take one of the book’s candidate questions, which was designed for learners to answer, and modify the question to ask learners’ managers to answer. Their reasoning: The training is strategically important to their business and they wanted to go beyond self-report data. Also, they wanted to send “stealth messages” to learners’ managers that they as managers had a role to play in ensuring application of the training to the job.

Here’s the question (aimed at learners’ managers):

In regard to the course topics taught, HOW EFFECTIVELY WAS YOUR DIRECT REPORT ABLE to put what he/she learned into practice in order to PERFORM MORE EFFECTIVELY ON THE JOB?

A. He/she has NOT AT ALL ABLE to put the concepts into practice.

B. He/she has GENERAL AWARENESS of the concepts taught, but WILL NEED MORE TRAINING / GUIDANCE to put the concepts into practice.

C. He/she WILL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.

D. He/she is at a FULLY COMPETENT LEVEL in using the concepts taught.

E. He/she is at an EXPERT LEVEL in using the concepts taught.

Question Example — Tailoring a Question to the Topic

In writing smile-sheet questions, there’s a tradeoff between generalization and precision. Sometimes we need a question to be relevant to multiple courses. We want to compare courses to one another. Personally, I think we overvalue this type of comparison, even when we might be comparing apples to oranges. For example, do we really want to compare scores on courses that teach such disparate topics as sexual harassment, word processing, leadership, and advanced statistical techniques? Still, there are times when such comparisons make sense.

The downside of generalizability is that we lose precision. Learners are less able to calibrate their answers. Analyzing the results becomes less meaningful. Also, learners see the learner-survey process as less valuable when questions are generic, so they give less energy and thought to answering the questions, and our data become less valuable and more biased.

Here is a question I developed for my own workshop (on how to create better smile sheets, by the way SMILE):

How READY are you TO WRITE QUESTIONS for a Performance-Focused Smile Sheet?






D. I CAN WRITE MY OWN QUESTIONS, but I’d LIKE to get SOME FEEDBACK before using them.





Note several things about this question. First to restate. It is infinitely more tailored than a generic question could be. It encourages more thoughtful responding and creates more meaningful feedback.

Second, you might wonder why all the CAPS! I advocate CAPS because (1) CAPS have been shown in research to slow reading speed. Too often, our learners burn through our smile-sheet questions. Anything we can do to make them attend more fully is worth trying. Also, (2) respondents often read the full question and then skim back over it when determining how to respond. I want them to have an easy way to parse the options. Full disclosure. To my knowledge, all CAPS has not been studied yet for smile sheets. At this point, my advocacy for all CAPS is based on my intuition about how people process smile-sheet questions. If you’d like to work with me to test this in a scientifically rigorous fashion, please contact me.

Third, notice the opportunity for learners to write clarifying comments. Open-ended questions, though not easily quantifiable, can be the most important questions on smile sheets. They can provide intimate granularity — a real sense of the respondents’ perceptions. In these questions, we’re using a hybrid format, a forced choice question followed by an open-ended opportunity for clarification. This not only enables the benefits of open-ended responding, but it also enables us to get clarifying meaning. In addition, in some way it provides a reality-check on our question design. If we notice folks responding in ways that aren’t afforded in the answer choices given, we can improve our question for later versions.


Question Example — Simplifying The Wording

In writing smile-sheet questions, there’s another tradeoff to consider. More words add more precision, but fewer words add readability and motivation to engage the question fully. In the book, I talk about what I once called, “The World’s Best Smile Sheet Question.” I liked it partly because the answer choices were more precise than a Likert-like scale. It did have one drawback; it used a lot of words. For some audiences this might be fine, but for others it might be entirely inappropriate.

Recently, in working with a company to improve their smile sheet, a first draft included the so-called World’s Best Smile Sheet Question. But they were thinking of piloting the new smile sheet for a course to teach basic electronics to facilities professionals. Given the topic and audience, I recommended a simpler version:

How able will you be to put what you’ve learned into practice on the job?  Choose one.

A. I am NOT AT ALL ready to use the skills taught.
B. I need MORE GUIDANCE to be GOOD at using these skills
C. I need MORE EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.

This version nicely balances precision with word count.


Question Example — Dealing with the Sticky Problem of “Motivation”

In the book, I advocate a fairly straightforward question asking learners about their motivation to apply what they’ve learned. In many organizations — in many organizational cultures — this will work fine. However in others, our trainers may be put off by this. They’ll say, “Hey, I can’t control people’s motivations.” They’re right, of course. They can’t control learners’ motivations, but they can influence them. Still, it’s critical to realize that motivation is a multidimensional concept. When we speak of motivation, we could be talking simply about a tendency to take action. We could be talking about how inspired learners are, or how much they believe in the value of the concepts, or how much self-efficacy they might have. It’s okay to ask about motivation in general, but you might generate clearer data if you ask about one of the sub-factors that comprise motivation.

Here is a question I developed recently for my Smile-Sheet Workshop:

How motivated are you to IMPLEMENT PERFORMANCE-FOCUSED SMILE SHEETS in your organization?



B. I will confer with my colleagues to SEE IF THERE IS INTEREST.

C. I WILL ADVOCATE FOR performance-focused smile sheet questions.

D. I WILL VIGOROUSLY CHAMPION performance-focused smile sheet questions.



In this question, I’m focusing on people’s predilection to act. Here I’ve circumnavigated any issues in asking learners to divulge their internal motivational state, and instead I’ve focused the question on the likelihood that they will utilize their newly-learned knowledge in developing, deploying, and championing performance-focused smile sheets.


Final Word

It’s been humbling to work on smile sheet improvements over many years. My earlier mistakes are still visible in the digital files on my hard drive. I take solace in making incremental improvements — and in knowing that the old way of creating smile-sheet questions is simply no good at all, as it provides us with perversely-irrelevant information.

As an industry — and the learning industry is critically important to the world — we really need to work on our learning evaluations. Smile sheets are just one tool in this. Unfortunately, poorly constructed smile sheets have become our go-to tool, and they have led us astray for decades.

I hope you find value in my book (SmileSheets.com). More importantly, I hope you’ll participate along with some of the world’s best-run companies and organizations in developing improved smile sheet questions. Again, please email me with your questions, your question improvements, and alternatively, with examples of poorly-crafted questions as well.

Neuroscience and Learning

The Debunker Club, formed to fight myths and misconceptions in the learning field, is currently seeking public comment on the possibility that so-called neuroscience-based recommendations for learning and education are premature, untenable, or invalid.


Click here to comment or review the public comments made so far…


Click here to join The Debunker Club…


Send me your tired, your salaried, your harried Chief Learning Officers (and other Talent Development Executives) so that I can collect data on the state of our industry — focusing on a practical science-of-learning perspective.


Areas Covered in this Research Effort:

  • what is your organization asking of the learning function?
  • how are your results measured? what matters to management?
  • what are the key struggles for your learning units?
  • research utilization — how utilized? who utilizes?
  • current strengths of learning professionals.
  • development needs of learning professionals
  • sources of knowledge for these learning professionals.
  • how does your organization deal with learning myths?
  • how do you separate good information from bad?
  • what vendors provide valuable research/data?
  • what vendors provide training that is science-based?
  • what do your learning units need to be most effective?

Organizations who contribute will be entitled to the full report. An executive summary will be available for free to others.

Send me names and contact information to info@work-learning.com.


Now that my book publishing responsibilities are out of the way, I'm ready to begin a research effort designed to find out more ways to improve the effectiveness of our smile sheets.

If you're organization would like to participate in this effort — and be among the first to see the results of the research — check out the full announcement on my book's website: http://smilesheets.com/2016/05/18/3705/.

In my 30th year in the workplace learning field, I've seen in my own work — and in that of my clients — that I cannot control everything that happens. I can make recommendations to clients, and I can get better as a persuader, but I can't force change.

I've also seen my clients and instructional designers everywhere struggle to make small improvements against floodwaters of organizational lethargy, misunderstandings about learning, conflicting priorities, and more. I've seen the same thing with CLO's, training directors, and other learning executives as well.

I've come to know many dogged professionals who keep at it year after year against Sisyphean challenges, making improvements — even small improvements — wherever and whenever they can. I admire them greatly.

To be effective in our jobs as learning professionals, we not only have to know the research and know our craft, we also have to develop an ability to marshal our resolve, maintain our perseverance, and retain at least some semblance of equanimity.

Perhaps by sharing our laments and our aspirations, we can get a little closer to these ideals.

I'm not religious, but it seems that a "serenity-prayer" approach for instructional designers might prove valuable.


Give me grace to accept with serenity the things that cannot be changed, courage to change the things which should be changed, and the wisdom to distinguish the one from the other.


Things that cannot be changed (in the short term):


  • Sometimes, I will need to crank out learning interventions that fail my standards for effectiveness.
  • Sometimes, I will need to fail to adequately evaluate my learning interventions.
  • Sometimes, I will need to do an inadequate job in doing needs assessment.
  • Sometimes, I will need to….

            List your "things that cannot be changed (in the short term)" below in the comments.


Things that require the courage to push for change:


  • Learning interventions that produce only awareness — should be improved to help learners be competent enough to perform a skill.
  • Learning interventions that don't utilize spaced repetitions — should be improved to space repetitions over time.
  • Learning interventions that don't provide substantial realistic practice — should be improved to provide skill practice that mimics the most salient aspects of the targeted (on-the-job) performance environment.
  • Smile sheets that utilize Likert-like scales or numeric scales — should be improved to utilize distinctive answer choices.
  • Et cetera…


Wisdom to know the difference:


  • By making a distinction between the short term and the long term, I can meet my task goals while pushing for lasting improvements.
  • By searching out the most dedicated learning professionals, by gathering together in a mutual effort to make the system work for better learning and performance improvement, I can maintain my motivation and energy to keep at it.
  • By meeting my organizational obligations with a steady excellence, I can develop the credibility and power to enable me to be persuasive and potent in pushing for improvements.
  • Et cetera…


These are just some examples… My hope is that you will add your thoughts and wisdom in the comments.

To Magazine Editors, Bloggers, Tweeters, and other Opinion Leaders in the learning field:

I’m wondering if you’d help publicize The Debunkers Club (www.Debunker.Club)

Launched just two months ago, we’ve already enrolled members from every continent except for Antarctica and South America.

The Debunker Club is dedicated to fighting myths and misconceptions in the learning field. In the past, individuals and organizations have taken on the responsibility of improving the information in the field, but still, bad information—sometimes dangerous information—continues to be conveyed.

The Debunker Club is taking a more proactive approach, hoping that we can make more of a difference. To check out the types of motivations that inspire members of The Debunker Club, we have a webpage that highlights their testimony.

I’d especially like you to publicize our upcoming April Fool’s Week, a week dedicated to debunking the myths in our field, one at a time. You can read about it here: http://www.debunker.club/april-fools-week.html

Thank you for considering this request!

We need your help! We're all in this together!


= Will Thalheimer

Organizer of The Debunker Club

This list will be updated periodically.

WILL’S NOTE:  Previously, I had given up trying to keep this list current because it was nearly impossible to research and catalog the “gazillions” of new providers entering this category. I’m going to make another effort to keep this list, but I need your help through the contact form below. THANKS!


To link to their websites, move your mouse above the name.

Subscription-Learning Authoring Tools — General Purpose

  • Ed (subscription and mobile-learning authoring tool)
  • KnowledgeGuru (subscription and game-based learning authoring tool)
  • Moblrn (subscription learning authoring tool)
  • Cameo (subscription-learning authoring tool, scenario-based questions, email, tracks learning)
  • Q MINDshare (subscription learning authoring tool)
  • Mindsetter (subscription-learning authoring tool)
  • Memrise (subscription-learning authoring tool, crowdsourcing, mnemonics, gamification)
  • Mindmarker (subscription learning authoring tool)
  • SimWriter (scenario-based simulation authoring tool)
  • Zunos (learning and knowledge-management authoring tool)
  • DailyBitsOf (subscription-learning authoring tool and course catalog)
  • Cerego (subscription-learning authoring tool and course catalog)
  • Trivie (subscription learning authoring tool focused on retrieval practice)
  • Flip Training (subscription-learning authoring tool)
  • Otto Learn (subscription-learning authoring tool)

Subscription-Learning Authoring Tools — Drill and Practice

Subscription-Learning Developers

  • Axonify (custom developer, utilizes spacing effect)
  • NexLearn (custom developer of simulations)
  • Wranx (subscription-learning provider)
  • GamEffective (gamified and performance-focused engagements with subscription learning).

Subscription-Learning Single-Focus Applications

Subscription-Learning Repositories of Courses

  • MyTools2Learn (subscription learning course repository)
  • Cerego (flashcard authoring tool and course catalog)
  • DailyBitsOf (subscription-learning authoring tool and course catalog)
  • Memrise (subscription-learning authoring tool, crowdsourcing, mnemonics, gamification)
  • Ed (subscription and mobile-learning authoring tool)
  • EdX (MOOC available as subscription and mobile learning)
  • Coursera (MOOC available as subscription and mobile learning)
  • Class Central (a repository and rating compilation of university courses)
  • https://gohighbrow.com/

Subscription-Learning After-Training Follow-up Tools

Microlearning — Without Threaded Content

Social Media is hot, but it is not clear how well we are measuring social media.

A couple of years ago I wrote an article for the eLearning Guild about measuring social media. But it's not clear that we've got this nailed yet.

With this worry in mind, I've created a research survey to begin a process to see how best social-media (of the kind we might use to bolster workplace learning-and-performance) can be measured.

Here's the survey link. Please take the survey yourself. You don't have to be an expert to take it.

Here's my thinking so far on this. Please send wisdom if I've missed something.

  1. We can think about measuring social media the same way we measure any learning intervention.
  2. We can also create a list of all the proposed benefits for social media, and the proposed costs, and all the proposed harms, and we can see how people are measuring these now. The survey will help us with this second approach.

Note: Survey results will be made available for free. If you take the survey, you'll get early releases of the survey results and recommendations.

Also, this is not the kind of survey that needs good representative sampling, so feel free to share this far and wide.

Here is the direct link to the survey:   http://tinyurl.com/4tlslol

Here is the direct link to this blog post:   http://tinyurl.com/465ekpa

In 2006, I reviewed the research on the spacing effect and published a research-to-practice report, Spacing Learning Over Time: What the Research Says…

Since then I have been buoyed by the enthusiastic response to that report and by the changes that it  engendered. More training and e-learning has been built using spacing and more and more learning software has been built that incorporates the spacing effect as an inherent part of its design. If I died today, I would at least know that I'd made a small difference in our field.

Examples Wanted

I am working on an updated version of the report to include the latest research and new examples.

If you know of any examples of the use of spacing effect, please let me know. Send me demo links or disks so that I can see for myself how the spacing effect has been used. Or, just write me an email.

Testimonials Wanted

Also, if you read the original version and want to write a short testimonial about how it changed the way you build learning, that would be awesome. Just write me an email.

One Product Example: A Cameo Appearance

Just to get your juices flowing, check out this YouTube Video produced by a company who built a product with the spacing effect in mind, Yukon Learning. Yukon has built a very nice tool to support learning using the spacing effect. The product name is Cameo and the link below will take you to the Cameo website.

You can check out Cameo at this website.

Are you an independent consultant or contractor in the workplace learning-and-performance field?

Worried about the economy or energized by it?

There is a lot of anecdotal worry that the economy is hitting the training-and-development field hard, perhaps especially so for independent consultants and contractors.

I decided we ought to gather our own data to see what's really going on.

If you're an independent in our field, take my survey at the link below.

In addition to getting a snapshot of the current situation, the survey will help us look at how 2009 is shaping up, and share strategies we independents are employing to survive/thrive.

I'm also asking whether independents might be interested in forming a group for mutual benefit.



Please let
people know about this so we get as wide an audience as possible.
Consider notifying people at both the center and periphery of your
social network so that we get a wider cross section of respondents.
Please also send notifications spread out over time so that we widen
our net as well. I will keep the survey open for two weeks or so (or
longer if responses are still rushing in).

Here is the link to send to others:


1. Send to your independent colleagues.
2. Post in groups you belong to.
3. Send to your social-network friends.
4. Post on your blog, twitter, etc.
5. Send to your email newsletter.