performance-focused Archives » Work-Learning Research

20 Benefits of Learner Surveys

In 2016 I published a book on how to radically transform learner surveys into something useful. The book won an award from ISPI and helped thousands of companies update their smile sheets. Now, I’m updating the book with the knowledge I’ve gained in consulting with companies in the learning-evaluation efforts. The second edition will be titled: Performance-Focused Learner Surveys: A Radical Rethinking of a Dangerous Art Form (Second Edition).

In the first edition, I listed nine benefits of learner surveys, but I had only touched the surface. In the coming book, I will offer 20 benefits. Here’s the current list:

Supporting Learning Design Effectiveness

Red-flagging training programs that are not sufficiently effective.
Gathering ideas for ongoing updates and revisions of learning programs.
Judging the strengths, weaknesses, and viability of program updates and pilots.
Providing learning architects and trainers with feedback to aid their development.
Judging the competence of learning architects and trainers.
Judging the contributions to learning made by people outside of the learning team.
Assessing the contributions of learning supports and organizational practices.

Supporting Learners in Learning and Application

Helping learners reflect on and reinforce what they learned.
Helping learners determine what (if anything) they plan to do with their learning.
Nudging learners to greater learning and application efforts.

Nudging Action Through Stealth Messaging

Guiding learning architects to create more effective learning by sharing survey questions before learning designs are finalized and sharing survey results after data is gathered.
Guiding trainers to utilize more effective learning methods by sharing survey questions before learning designs are finalized and sharing survey results after data is gathered.
Guiding organizational stakeholders to support learning efforts more effectively by sharing survey questions and survey results.
Guiding organizational decision makers to better appreciate the complexity and depth of learning and development—helping the learning team to gain credibility and autonomy.

Supporting Relationships with Learners and Other Key Stakeholders

Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the learning intervention and/or the instructors.
Upholding the spirit of common courtesy by giving learners a chance for feedback.
Enabling learner frustrations to be vented—to limit damage from negative back-channel communications.

Maintaining Organizational Credibility

Engaging in visibly credible efforts to assess learning effectiveness.
Engaging in visibly credible efforts to utilize data to improve effectiveness.
Reporting out data to demonstrate learning effectiveness.

If you want to learn when the new edition is available, sign up for my list. https://www.worklearning.com/sign-up/.

The second edition will include new and improved question wording, additional questions, additional chapters, etc.

Matt Richter and I, in our Truth-in-Learning Podcast, will be discussing learner surveys in our next episode. Matt doesn’t believe in smile sheets and I’m going to convince him of the amazing power of well-crafted learner surveys. This blog post is my first shot across the bow. To join us, subscribe to our podcast in your podcast app.

2 September 2020

Do NOT buy Performance-Focused Smile Sheet Book

Books, Work-Learning Research

Today is May 15, 2020.

This week, I asked my publisher to take all 254 remaining copies of Performance-Focused Smile Sheets and throw them in the recycling bin!

Yes! The book was one of the most important books written in the learning-and-development field over the last decade.

But I threw it out!

Why?

Because I’m writing a new, completely revised edition based on my experience developing learner surveys for organizations around the world. My learner-survey approach has been used at The Centers for Disease Control and Prevention, the World Bank, at Bloomberg, at Roche (Genentech), at Oxfam. It has revolutionized the moribund smile sheet!

But time doesn’t stand still, and I’ve gained a ton of insights building learner surveys over the past five years, hearing from clients and others, pilot testing learner surveys, and creating smile sheets for my own workshops.

The new book will have an all new set of recommended questions. It will have information on open-ended questions. It will describe how learner surveys fit into the new LTEM framework (the learning-evaluation model that is replacing the Kirkpatrick-Katzell Four-Level Model around the world). The new book will also share the thinking behind the new learning-evaluation approach, LEADS (Learning Evaluation As Decision Support), which I’m developing.

When will the new book be out? Within three months!

The book may still be available in some pipelines, but I’m working to get it off the shelves. My advice is don’t buy it!

Wait for the second edition, which I’ve re-titled, Performance-Focused Learner Surveys.

Stay tuned!

And sign up for my newsletter at https://www.worklearning.com/sign-up/ to be the first to know when the book is available.

Thanks for your patience,

= Will Thalheimer

15 May 2020

Guest Post by Annette Wisniewski: Thrilling a Client with Better Smile-Sheet Questions

Guest Post, Learning Measurement

This is a guest post by Annette Wisniewski, Learning Strategist at Judge Learning Solutions. In this post she shares an experience building a better smile sheet for a client.

She also does a nice job showing how to improve questions by getting rid of Likert-like scales and replacing them with more concrete answer choices.

______________________________

Using a “Performance-focused Smile Sheets” Approach for Evaluating a Training Program

Recently, one of our clients had experienced an alarming drop in customer confidence, so they hired us, Judge Learning Solutions, to evaluate the effectiveness of their customer support training program. I was the learning strategist assigned to the project. Since training never works in isolation, I convinced the client to let me evaluate both the training program and the work environment.

I wanted to create the best survey possible to gauge the effectiveness of the training program as well as evaluate the learners’ work environment, including relevant tools, processes, feedback, support, and incentives. I also wanted to create a report that included actionable recommendations on how to improve both the training program and workforce performance.

I had recently finished reading Will’s book, Performance-focused Smile Sheets, so I knew that traditional Likert-based questions are problematic. They are very subjective, don’t give clear distinction between answer choices, and limit respondents to one, sometimes insufficient, option.

For example, most smile sheets ask learners to evaluate their instructor. A traditional smile sheet question might ask learners to rank the instructor using a Likert-scale.

How would you rate your course instructor?

Very ineffective
Somewhat ineffective
Somewhat effective
Very effective

But the question leaves too much open to interpretation. What does “ineffective” mean? What does “effective” mean? One learner might have completely different criteria for an “effective” instructor than another. What is the difference between “somewhat ineffective” and “somewhat effective”? Could it be the snacks the instructor brought in mid-afternoon? It’s hard to tell. Also, how can the instructor use this feedback to improve next time? There’s just not enough information in this question to make it very useful.

For my evaluation project, I wrote the survey question using Will’s guidelines to provide distinct, meaningful options, and then allowed learners to select as many responses as they wanted.

What statements are true about your course instructor? Select all that apply.

Was OFTEN UNCLEAR or DISORGANIZED.
Was OFTEN SOCIALLY AWKWARD OR INAPPROPRIATE.
Exhibited UNACCEPTABLE LACK OF KNOWLEDGE.
Exhibited LACK OF REAL-WORLD EXPERIENCE.
Generally PERFORMED COMPETENTLY AS A TRAINER.
Showed DEEP SUBJECT-MATTER KNOWLEDGE.
Demonstrated HIGH LEVELS OF REAL-WORLD EXPERIENCE.
MOTIVATED ME to ENGAGE DEEPLY IN LEARNING the concepts.
Is a PERSON I CAME TO TRUST.

It’s still just one question, but in this case, the learner was able to provide more useful feedback to both the instructor and to the course sponsors. As Will recommended, I added proposed standards, and then tracked percentages of each response to include in my report:

I used this same approach when asking learners about the course learning objectives.

Instead of asking a question using a typical Likert scale:

After taking the course, I am now able to navigate the system.

Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree

I created a more robust question that provided better information about how well the learner was able to navigate the system and what the learner felt he/she needed to become more proficient. I formatted the question as a matrix, so I could ask about all of the learning objectives at once. The learner perceived this to be one question, but I gleaned nine questions-worth of data out of it. Here’s a redacted excerpt of that question as it appeared in my report, shortened to the first four learning objectives.

The questions took a little more time to write, but the same amount of time for respondents to answer. At first, the client was hesitant to use this new approach to survey questions, but it didn’t take them long to see how I would be able to gather much more valuable data.

The descriptive answer choices of the survey, combined with interviews and extant data reviews, allowed me to provide my client with a very thorough evaluation report. The report not only included a clear picture of the current training program, but also provided detailed and prioritized recommendations on how to improve both the training program and the work environment.

The client was thrilled. I had given them not only actionable recommendations but also the evidence they needed to procure funding to make the improvements. When my colleague checked back with them several months later, they had already implemented several of my recommendations and were in the process of implementing more.

I was amazed at how easy it was to improve the quality of the data I gathered, and it certainly impressed my client. I will never write evaluation questions again any other way.

If you plan on conducting a survey, try using Will’s approach to writing performance-focused questions. Whether you are evaluating a training program or looking for insights on improving workforce performance, you will be happy you did!

6 May 2019

Gold-Certification Workshop Now Open For Registration!

Learning Measurement, Work-Learning Academy, Work-Learning Research

I’m thrilled to announce that my Gold-Certification Workshop on Performance-Focused Smile Sheets is now open for registration, with access available in about a week on Tuesday May 14 (2019).

This certification workshop is the culmination of years of work and practice. First there was my work with clients on evaluation. Then there was the book. Then I gained extensive experience building and piloting smile sheets with a variety of organizations. I taught classroom and webinar workshops. I spoke at conferences and gave keynotes. And of course, I developed and launched LTEM (The Learning-Transfer Evaluation Model), which is revolutionizing the practice of workplace learning—and providing the first serious alternative to the Kirkpatrick-Katzell Four-Level Model.

Over the last year, I’ve been building an online, asynchronous workshop that was rigorous, comprehensive, and challenging enough to offer a certification. It’s now ready to go!

I’d love if you would enroll and join me and others in learning!

You can learn more about this Gold-Certification Workshop by clicking here.

6 May 2019

Brett Christensen Uses the Performance-Focused Smile Sheet Methodology

Learning Measurement

This week, Brett Christensen published an article on how he’s used a Performance-Focused Smile Sheet to support him in teaching one of ISPI’s flagship workshops.

What I found particularly striking is how Brett used the smile-sheet results to make sense of learning effectiveness. His goal was to help his learners be able to take what they’ve learned and use it back on the job.

One smile-sheet question he used pointed to results that suggested that learners felt they had gained awareness of concepts, but they might not be fully able to put what they learned into practice. This raised a red flag, so Brett examined results from another question on the amount of practice received in the workshop. The learners told him that practice was only a little more than 50% of the workshop, and Brett used this information to consider changes for adding more practice.

He also used a question to get a sense of whether the spacing effect was utilized to support long-term remembering–a key research-based learning approach. He got good news there–so that even in a one-day workshop–many learners felt repetitions were delivered after a delay of an hour or more. Good instructional design!

For a century or more, our learner-feedback questions have focused on satisfaction, course reputation, and other factors that are NOT directly related to learning effectiveness. Now we have a new methodology, first described in the award-winning book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. We ought to use this to get feedback about what we can do better.

Brett offers a wonderful case study from his work teaching a course offered through ISPI (Developed by Dr. Roger Chevalier). We are no longer hogtied with evaluations that provide us with bogus information. We can look for ways to get better feedback, improve our learning interventions, and get better results.

To read Brett’s full article, click here…

13 December 2017

Getting Better Responses on Your Smile Sheets.

Learning Measurement, Thoughts and Ideas

One of the most common questions I get when I speak about the Performance-Focused Smile-Sheet approach (see the book’s website at SmileSheets.com) is “What can be done to get higher response rates from my smile sheets?”

Of course, people also refer to smile sheets as evals, level 1’s, happy sheets, hot or warm evaluation, response forms, reaction forms, etc. They also refer to both paper-and-pencil forms and online surveys. Indeed, as smile sheets go online, more and more people are finding that online surveys get a much lower response rate than in-classroom paper surveys.

Before I give you my list for how to get a higher response rate, let me blow this up a bit. The thing is, while we want high response rates, there’s something much more important than response rates. We also want response relevance and precision. We want the questions to relate to learning effectiveness, not just learning reputation and learner satisfaction. We also want the learners to be able to answer the questions knowledgeably and give our questions their full attention.

If we have bad questions — one’s that use Likert-like or numeric scales for example — it won’t matter that we have high response rates. In this post, I’m NOT going to focus on how to write better questions. Instead, I’m just tackling how we can motivate our learners to give our questions more of their full attention, thus increasing the precision of their responding while also increasing our response rates as well.

How to get Better Responses and Higher Response Rates

Ask with enthusiasm, while also explaining the benefits.
Have a trusted person make the request (often an instructor who our learners have bonded with).
Mention the coming smile sheet early in the learning (and more than once) so that learners know it is an integral part of the learning, not just an add-on.
While mentioning the smile sheet, let folks know what you’ve learned from previous smile sheets and what you’ve changed based on the feedback.
Tell learners what you’ll do with the data, and how you’ll let them know the results of their feedback.
Highlight the benefits to the instructor, to the instructional designers, and to the organization. Those who ask can mention how they’ve benefited in the past from smile sheet results.
Acknowledge the effort that they — your learners — will be making, maybe even commiserating with them that you know how hard it can be to give their full attention when it’s the end of the day or when they are back to work.
Put the time devoted to the survey in perspective, for example, “We spent 7 hours today in learning, that’s 420 minutes, and now we’re asking you for 10 more minutes.”
Ensure your learners that the data will be confidential, that the data is aggregated so that an individual’s responses are never shared.
Let your learners know the percentage of people like them who typically complete the survey (caveat: if it’s relatively high).
Use more distinctive answer choices. Avoid Likert-like answer choices and numerical scales — because learners instinctively know they aren’t that useful.
Ask more meaningful questions. Use questions that learners can answer with confidence. Ask questions that focus on meaningful information. Avoid obviously biased questions — as these may alienate your learners.

How to get Better Responses and Higher Response Rates on DELAYED SMILE SHEETS

Sometimes, we’ll want to survey our learners well after a learning event, for example three to five weeks later. Delayed smile sheets are perfectly positioned to find out more about how the learning is relevant to the actual work or to our learners’ post-learning application efforts. Unfortunately, prompting action — that is getting learners to engage our delayed smile sheets — can be particularly difficult when asking for this favor well after learning. Still, there are some things we can do — in addition to the list above — that can make a difference.

Tell learners what you learned from the end-of-learning smile sheet they previously completed.
Ask the instructor who bonded with them to send the request (instead of an unknown person from the learning unit).
Send multiple requests, preferably using a mechanism that only sends these requests to those who still need to complete the survey.
Have the course officially end sometime AFTER the delayed smile sheet is completed, even if that is largely just a perception. Note that multiple-event learning experiences lend themselves to this approach, whereas single-event learning experiences do not.
Share with your learners a small portion of the preliminary data from the delayed smile sheet. “Already, 46% of your fellow learners have completed the survey, with some intriguing tentative results. Indeed, it looks like the most relevant topic as rated by your fellow learners is… and the least relevant is…”
Share with them the names or job titles of some of the people who have completed the survey already.
Share with them the percentage of folks from his/her unit who have responded already or share a comparison across units.

What about INCENTIVES?

When I ask audiences for their ideas for improving responses and increasing response rates, they often mention some sort of incentive, usually based on some sort of lottery or raffle. “If you complete the survey, your name will be submitted to have chance to win the latest tech gadget, a book, time off, lunch with an executive, etc.”

I’m a skeptic. I’m open to being wrong, but I’m still skeptical about the cost/benefit calculation. Certainly for some audiences an incentive will increase rates of completion. Also, for some audiences, the harms that come with incentives may be worth it.

What harms you might ask? When we provide an external incentive, we might be sending a message to some learners that we know the task has no redeeming value or is tedious or difficult. People who see their own motivation as caused by the external incentive are potentially less likely to seriously engage our questions, producing bad data. We’re also not just having an effect on the current smile sheet. When we incentivize people today, they may be less willing next time to engage in answering our questions. They may also be pushed into believing that smile sheets are difficult, worthless, or worse.

Ideally, we’d like our learners to want to provide us with data, to see answering our questions as a worthy and helpful exercise, one that is valuable to them, to us, and to our organization. Incentives push against this vision.

22 November 2017

Book Publication Day!!

Learning Industry, Learning Measurement, Work-Learning Research

Wow!!

I almost can't believe it. Finally, after 17 years of research and writing, I'm finally a published author.

Today is the day!

It's kind of funny really.

When I began this journey back in 1997 I had a well-paying job running a leadership-development product line, building multimedia simulations, and managing and working with a bunch of great folks.

As I looked around the training-and-development field — that's what we called it back then — I saw that we jumped from one fad to another and held on sanctimoniously to learning methods that didn't work that well. I concluded that what was needed was someone to play a role in bridging the gap between the research side and the practice side.

I had a very naive idea about how I might help. I thought the field needed a book that would specify the fundamental learning factors that should be baked into every learning design. I thought I could write such a book in two or three years, that I'd get it published, that consulting gigs would roll in, that I'd make good money, that I'd make a difference.

Hah! The blind optimism of youth and entrepreneurship!

I've now written over 700 pages on THAT book…without an end in sight.

How The Smile-Sheet Book Got its Start

Back in 2007, as I was mucking around in the learning research, I began to see biases in how we were measuring learning. I noticed, for instance, that we always measured at the top of the learning curve, before the forgetting curve had even begun. We measured with trivial multiple-choice questions on definitions and terminology — when these clearly had very little relevance for on-the-job performance. I wrote a research-to-practice report on these learning measurement biases and suddenly I was getting invited to give keynotes…

In my BIG book, I wrote hundreds of paragraphs on learning measurement. I talked about our learning-measurement blind spots to clients, at conferences, and on my blog.

Where feedback is the lifeblood of improvement, we as learning professionals were getting very little good feedback. We were practicing in the dark.

I'd also come to ruminate on the meta-analytic research findings that showed that traditional smile sheets were virtually uncorrelated with learning results. If smile sheets were feeding us bad information, maybe we should just stop using them.

It was about three or four years ago that I saw a big client get terrible advice about their smile sheets from a well-known learning-measurement vendor. And, of course, because the vendor had an industry-wide reputation, the client almost couldn't help buying into their poor smile-sheet designs.

I concluded that smile-sheets were NOT going away. They were too entrenched and there were some good reasons to use them.

I also concluded that smile sheets could be designed to be more effective, more aligned with the research on learning, and designed to better support learners in making smile-sheet decisions.

I decided to write a shorter book than the aforementioned BIG book. That was about 2.5 years ago.

I wrote a draft of the book and I knew I had something. I got feedback from learning-measurement luminaries like Rob Brinkerhoff, Jack Phillips, and Bill Coscarelli. I got feedback from learning gurus Julie Dirksen, Clark Quinn, and Adam Neaman. I made major improvement based on the feedback from these wonderful folks. The book then went through several rounds of top-tier editing, making it a much better read.

As the publication process unfolded, I realized that I didn't have enough money on hand to fund the printing of the book. Kickstarter and 227 people raised their hands to help, reserving over 300 books in return for their generous Kickstarter contributions. I will be forever indepted to them.

Others reached out to help as well, from people on my newsletter list, to my beloved clients, to folks in trade organizations and publications, to people I've met through the years, to people I haven't met, to followers on Twitter, to the industry luminaries who agreed to write testimonials after getting advanced drafts of the book, to family members, to friends.

Today, all the hard work, all the research, all the client work, all the love and support comes together for me in gratitude.

Thank you!

= Will Thalheimer

P.S. To learn more about the book, or buy it: SmileSheets.com

2 February 2016