I’ve significantly improved my Scenario-Question Generator Template, now in Version 5.

Here it is: https://www.worklearning.com/wp-content/uploads/2026/06/Scenario-Questions-Generator-v5.docx

 

 

This is the fifth generation of the template. It is designed to help you think through how to craft a scenario question that works.

The template specifically:

  • Helps generate questions that distinguish between those who can make work decisions and those who can’t.
  • Encourages a focus on work issues and work settings.
  • Pushes you to be specific about your learning objectives.
  • Nudges you to highlight common mistakes that might be overlooked.
  • Aligns with the SEDA model’s insistence that learners ought to be able to EVALUATE realistic situations.
  • Reminds scenario-writers to avoid unnecessary hints that short-circuit cognition.
  • Prompts scenario designs that require deeper cognitive processing than most multiple-choice approaches.
  • Provides a detailed example scenario question.
  • Provides a detailed worked-example of a scenario question.

Stay Tuned for my AI-Scenario-Question-Development Tool, now in production. 

Also, I’m working with Shilpa Subramaniam to build a Scenario-Question Writing Clinic for November 2026. 

 

To keep up with these important evaluation innovations, sign up for my personal newsletter. https://www.worklearning.com/sign-up/

GROWLE — Global Research on Workplace Learning Evaluation

GROWLE BACKGROUND AND PURPOSE:

GROWLE 2026 is the most comprehensive global research study ever conducted on learning evaluation. L&D professionals from around the world were recruited in January and February of 2026 through dozens of channels, ensuring a broad representation of perspectives.

Results will be shared free to the L&D community in late April of 2026 via webinars and a free report.

Research was conducted by Work-Learning Research with support from dozens of research partners who generously reached out to their L&D communities to encourage people to share their perspectives.

Learning evaluation is one of the most important practices performed by L&D professionals. It enables us to gather data to maximize our effectiveness and garner credibility with our organizational stakeholders, customers, and the global market.

GROWLE is intended to “Grow Learning Evaluation” by gathering critical data on a periodic basis (every three years).

The report will be released via webinar on the 29th of April 2026 (April 30th in some time zones). The webinar is being repeated at different times to ensure L&D professionals can attend at a time-zone-friendly time.

WEBINAR DATES AND TIMES:

Two Time-Zone-Friendly Webinar Options

SESSION 1 — Wednesday, April 29 | 10:00am to 11:00am New York Time

Registration Required:

 

SESSION 2 — Wednesday, April 29 | 9:00pm to 10:00pm New York Time

(BUT NOTE: Thursday, April 30 for APAC/Oceania)

Registration Required:

QUESTIONS THAT GROWLE WILL ANSWER:

Times of Webinars Worldwide

SESSION 1 — Wednesday, April 29 | 10:00am to 11:00am New York Time

Americas

Los Angeles / Vancouver / Seattle — 7:00am to 8:00am

Denver / Calgary — 8:00am to 9:00am

Chicago / Mexico City / Dallas / Bogata — 9:00am to 10:00am

New York / Toronto / Miami — 10:00am to 11:00am

Rio de Janeiro / São Paulo — 11:00am to 12:00pm

 

Europe / UK

London / Dublin / Lisbon — 3:00pm to 4:00pm

Paris / Berlin / Rome / Amsterdam — 4:00pm to 5:00pm

Helsinki / Kyiv / Istanbul — 5:00pm to 6:00pm

 

Africa

Accra / Dakar — 2:00pm to 3:00pm

Lagos / Abuja — 3:00pm to 4:00pm

Johannesburg / Cape Town — 4:00pm to 5:00pm

Nairobi / Addis Ababa — 5:00pm to 6:00pm

 

Middle East / South Asia

Tel Aviv, Israel — 5:00pm to 6:00pm

India — 7:30pm to 8:30pm

SESSION 2 — Wednesday, April 29 | 9:00pm to 10:00pm New York Time

(BUT NOTE: Thursday, April 30 for APAC/Oceania)

 

APAC / Oceania / South Asia (Thursday, April 30)

India — 6:30am to 7:30am

Vietnam — 8:00am to 9:00am

China / Singapore / Malaysia / Philippines / Taiwan / Perth — 9:00am to 10:00am

Japan / Korea — 10:00am to 11:00am

Sydney / Melbourne / Brisbane — 11:00am to 12:00pm

New Zealand — 1:00pm to 2:00pm

 

Americas (Evening Wednesday, April 29)

Los Angeles / Vancouver / Seattle — 6:00pm to 7:00pm

Denver / Calgary — 7:00pm to 8:00pm

Chicago / Mexico City / Dallas / Bogata — 8:00pm to 9:00pm

New York / Toronto / Miami — 9:00pm to 10:00pm

Rio de Janeiro / São Paulo — 10:00pm to 11:00pm

One of the most counterproductive ideas in the L&D field is also one of the most common. Even today. That idea is this: Measuring the impact of our programs should be the overriding priority in learning evaluation. This idea is harmful enough that it deserves a firm rebuttal.

In the attached PDF I make the case.

Ten Reasons NOT to Measure Impact

 

Scenario Questions! One of the most important tools in any learning architect’s toolkit. Powerful support for learning and remembering. Aligned with the learning science findings on retrieval practice and context alignment. Can be augmented with repetitions, feedback, spacing.

But also for learning evaluation! Scenario questions offer us a great way to measure people’s decision-making competence, for example, as suggested for LTEM Tier-5 Decision Making.

I used to hire and train people to write scenario questions, which we used in simulations to teach leadership. I’ve written thousands of scenario questions myself, on all sorts of topics. I teach a full-day workshop on writing scenario questions. I teach about scenario questions in my LTEM Boot Camp–an intensive workshop on LTEM and Learning Evaluation. Click here to learn more about the LTEM Boot Camp.

And now I’m updating my Scenario-Question Generator Template. Now in Version 4.

You can access the template by clicking here. 

The LTEM Boot Camp is the preeminent learning-evaluation workshop in the learning-and-development field. It teaches LTEM, the Learning-Transfer Evaluation Model, but it covers so much more. If you want to know how to think clearly and effectively about learning evaluation, and avoid the myths and poor practices endemic in evaluation, the LTEM Boot Camp is the workshop for you and your organization.

You can learn more about the workshop in three ways:

  1. Go to the workshop enrollment page by clicking here. Note: This is for the September 2024 open-enrollment only.
  2. Go to the workshop description page by clicking here.
  3. Download the 100+ learning points taught in the workshop by clicking here.

The learning-points document is designed as a job aid, but can be used just for review or to diagnose how well you and your team are thinking about learning evaluation. It’s over 20 pages long, and has space for you to add your own insights.

 

Learner surveys are the most common method of measuring learning. I’ve been studying them for over a decade, ever since I learned of how they were deficient. Here, I will describe their benefits, but of course they must be well designed.

In 2016, I published a book on how to radically transform learner surveys into something useful. The title was Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. 

In 2022, I published the book again in a second edition, after updating it considerably based on the knowledge I gained in consulting with organizations in their learning-evaluation efforts. The second edition is titled: Performance-Focused Learner Surveys: Using Distinctive Questioning to Get Actionable Data and Guide Learning Effectiveness (Second Edition). It’s available on Amazon and you can learn more by clicking here. (This is the US Amazon, it’s available worldwide).

In the first edition, I listed nine benefits of learner surveys, but I had only touched the surface. In the second edition I share 20 benefits. Here’s the list:

Supporting Learning Design Effectiveness

  1. Red-flagging training programs that are not sufficiently effective.
  2. Gathering ideas for ongoing updates and revisions of learning programs.
  3. Judging the strengths, weaknesses, and viability of program updates and pilots.
  4. Providing learning architects and trainers with feedback to aid their development.
  5. Judging the competence of learning architects and trainers.
  6. Judging the contributions to learning made by people outside of the learning team.
  7. Assessing the contributions of learning supports and organizational practices.

Supporting Learners in Learning and Application

  1. Helping learners reflect on and reinforce what they learned.
  2. Helping learners determine what (if anything) they plan to do with their learning.
  3. Nudging learners to greater learning and application efforts.

Nudging Action Through Stealth Messaging

  1. Guiding learning architects to create more effective learning by sharing survey questions before learning designs are finalized and sharing survey results after data is gathered.
  2. Guiding trainers to utilize more effective learning methods by sharing survey questions before learning designs are finalized and sharing survey results after data is gathered.
  3. Guiding organizational stakeholders to support learning efforts more effectively by sharing survey questions and survey results.
  4. Guiding organizational decision makers to better appreciate the complexity and depth of learning and development—helping the learning team to gain credibility and autonomy.

Supporting Relationships with Learners and Other Key Stakeholders

  1. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the learning intervention and/or the instructors.
  2. Upholding the spirit of common courtesy by giving learners a chance for feedback.
  3. Enabling learner frustrations to be vented—to limit damage from negative back-channel communications.

Maintaining Organizational Credibility

  1. Engaging in visibly credible efforts to assess learning effectiveness.
  2. Engaging in visibly credible efforts to utilize data to improve effectiveness.
  3. Reporting out data to demonstrate learning effectiveness.

If you want to keep learning from me, sign up for my list. https://www.worklearning.com/sign-up/.

If you want to contact me or set up a meeting, visit my contact page. https://www.worklearning.com/contact/.

 

The LEARNNOVATORS team (specifically Santhosh Kumar) asked if I would join them in their Crystal Balling with Learnnovators interview series, and I accepted! They have some really great people on the series, I recommend that you check it out!

The most impressive thing was that they must have studied my whole career history and read my publication list and watched my videos because they came up with a whole set of very pertinent and important questions. I was BLOWN AWAY—completely IMPRESSED! And, given their dedication, I spent a ton of time preparing and answering their questions.

It’s a two part series and here are the links:

Here are some of the quotes they pulled out and/or I’d like to highlight:

Learning is one of the most wondrous, complex, and important areas of human functioning.

The explosion of different learning technologies beyond authoring tools and LMSs is likely to create a wave of innovations in learning.

Data can be good, but also very very bad.

Learning Analytics is poised to cause problems as well. People are measuring all the wrong things. They are measuring what is easy to measure in learning, but not what is important.

We will be bamboozled by vendors who say they are using AI, but are not, or who are using just 1% AI and claiming that their product is AI-based.

Our senior managers don’t understand learning, they think it is easy, so they don’t support L&D like they should.

Because our L&D leaders live in a world where they are not understood, they do stupid stuff like pretending to align learning with business terminology and business-school vibes—forgetting to align first with learning.

We lie to our senior leaders when we show them our learning data—our smile sheets and our attendance data. We then manage toward these superstitious targets, causing a gross loss of effectiveness.

Learning is hard and learning that is focused on work is even harder because our learners have other priorities—so we shouldn’t beat ourselves up too much.

We know from the science of human cognition that when people encounter visual stimuli, their eyes move rapidly from one object to another and back again trying to comprehend what they see. I call this the “eye-path phenomenon.” So, because of this inherent human tendency, we as presenters—as learning designers too!—have to design our presentation slides to align with these eye-path movements.

Organizations now—and even more so in the near future—will use many tools in a Learning-Technology Stack. These will include (1) platforms that offer asynchronous cloud-based learning environments that enable and encourage better learning designs, (2) tools that enable realistic practice in decision-making, (3) tools that reinforce and remind learners, (4) spaced-learning tools, (5) habit-support tools, (6) insight-learning tools (those that enable creative ideation and innovation), et cetera

Learnnovators asked me what I hoped for the learning and development field. Here’s what I said:

Nobody is good at predicting the future, so I will share the vision I hope for. I hope we in learning and development continue to be passionate about helping other people learn and perform at their best. I hope we recognize that we have a responsibility not just to our organizations, but beyond business results to our learners, their coworkers/families/friends, to the community, society, and the environs. I hope we become brilliantly professionalized, having rigorous standards, a well-researched body of knowledge, higher salaries, and career paths beyond L&D. I hope we measure better, using our results to improve what we do. I hope we, more-and-more, take a small-S scientific approach to our practices, doing more A-B testing, compiling a database of meaningful results, building virtuous cycles of continuous improvement. I hope we develop better tools to make building better learning—and better performance—easier and more effective. And I hope we continue to feel good about our contributions to learning. Learning is at the heart of our humanity!

For those of you who don’t know Matt Richter, President of the Thiagi Group, he’s one of the most innovative thinkers when it comes to creating training that both sizzles and supports work performance. Recently, Matt and I began partnering in a new podcast, Truth In Learning, which I’ll have more to say about later once I figure out where the escape hatch is.

NOW, I want to share with you a brilliant new article, that Matt surprised me with, on his efforts to brainstorm innovative ways to use LTEM (The Learning-Transfer Evaluation Model).

You should read his article, but just to give you the list of seven uses for LTEM:

  1. Learning Evaluation—The primary intent of the LTEM framework.
  2. Instructional Design—To negotiate with stakeholders the outcomes desired.
  3. Training Game Design—To ensure games/activities have an instructional purpose.
  4. Coaching—Helping to build a development plan for those who are coached.
  5. Performance Consulting—To focus on performances that matter along the journey.
  6. Keynoting/Presenting—To ensure a focus on meaningful outcomes, not just infotainment.
  7. Sales/Business Development—To keep sales conversations focused on meaningful outcomes.

We are All in this Together

One of the great benefits of publishing LTEM is that since its publication last year I’m regularly being contacted by people whose organizations are finding new and innovative ways to utilize LTEM—and not just for learning evaluation but as a central element of their learning strategy and practice.

I’m especially pleased with those who have taken LTEM really deep, and I’d like to give a shout out to Elham Arabi who is doing her doctoral dissertation using LTEM as a spur to supporting a hospital’s effort to maximize the benefits or their learning interventions. Congrats to her for being accepted as a speaker at the upcoming eLearning Guild Learning Solutions Conference, March 31 to April 2 (2020) in Orlando. The title of her talk is: Using Evaluation Data to Enhance Your Training Programs.

Share Your Examples and Innovations

Please share your innovations and ideas about using LTEM in your workplace, on social media, or by contacting me at https://www.worklearning.com/contact/. I would really love to hear how it’s going, including any obstacles you’ve faced, your success stories, etc.

And, of course, if you’d like me to help your organization utilize LTEM, or just be the face of LTEM to your organization, please contact me so we can set up a time to talk, and consider my LTEM workshop to introduce LTEM to your team.

 

 

People keep asking me for references to the claim that learner surveys are not correlated—or are virtually uncorrelated—with learning results. In this post, I include them, with commentary.

 

 

Major Meta-Analyses

Here are the major meta-analyses (studies that compile the results of many other scientific studies using statistical means to ensure fair and valid comparisons):

For Workplace Training

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-357.

Hughes, A. M., Gregory, M. E., Joseph, D. L., Sonesh, S. C., Marlow, S. L., Lacerenza, C. N., Benishek, L. E., King, H. B., Salas, E. (2016). Saving lives: A meta-analysis of team training in healthcare. Journal of Applied Psychology, 101(9), 1266-1304.

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93, 280-295.

For University Teaching

Uttl, B., White, C. A., Gonzalez (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.

What these Results Say

These four meta-analyses, covering over 200 scientific studies, find correlations between smile-sheet ratings and learning to average about 10%, which is virtually no correlation at all. Statisticians consider correlations below 30% to be weak correlations, and 10% then is very weak.

What these Results Mean

These results suggest that typical learner surveys are not correlated with learning results.

From a practical standpoint:

 

If you get HIGH MARKS on your smile sheets:

You are almost equally likely to have

(1) An Effective Course

(2) An Ineffective Course

 

If you get LOW MARKS on your smile sheets:

You are almost equally likely to have

(1) A Poorly-Designed Course

(2) A Well-Designed Course

 

Caveats

It is very likely that the traditional smile sheets that have been used in these scientific studies, while capturing data on learner satisfaction, have been inadequately designed to capture data on learning effectiveness.

I have developed a new approach to learner surveys to capture data on learning effectiveness. This approach is the Performance-Focused Smile Sheet approach as originally conveyed in my 2016 award-winning book. As of yet, no scientific studies have been conducted to correlate the new smile sheets with measures of learning. However, many many organizations are reporting substantial benefits. Researchers or learning professionals who want my updated list of recommended questions can access them here.

Reflections

  1. Although I have written a book on learner surveys, in the new learning evaluation model, LTEM (Learning-Transfer Evaluation Model), I place these smile sheets at Tier 3, out of eight tiers, less valuable than measures of knowledge, decision-making, task performance, transfer, and transfer effects. Yes, learner surveys are worth doing, if done right, but they should not be the only tool we use when we evaluate learning.
  2. The earlier belief—and one notably advocated by Donald, Jim, and Wendy Kirkpatrick—that there was a causal chain from learner reactions to learning, behavior, and results has been shown to be false.
  3. There are three types of questions we can utilize on our smile sheets: (1) Questions that focus on learner satisfaction and the reputation of the learning, (2) Questions that support learning, and (3) Questions that capture information about learning effectiveness.
  4. It is my belief that we focus too much on learner satisfaction, which has been shown to be uncorrelated with learning results—and we also focus too little on questions that gauge learning effectiveness (the main impetus for the creation of Performance-Focused Smile Sheets).
  5. I do believe that learner satisfaction is important, but it is not most important.

Learning Opportunities regarding Learner Surveys

Every year or so, based on work with clients and new analysis, I like to provide to the public an updated recommended set of smile-sheet questions (free).

 

You can access the New Questions by clicking here.

 

Related Resources

LTEM Boot Camp now accepting enrollments!
The World's Preeminent Workshop on LTEM & Learning Evaluation.
January-February 2027 Cohort Coming Soon!

Will Thalheimer facilitates this intensive six-session program,
bringing research-based wisdom to learning evaluation.

Click Right Here to Learn More!

 

Click ESC to Close...

This will close in 15 seconds