Is My Book Award Worthy?

, ,

Is my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, award worthy?

I think so, buy I'm hugely biased! SMILE.

Boxshot-rendering redrawn-no-shadow2

Here's what I wrote today on an award-submission application:

Performance-Focused Smile Sheets: A Radical Rethinking of Dangerous Art Form is a book, published in February 2016, written by Will Thalheimer, PhD, President of Work-Learning Research, Inc.

The book reviews research on smile sheets (learner feedback forms), demonstrates the limitations of traditional smile sheets, and provides a completely new formulation on how to design and deploy smile sheets.

The ideas in the book — and the example questions provided — help learning professionals focus on "learning effectiveness" in supporting post-learning performance. Where traditional smile sheets focus on learner satisfaction and the credibility of training, Performance-Focused Smile Sheets can also focus on science-of-learning factors that matter. Smile sheets can be transformed by focusing on learner comprehension, factors that influence long-term remembering, learner motivation to apply what they've learned, and after-learning supports for learning transfer and application of learning to real-world job tasks.

Smile sheets can also be transformed by looking beyond Likert-like responses and numerical averages that dumb-down our metrics and lead to bias and paralysis. We can go beyond meaningless averages ("My course is a 4.1!") and provide substantive information to ourselves and our stakeholders.

The book reviews research that shows that so-called "learner-centric" formulations are filled with dangers — as research shows that learners don't always know how they learn best. Smile-sheet questions must support learners in making smile-sheet decisions, not introduce biases that warp the data.

For decades our industry has been mired in the dishonest and disempowering practice of traditional smile sheets. Thankfully, a new approach is available to us.

Sure! I'd love to see my work honored. More importantly, I'd love to see the ideas from my book applied wisely, improved, and adopted for training evaluation, student evaluations, conference evaluations, etc. 

You can help by sharing, by piloting, by persuading, by critiquing and improving! That will be my greatest award!

Personalized Learning with Video… by Consensus.

, ,

Research shows that one-on-one tutoring is generally highly effective — with a good tutor, of course. Similarly, John Anderson — the cognitive psychologist — along with others — developed Intelligent tutoring systems (ITS) that tailor content to learners based on cognitive models of what they know and don't know.

These learning approaches are beneficial because they personalize learning based on a diagnosis of learner knowledge. This is not a new concept. Indeed, the Socratic Method also took learners on a journey based on their responses. B. F. Skinner's operant conditioning provided reinforcement based on learner actions. Skinner's Programmed Learning and Fred Keller's Personalized System of Instruction are practical applications based on operant conditioning principles.

We've known for millennium that personalized learning is good — and we've even dabbled in scalable implementations like programmed learning — but for the most part we are still awaiting the promise of such personalization.

Now may be the time. New technologies are beginning to show promise. For example, subscription-learning threads based on personalized spacing schedules personalize learning nuggets based on learner responses. Still, one area where personalization hasn't been much in evidence is video. That may be changing.

Consensus Demo

Consensus allows its users to create videos tailored to each learner. Their focus is on sales and marketing — and particularly on creating demos — but the technology could be used for other needs as well. Consensus tailors content to users based on their interests, background, etc. When folks choose a topic as “Very Important,” users get the full video segment and get it first. When folks choose a topic as “Somewhat Important,” users get a summary video. When they choose “Not Important,” users don’t get any segments associated with those topics.

Consensus can also segment users based on their role in an organization, based on the size of their company, or based on other demographics as well.

They’ve got a great video that you can tailor to your needs. Worth watching! Click to check out Consensus.

 

Learning Executive: Using Credentialing to Accelerate Change!

,

In my ongoing research interviewing learning executives, I occasionally come across stories or ideas that just can't wait until the full set of data is collected.

This week, I interviewed a director of employee development and training at a mid-sized distribution company. She expressed many of the frustrations I've heard before in my consulting work with L&D (Learning and Development) leaders. For example:

  • Lack of good learning measurement, causing poor feedback to L&D stakeholders.
  • Too many task requirements to allow for strategic thinking in L&D.
  • While some SME's are great trainers, too many deliver poorly-designed sessions.
  • Lack of some sort of competency testing of learners.
  • Lack of follow-through after training, limiting likelihood of successful application to the job.

There were so many changes to make that it appeared overwhelming — as if making positive change was going to take forever.

Then she got an idea. Her organization had begun to train its customers (in addition to training its employees), and they began to search for ways to demonstrate the value and credibility of the customer-focused courses.

What this director realized was that accreditation might serve multiple purposes — if it provided a rigorous evaluation scheme; one that demanded living up to certain standards.

She found an accrediting agency that fit the bill. IACET, the International Association of Continuing Education and Training would certify her organization, but the organization would have to prove that it engaged in certain practices.

This turned out to be a game changer. The requirements, more often than not, propelled her organization in directions she had hoped they would travel anyway. The accreditation process had become a powerful lever in the director's change-management efforts.

Some of things that the accreditation required:

  • The L&D organization had to demonstrate training needs, not just take orders for courses.
  • They had to map learning evaluations back to learning objectives, ensuring relevance in evaluations.
  • They had to have objectives that tied into learning outcomes for each course.
  • Trainers had to be certified in training skills (aligned to research-based best practices).
  • Trainers had to be regularly trained to maintain their certifications.
  • Et cetera…

While before it was difficult for her to get some of her SME's to take instructional design seriously, now accreditation constraints propelled them in the right direction. Whereas before, SME's balked at creating tests of competence, now the accreditation requirements demanded compliance. Whereas before, her SME's could skip out on training on evidence-based learning practice, now they were compelled to take it seriously — otherwise they may lose their accreditation; thus losing the differentiation their training provides to customers.

The accreditation process was a catalyst, but it wouldn't work on it's own — and it's not a panacea. The director acknowledges that a full and long-term change management effort is required, but accreditation has helped her move the needle toward better learning practices.

Neuroscientists, Just Now, Detail 97 Newly-Found Regions in Brain

The journal Nature reported today that a new map of the brain reveals 97 newly-found regions, each specialized to certain functions.

In an article, entitled, "A multi-modal parcellation of human cerebral cortex," the 12 scientists found 97 new structures and validated the "83 areas previously reported using post-mortem microscopy," bringing the total structures per hemisphere to 180.

New Brain Map

As the scientists declare, such a map is critical to neuroscientists in evaluating neurological functioning.

"Understanding the amazingly complex human cerebral cortex requires a map (or parcellation) of its major subdivisions, known as cortical areas. Making an accurate areal map has been a century-old objective in neuroscience."

As should be obvious, we are still in the infancy of neuroscience. Any recommendations about learning — supposedly based on neuroscience, should be taken with extreme skepticism. See related article.

For nice review of the findings, see article in the New York Times.

 

10-Minute Videos for Learning

Here's another example of short videos being used to share content.

It's called 10-minute insights, and it was created by Edcast, a platform that enables folks to create micro-learning platforms.

Here are some examples:

These are bare-boned videos, some more successful than others, some with reasonable production values, some with some serious degradations in lighting, etc.

Interestingly, Educast says they use "bite-sized content curated from thousands of high-quality sources using AI-based algorithms."

Critique:

The 10-Minute Insights website is about content and seemingly about celebrity. There are no supports for learning — no practice, no feedback, no support for remembering or application, no metrics for evaluating learning results. It's just content. That can be okay…but it's usually not sufficient…

 

Blood Glucose — Probably NOT a determinant of Willpower.

New research just published (July 11, 2016) shows that the blood-glucose hypothesis about willpower is probably not true, at least not as evidenced by current scientific studies.

The idea — previously held — was that blood glucose mediated the propensity of people to exert willpower, with decreased blood glucose making it less likely that someone would persevere in a task.

Bigstock-Group-of-business-people-Busi-14495489

I myself had read some of the previous research and shared the blood-glucose hypothesis. Apparently, current research doesn't back up this claim. Miguel A. Vadillo, Natalie Gold, and Magda Osman — writing in the journal Psychological Science, utilized a "new meta-analytic tool, p-curve analysis, to examine the reliability of the evidence from" 19 studies focusing on blood glucose and willpower. They found that overall there was not a reliable effect of glucose.

Here are two quotes from the article:

The findings from the present study are a surprise in the context of the wide acceptance of the glucose hypothesis in general scientific research and its popularity, as evidenced by the number of citations of Gailliot et al. (2007) in the literature and the continued influence of this hypothesis in recent reviews on ego depletion (e.g., Baumeister, 2014; Baumeister & Alghamdi, 2015). Moreover, the hypothesis has intuitive and seemingly practical appeal. If one accepts that a failure of self-control in regulating actions contributes to the many personal and societal problems that people face (Baumeister et al., 2000), then glucose supplements would provide a simple means to enhance willpower and ameliorate these problems (Baumeister & Tierny, 2011). In light of our results, it is doubtful that such a recommendation will work in the real world. This conclusion converges with recent evidence that glucose might have little or no impact on domain-general decision-making tasks (Orquin & Kurzban, 2016) and with an intriguing series of meta-analyses and preregistered replications suggesting that the ego-depletion effect itself might be less robust than previously thought (Carter, Kofler, Forster, & McCullough, 2015; Hagger et al., in press).

Our results suggest that, on average, these studies have little or no evidential value, but they do not allow us to determine whether the significant results are due to publication bias, selective reporting of outcomes or analyses, p-hacking, or all of these. It is not impossible that some of these studies are exploring small but true effects and that their evidential value may be diluted by the biases that pervade the rest of the studies. Perhaps future research will show that glucose does play a role in ego depletion effects, but our conclusions are based on the analysis of the extant literature in this area. Thus, our contribution must be seen as an additional piece of information in the wider context of attempts to verify the reliability of the glucose model of ego depletion.

 

Practical Ramifications

In the past, I used the glucose-depletion idea as a partial explanation why day-long training sessions were difficult for learners. I also used it as a rationale for plying my workshop participants with treats in the afternoon. Well, at least I tried to make them somewhat healthy! As the meta-analysis above reveals, it's likely that some other mechanism is involved in the difficulties learners have during intensive learning sessions. As trainers and instructional designers, we still have to figure out a way to support learners during long learning sessions to prevent attention-zapping fatigue…

 

Research Reviewed

Vadillo, M. A., Gold, N., and Osman, M. (2016). The bitter truth about sugar and willpower: The limited evidential value of the glucose model of ego depletion. Psychological Science, Published Online July 11, 2016. Available at: http://pss.sagepub.com/content/early/2016/07/08/0956797616654911.full

 

The Last Two Decades of Neuroscience Research (via fMRI) Called Into Question!

,

Updated July 11, 2016. An earlier version was more apocalyptic.

==============================

THIS IS HUGE. A large number of studies from the last 15 years of neuroscience research (via fMRI) could be INVALID!

A recent study in the journal PNAS looked at the three most commonly used software packages used with fMRI machines. Where they expected to find a normal familywise error rate of 5%, they found error rates up to 70%.

Here’s what the authors’ wrote:

“Using mass empirical analyses with task-free fMRI data, we have found that the parametric statistical methods used for group fMRI analysis with the packages SPM, FSL, and AFNI can produce FWE-corrected cluster P values that are erroneous, being spuriously low and inflating statistical significance. This calls into question the validity of countless published fMRI studies based on parametric clusterwise inference. It is important to stress that we have focused on inferences corrected for multiple comparisons in each group analysis, yet some 40% of a sample of 241 recent fMRI papers did not report correcting for multiple comparisons (26), meaning that many group results in the fMRI literature suffer even worse false-positive rates than found here (37).”

In a follow-up blog post, the authors estimated that up to 3,500 scientific studies may be affected, which is down from their initial published estimate of 40,000. The discrepancy results because only studies at the edge of statistical reliability are likely to have results that might be affected. For an easy-to-read review of their walk-back, Wired has a nice piece.

The authors also point out that there is more to worry about than those 3,500 studies. An additional 13,000 studies don’t use any statistical correction at all (so they’re not affected by the software glitch reported in the scientific paper). However, these 13,000 studies use an approach that “has familywise error rates well in excess of 50%.” (cited from the blog post)

Here’s what the authors say in their walk-back:

“So, are we saying 3,500 papers are “wrong”? It depends. Our results suggest CDT P=0.01 results have inflated P-values, but each study must be examined… if the effects are really strong, it likely doesn’t matter if the P-values are biased, and the scientific inference will remain unchanged. But if the effects are really weak, then the results might indeed be consistent with noise. And, what about those 13,000 papers with no correction, especially common in the earlier literature? No, they shouldn’t be discarded out of hand either, but a particularly jaded eye is needed for those works, especially when comparing them to new references with improved methodological standards.”

 

Some Perspective

Let’s take a deep breadth here. Science works slowly and we need to see what other experts have to say in the coming months.

The authors reported that there were about 40,000 published studies in the last 15 years that might be affected. Of this amount, only some of 3,500 + 13,000 = 16,500 are affected. That’s 41% of published articles with a potential to have invalid results.

But, of course, in the learning field, we don’t care about all these studies as most of them have very little to do with learning or memory. Indeed, a search of the whole history of PsycINFO (a social-science database) finds a total of 22,347 articles mentioning fMRI at all. Searching for articles that have a learning or memory aspect culls this number down to 7,056. This is a very rough accounting, but it does put the overall findings in some perspective.

As the authors warn, it’s not appropriate to dismiss the validity of all the research articles, even if they’re in one of the suspect groups of studies. Instead, when looking at the potentially-invalidate articles, each one has to be examined individually to know whether it has problems.

Despite these comforting caveats, the findings by the scientists have implications for many neuroscience research studies over the past 15 years (when the bulk of neuroscience research has been done).

On the other hand, there truly haven’t been many neuroscience findings that have much practical relevance to the learning field as of yet. See my review for a critique of overblown claims about neuroscience and learning. Indeed, as I’ve argued elsewhere, neuroscience’s potential to aid learning professionals probably rests in the future. So, being optimistic, maybe these statistical glitches will end up being a good thing. First, perhaps they’ll propel greater scrutiny to research methodologies, improving future neuroscience research. Second, perhaps they’ll put the brakes on the myth-creating industrial complex around neuroscience until we have better data to report and utilize.

Still, a dark cloud of low credibility may settle over the whole neuroscience field itself, hampering researchers from getting funding, and making future research results difficult for practitioners to embrace. Time will tell.

 

Popular Press Articles Citing the Original Article (Published Before the Walk-Backs).

Here are some articles from the scientific press pointing out the potential danger:

  • http://arstechnica.com/science/2016/07/algorithms-used-to-study-brain-activity-may-be-exaggerating-results/
  • http://cacm.acm.org/news/204439-a-bug-in-fmri-software-could-invalidate-15-years-of-brain-research/fulltext
  • http://www.wired.co.uk/article/fmri-bug-brain-scans-results
  • http://www.zmescience.com/medicine/brain-imageflaw-57869/

==============================

Notes:

From Wikipedia (July 11, 2016): “In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.”

CLO: Huge Need for Angel Investing in Learning — from CEO’s

I was speaking to a CLO (Chief Learning Officer) today about research-to-practice stuff, and he made a statement that blew my mind. It was brilliant!

First, a little background.

He’s been a CLO at two organizations with over a decade of tenure in learning-executive positions. He knows what he’s talking about.

He talked persuasively about how many learning and development units are stuck in the old-world view of learning as delivering training (and training alone). In such learning cultures, L&D folks are order takers. A more enlightened approach includes training, but it also involves performance-consulting, performance-facilitation, knowledge management, improved evaluations, etc., with a key goal of helping to create a partnership between L&D and its organizational stakeholders.

Okay, that’s not news. We know this is the right thing to do.

Here’s what struck me as illuminating. He said that what’s really needed to make the leap from the old way to the new way, you need a large capital-like investment to get you started, to bring in committed change agents, to overcome inertia, to have resources and tools to let you leverage research-based methods. Without a large resource infusion, you get incremental change, but it’s just not enough to change the culture so you end up fighting long exhausting battles with only some success.

But what CEO’s are going to be so enlightened as to pony up the dough?

You’d think they might. After all, making huge investments upfront is what successful businesses do, what successful entrepreneurs do, what successful athletes do, etc.

We in L&D have to figure out a way to convince our CEO’s to make the investment. Otherwise, we’ll likely remain in the purgatory of order taking henceforth.

 

Smile Sheet Questions — New Examples July 2016

,

Update January 2018: To see my latest recommendations for smile-sheet question design, go to this web page.

===================

The Original Post:

The response to the book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, has been tremendous! Since February, when it was published, I’ve received hundreds of thank you’s from folks the world over who are thrilled to have a new tool — and to finally have a way to get meaningful data from learner surveys. At the ATD conference where I spoke recently, the book sold out it was so popular! If you want to buy the book, the best place is still SmileSheets.com, the book’s website.

Since publication, I’ve begun a research effort to learn how companies are utilizing the new smile-sheet approach — and to learn what’s working, what the roadblocks are, and what new questions they’ve developed. As I said in the book in the chapter that offers 26 candidate questions, I hope that folks tailor questions, improve them, and develop new questions. This is happening, and I couldn’t be more thrilled. If your company is interested in being part of my research efforts, please contact me by clicking here. Likewise, if you’ve got new questions to offer, let me know as well.

Avoiding Issues of Traditional Smile Sheets

Traditional smile sheets tend to focus on learners’ satisfaction and learners’ assessments of the value of the learning experience. Scientific research shows us that such learner surveys are not likely to be correlated with learning results. Performance-focused smile sheets offer several process improvements:

  1. Avoid Likert-like scales and numerical scales which create a garbage-in garbage-out problem, which don’t offer clear delineations between answer choices, which don’t support respondent decision-making, and which open responding to bias.
  2. Instead, utilize concrete answer choices, giving respondents more granularity, and enabling much more meaningful results.
  3. In addition to, or instead of, focusing on factors related to satisfaction and perceived value; target factors that are related to learning effectiveness.

New Example Questions

As new questions come to my attention, I’ll share them here on my blog and elsewhere. You can sign up for my email newsletter if you want to increase the likelihood that you’ll see new smile-sheet questions (and for other learning-research related information as well).

Please keep in mind that there are no perfect assessment items, no perfect learning metrics, and no perfect smile-sheet questions. I’ve been making improvements to my own workshop smile sheets for years, and every time I update them, I find improvements to make. If you see something you don’t like in the questions below, that’s wonderful! When evaluating an assessment item, it’s useful to ask whether the item (1) is targeting something important and (2) is it better than other items that we could use or that we’ve used in the past.

Question Example — A Question for Learners’ Managers

My first example comes from a world renowned data and media company. They decided to take one of the book’s candidate questions, which was designed for learners to answer, and modify the question to ask learners’ managers to answer. Their reasoning: The training is strategically important to their business and they wanted to go beyond self-report data. Also, they wanted to send “stealth messages” to learners’ managers that they as managers had a role to play in ensuring application of the training to the job.

Here’s the question (aimed at learners’ managers):

In regard to the course topics taught, HOW EFFECTIVELY WAS YOUR DIRECT REPORT ABLE to put what he/she learned into practice in order to PERFORM MORE EFFECTIVELY ON THE JOB?

A. He/she has NOT AT ALL ABLE to put the concepts into practice.

B. He/she has GENERAL AWARENESS of the concepts taught, but WILL NEED MORE TRAINING / GUIDANCE to put the concepts into practice.

C. He/she WILL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.

D. He/she is at a FULLY COMPETENT LEVEL in using the concepts taught.

E. He/she is at an EXPERT LEVEL in using the concepts taught.

Question Example — Tailoring a Question to the Topic

In writing smile-sheet questions, there’s a tradeoff between generalization and precision. Sometimes we need a question to be relevant to multiple courses. We want to compare courses to one another. Personally, I think we overvalue this type of comparison, even when we might be comparing apples to oranges. For example, do we really want to compare scores on courses that teach such disparate topics as sexual harassment, word processing, leadership, and advanced statistical techniques? Still, there are times when such comparisons make sense.

The downside of generalizability is that we lose precision. Learners are less able to calibrate their answers. Analyzing the results becomes less meaningful. Also, learners see the learner-survey process as less valuable when questions are generic, so they give less energy and thought to answering the questions, and our data become less valuable and more biased.

Here is a question I developed for my own workshop (on how to create better smile sheets, by the way SMILE):

How READY are you TO WRITE QUESTIONS for a Performance-Focused Smile Sheet?

CIRCLE ONE OR MORE ANSWERS

AND/OR WRITE YOUR REASONING BELOW

A. I’m STILL NOT SURE WHERE TO BEGIN.

B. I KNOW ENOUGH TO GET STARTED.

C. I CAN TELL A GOOD QUESTION FROM A BAD ONE.

D. I CAN WRITE MY OWN QUESTIONS, but I’d LIKE to get SOME FEEDBACK before using them.

E. I CAN WRITE MY OWN QUESTIONS, and I’m CONFIDENT they will be reasonably WELL DESIGNED.

More
Thoughts?

 

 

Note several things about this question. First to restate. It is infinitely more tailored than a generic question could be. It encourages more thoughtful responding and creates more meaningful feedback.

Second, you might wonder why all the CAPS! I advocate CAPS because (1) CAPS have been shown in research to slow reading speed. Too often, our learners burn through our smile-sheet questions. Anything we can do to make them attend more fully is worth trying. Also, (2) respondents often read the full question and then skim back over it when determining how to respond. I want them to have an easy way to parse the options. Full disclosure. To my knowledge, all CAPS has not been studied yet for smile sheets. At this point, my advocacy for all CAPS is based on my intuition about how people process smile-sheet questions. If you’d like to work with me to test this in a scientifically rigorous fashion, please contact me.

Third, notice the opportunity for learners to write clarifying comments. Open-ended questions, though not easily quantifiable, can be the most important questions on smile sheets. They can provide intimate granularity — a real sense of the respondents’ perceptions. In these questions, we’re using a hybrid format, a forced choice question followed by an open-ended opportunity for clarification. This not only enables the benefits of open-ended responding, but it also enables us to get clarifying meaning. In addition, in some way it provides a reality-check on our question design. If we notice folks responding in ways that aren’t afforded in the answer choices given, we can improve our question for later versions.

 

Question Example — Simplifying The Wording

In writing smile-sheet questions, there’s another tradeoff to consider. More words add more precision, but fewer words add readability and motivation to engage the question fully. In the book, I talk about what I once called, “The World’s Best Smile Sheet Question.” I liked it partly because the answer choices were more precise than a Likert-like scale. It did have one drawback; it used a lot of words. For some audiences this might be fine, but for others it might be entirely inappropriate.

Recently, in working with a company to improve their smile sheet, a first draft included the so-called World’s Best Smile Sheet Question. But they were thinking of piloting the new smile sheet for a course to teach basic electronics to facilities professionals. Given the topic and audience, I recommended a simpler version:

How able will you be to put what you’ve learned into practice on the job?  Choose one.

A. I am NOT AT ALL ready to use the skills taught.
B. I need MORE GUIDANCE to be GOOD at using these skills
C. I need MORE EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.

This version nicely balances precision with word count.

 

Question Example — Dealing with the Sticky Problem of “Motivation”

In the book, I advocate a fairly straightforward question asking learners about their motivation to apply what they’ve learned. In many organizations — in many organizational cultures — this will work fine. However in others, our trainers may be put off by this. They’ll say, “Hey, I can’t control people’s motivations.” They’re right, of course. They can’t control learners’ motivations, but they can influence them. Still, it’s critical to realize that motivation is a multidimensional concept. When we speak of motivation, we could be talking simply about a tendency to take action. We could be talking about how inspired learners are, or how much they believe in the value of the concepts, or how much self-efficacy they might have. It’s okay to ask about motivation in general, but you might generate clearer data if you ask about one of the sub-factors that comprise motivation.

Here is a question I developed recently for my Smile-Sheet Workshop:

How motivated are you to IMPLEMENT PERFORMANCE-FOCUSED SMILE SHEETS in your organization?

CIRCLE ONLY ONE ANSWER. ONLY ONE!

A. I’m NOT INTERESTED IN WORKING TOWARD IMPLEMENTING this.

B. I will confer with my colleagues to SEE IF THERE IS INTEREST.

C. I WILL ADVOCATE FOR performance-focused smile sheet questions.

D. I WILL VIGOROUSLY CHAMPION performance-focused smile sheet questions.

E. Because I HAVE AUTHORITY, I WILL MAKE THIS HAPPEN.

More
Thoughts?

In this question, I’m focusing on people’s predilection to act. Here I’ve circumnavigated any issues in asking learners to divulge their internal motivational state, and instead I’ve focused the question on the likelihood that they will utilize their newly-learned knowledge in developing, deploying, and championing performance-focused smile sheets.

 

Final Word

It’s been humbling to work on smile sheet improvements over many years. My earlier mistakes are still visible in the digital files on my hard drive. I take solace in making incremental improvements — and in knowing that the old way of creating smile-sheet questions is simply no good at all, as it provides us with perversely-irrelevant information.

As an industry — and the learning industry is critically important to the world — we really need to work on our learning evaluations. Smile sheets are just one tool in this. Unfortunately, poorly constructed smile sheets have become our go-to tool, and they have led us astray for decades.

I hope you find value in my book (SmileSheets.com). More importantly, I hope you’ll participate along with some of the world’s best-run companies and organizations in developing improved smile sheet questions. Again, please email me with your questions, your question improvements, and alternatively, with examples of poorly-crafted questions as well.