Purpose of Workplace Learning and Development. Survey Inquiry

, ,

Learner-Feedback Current Practices Survey 2017-2018

, ,

Jobs in Learning Measurement — Let’s Try Something

, ,

Dear Readers,

Many of you are now following me and my social-media presence because you’re interested in LEARNING MEASUREMENT. Probably because of my recent book on Performance-Focused Smile Sheets (which you can learn about at the book’s website, SmileSheets.com).

More and more, I’m meeting people who have jobs that focus on learning measurement. For some, that’s their primary focus. For most, it’s just a part of their job.

Today, I got an email from a guy looking for a job in learning measurement and analytics. He’s a good guy, smart and passionate, and so he ought to be able to find a good job where he can really help. So here’s what I’m thinking. You, my readers are some of the best and brightest in the industry — you care about our work and you look to the scientific research as a source of guidance. You are also, many of you, enlightened employers, looking to recruit and hire the best and brightest. So it seems obvious that I should try to connect you…

So here’s what we’ll try. If you’ve got a job in learning measurement, let me know about it. I’ll post it here on my blog. This will be an experiment to see what happens. Maybe nothing… but it’s worth a try.

Now, I know many of you are also loyal readers because of things BESIDES learning measurement, for example, learning research briefs, research-based insights, elearning, subscription learning, learning audits, and great jokes… but let’s keep this experiment to LEARNING MEASUREMENT JOBS at first.

BOTTOM LINE: If you know of a learning-measurement job, let me know. Email me here…

Improving Our Performance in Watching the Presidential Debates

, ,

I work in the learning-and-performance field. I watched the first two presidential debates. They were problematic to say the least. So, I thought to myself, “Hey Will, you’re a learning-and-performance professional. What might be done to improve them?” Actually, it was more like, “Damn, somebody’s got to make these debates better. They’re not creating the best outcomes. They’re not really educating us on the important issues for the presidential election. Our debates are not helping the democratic process. Indeed, they’re probably creating harm.”

24564574914_0cdd268f92_zPicture above by Donkey Hotey 2016 Creative Commons on Flickr

In looking at the debate process, there are several leverage points, including the following:

  1. The debate format.
  2. The questions asked.
  3. The candidates’ responses.
  4. The citizenry’s cognitive processing of the debate.
  5. The news media’s and social media’s messaging about the debate.
  6. The citizenry’s cognitive processing of the media messaging.

For most of these, we — the citizenry — have very little direct leverage to influence the process. Together, we can influence the social-media messages, but as individuals, we have very little influence there. Tears in the ocean.

But certainly the media and the candidates are acting with intentionality to influence the citizenry. Candidates act bold and confident because they know that people, in general, are suckers for those who display confidence. Candidates use certain words to influence — even if those words are too general to be meaningful (words like freedom, equality, strength, diversity). Candidates avoid talking about complicated issues, because they think we can’t understand. Candidates speak in terms of black and white, good and bad, either this or that — though the world is shaded in sepia tones — because they think we need certainty.

They act because they think we as citizens will react in predictable ways. But what if we changed and improved our responses to the debates? What if we as citizens improved our cognitive processing and instead of having knee-jerk reactions, we improved our thinking? What if instead of being persuaded by irrelevancies, we were persuaded by a clear understanding of the issues? Is there anything we can do to improve and deepen our thinking about the debates?

Yes! We can do a better job! Not a perfect job, not a brilliant job, but we can improve and deepen our thinking if we take some time to think about what we care about and what matters.

I’m sure my efforts here will be inadequate, but I offer a one-page checklist to spur your thinking. Take a look, try it out at Wednesday’s final Presidential debate and let me know how you’d improve it. Tell me what works and what doesn’t. Better yet, let me know what else we can do to improve the results of the debates.

 

Try the Presidential Debates Insight Checklist

 

Read My Article on LinkedIn

 

 

WHAT ELSE? WHAT DO YOU THINK?

Okay, I really feel like this effort is too inconsequential and too unlikely to make a difference, so I’d love to hear your ideas:

  • What’s wrong with this approach?
  • What could make it better?
  • What else could be done to improve the outcome of the debates — that is, creating a better informed citizenry.
  • Are there other tools in our learning-and-performance toolbox we might use?
  • What could make our debates great again? (sorry, couldn’t resist)

Sweet Mother of Deception — Sugar Industry Lessons for the Learning Professional

,

OMG. If you haven’t heard, the sugar industry engaged in a sophisticated plot to manipulate the public into believing that saturated fat was the enemy of good health–NOT sugar. They were completely successful in capturing the narrative, so much so that they likely caused millions of deaths, complications from diabetes, illnesses due to obesity, cancer and heart disease, and more.

For articles on this blatantly immoral activity: New York Times, Stat — Frontiers of Health and Medicine

Of course, this kind of behavior happens over and over.

  • The great recession found malfeasance within the banking industry.
  • The tobacco industry famously tried to claim smoking wasn’t harmful.
  • The oil and gas industry has fought global warming warnings for years.

In the learning field, are we naive enough to believe that everything is pure?

I’ve seen things that make me mad–real mad! For example, some of the top-10, top-20 lists are basically bought and paid for… The neuroscience hype to sell products… Well-known “gurus” who don’t mention improved practices because they are protecting their legacy offerings … Trade associations that provide members with a diet of information tilted to their vendor benefactors…

What have you seen?

What’s your motto? Regrettably, mine is, “I’m mad as hell, and I’m still taking it…”

I’m dying to know though, have you seen malfeasance? Slippery slopes? Money changing hands?

Sadder still, given all the great, passionate, caring, and honest folks in our field.

Connect with me privately if you need to… (info a.t work-learning d.o.t com)

Sunshine as disinfectant and all that…

Is My Book Award Worthy?

, ,

Is my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, award worthy?

I think so, buy I'm hugely biased! SMILE.

Boxshot-rendering redrawn-no-shadow2

Here's what I wrote today on an award-submission application:

Performance-Focused Smile Sheets: A Radical Rethinking of Dangerous Art Form is a book, published in February 2016, written by Will Thalheimer, PhD, President of Work-Learning Research, Inc.

The book reviews research on smile sheets (learner feedback forms), demonstrates the limitations of traditional smile sheets, and provides a completely new formulation on how to design and deploy smile sheets.

The ideas in the book — and the example questions provided — help learning professionals focus on "learning effectiveness" in supporting post-learning performance. Where traditional smile sheets focus on learner satisfaction and the credibility of training, Performance-Focused Smile Sheets can also focus on science-of-learning factors that matter. Smile sheets can be transformed by focusing on learner comprehension, factors that influence long-term remembering, learner motivation to apply what they've learned, and after-learning supports for learning transfer and application of learning to real-world job tasks.

Smile sheets can also be transformed by looking beyond Likert-like responses and numerical averages that dumb-down our metrics and lead to bias and paralysis. We can go beyond meaningless averages ("My course is a 4.1!") and provide substantive information to ourselves and our stakeholders.

The book reviews research that shows that so-called "learner-centric" formulations are filled with dangers — as research shows that learners don't always know how they learn best. Smile-sheet questions must support learners in making smile-sheet decisions, not introduce biases that warp the data.

For decades our industry has been mired in the dishonest and disempowering practice of traditional smile sheets. Thankfully, a new approach is available to us.

Sure! I'd love to see my work honored. More importantly, I'd love to see the ideas from my book applied wisely, improved, and adopted for training evaluation, student evaluations, conference evaluations, etc. 

You can help by sharing, by piloting, by persuading, by critiquing and improving! That will be my greatest award!

Smile Sheet Questions — New Examples July 2016

,

The response to the book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, has been tremendous! Since February, when it was published, I’ve received hundreds of thank you’s from folks the world over who are thrilled to have a new tool — and to finally have a way to get meaningful data from learner surveys. At the ATD conference where I spoke recently, the book sold out it was so popular! If you want to buy the book, the best place is still SmileSheets.com, the book’s website.

Since publication, I’ve begun a research effort to learn how companies are utilizing the new smile-sheet approach — and to learn what’s working, what the roadblocks are, and what new questions they’ve developed. As I said in the book in the chapter that offers 26 candidate questions, I hope that folks tailor questions, improve them, and develop new questions. This is happening, and I couldn’t be more thrilled. If your company is interested in being part of my research efforts, please contact me by clicking here. Likewise, if you’ve got new questions to offer, let me know as well.

Avoiding Issues of Traditional Smile Sheets

Traditional smile sheets tend to focus on learners’ satisfaction and learners’ assessments of the value of the learning experience. Scientific research shows us that such learner surveys are not likely to be correlated with learning results. Performance-focused smile sheets offer several process improvements:

  1. Avoid Likert-like scales and numerical scales which create a garbage-in garbage-out problem, which don’t offer clear delineations between answer choices, which don’t support respondent decision-making, and which open responding to bias.
  2. Instead, utilize concrete answer choices, giving respondents more granularity, and enabling much more meaningful results.
  3. In addition to, or instead of, focusing on factors related to satisfaction and perceived value; target factors that are related to learning effectiveness.

New Example Questions

As new questions come to my attention, I’ll share them here on my blog and elsewhere. You can sign up for my email newsletter if you want to increase the likelihood that you’ll see new smile-sheet questions (and for other learning-research related information as well).

Please keep in mind that there are no perfect assessment items, no perfect learning metrics, and no perfect smile-sheet questions. I’ve been making improvements to my own workshop smile sheets for years, and every time I update them, I find improvements to make. If you see something you don’t like in the questions below, that’s wonderful! When evaluating an assessment item, it’s useful to ask whether the item (1) is targeting something important and (2) is it better than other items that we could use or that we’ve used in the past.

Question Example — A Question for Learners’ Managers

My first example comes from a world renowned data and media company. They decided to take one of the book’s candidate questions, which was designed for learners to answer, and modify the question to ask learners’ managers to answer. Their reasoning: The training is strategically important to their business and they wanted to go beyond self-report data. Also, they wanted to send “stealth messages” to learners’ managers that they as managers had a role to play in ensuring application of the training to the job.

Here’s the question (aimed at learners’ managers):

In regard to the course topics taught, HOW EFFECTIVELY WAS YOUR DIRECT REPORT ABLE to put what he/she learned into practice in order to PERFORM MORE EFFECTIVELY ON THE JOB?

A. He/she has NOT AT ALL ABLE to put the concepts into practice.

B. He/she has GENERAL AWARENESS of the concepts taught, but WILL NEED MORE TRAINING / GUIDANCE to put the concepts into practice.

C. He/she WILL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.

D. He/she is at a FULLY COMPETENT LEVEL in using the concepts taught.

E. He/she is at an EXPERT LEVEL in using the concepts taught.

Question Example — Tailoring a Question to the Topic

In writing smile-sheet questions, there’s a tradeoff between generalization and precision. Sometimes we need a question to be relevant to multiple courses. We want to compare courses to one another. Personally, I think we overvalue this type of comparison, even when we might be comparing apples to oranges. For example, do we really want to compare scores on courses that teach such disparate topics as sexual harassment, word processing, leadership, and advanced statistical techniques? Still, there are times when such comparisons make sense.

The downside of generalizability is that we lose precision. Learners are less able to calibrate their answers. Analyzing the results becomes less meaningful. Also, learners see the learner-survey process as less valuable when questions are generic, so they give less energy and thought to answering the questions, and our data become less valuable and more biased.

Here is a question I developed for my own workshop (on how to create better smile sheets, by the way SMILE):

How READY are you TO WRITE QUESTIONS for a Performance-Focused Smile Sheet?

CIRCLE ONE OR MORE ANSWERS

AND/OR WRITE YOUR REASONING BELOW

A. I’m STILL NOT SURE WHERE TO BEGIN.

B. I KNOW ENOUGH TO GET STARTED.

C. I CAN TELL A GOOD QUESTION FROM A BAD ONE.

D. I CAN WRITE MY OWN QUESTIONS, but I’d LIKE to get SOME FEEDBACK before using them.

E. I CAN WRITE MY OWN QUESTIONS, and I’m CONFIDENT they will be reasonably WELL DESIGNED.

More
Thoughts?

 

 

Note several things about this question. First to restate. It is infinitely more tailored than a generic question could be. It encourages more thoughtful responding and creates more meaningful feedback.

Second, you might wonder why all the CAPS! I advocate CAPS because (1) CAPS have been shown in research to slow reading speed. Too often, our learners burn through our smile-sheet questions. Anything we can do to make them attend more fully is worth trying. Also, (2) respondents often read the full question and then skim back over it when determining how to respond. I want them to have an easy way to parse the options. Full disclosure. To my knowledge, all CAPS has not been studied yet for smile sheets. At this point, my advocacy for all CAPS is based on my intuition about how people process smile-sheet questions. If you’d like to work with me to test this in a scientifically rigorous fashion, please contact me.

Third, notice the opportunity for learners to write clarifying comments. Open-ended questions, though not easily quantifiable, can be the most important questions on smile sheets. They can provide intimate granularity — a real sense of the respondents’ perceptions. In these questions, we’re using a hybrid format, a forced choice question followed by an open-ended opportunity for clarification. This not only enables the benefits of open-ended responding, but it also enables us to get clarifying meaning. In addition, in some way it provides a reality-check on our question design. If we notice folks responding in ways that aren’t afforded in the answer choices given, we can improve our question for later versions.

 

Question Example — Simplifying The Wording

In writing smile-sheet questions, there’s another tradeoff to consider. More words add more precision, but fewer words add readability and motivation to engage the question fully. In the book, I talk about what I once called, “The World’s Best Smile Sheet Question.” I liked it partly because the answer choices were more precise than a Likert-like scale. It did have one drawback; it used a lot of words. For some audiences this might be fine, but for others it might be entirely inappropriate.

Recently, in working with a company to improve their smile sheet, a first draft included the so-called World’s Best Smile Sheet Question. But they were thinking of piloting the new smile sheet for a course to teach basic electronics to facilities professionals. Given the topic and audience, I recommended a simpler version:

How able will you be to put what you’ve learned into practice on the job?  Choose one.

A. I am NOT AT ALL ready to use the skills taught.
B. I need MORE GUIDANCE to be GOOD at using these skills
C. I need MORE EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.

This version nicely balances precision with word count.

 

Question Example — Dealing with the Sticky Problem of “Motivation”

In the book, I advocate a fairly straightforward question asking learners about their motivation to apply what they’ve learned. In many organizations — in many organizational cultures — this will work fine. However in others, our trainers may be put off by this. They’ll say, “Hey, I can’t control people’s motivations.” They’re right, of course. They can’t control learners’ motivations, but they can influence them. Still, it’s critical to realize that motivation is a multidimensional concept. When we speak of motivation, we could be talking simply about a tendency to take action. We could be talking about how inspired learners are, or how much they believe in the value of the concepts, or how much self-efficacy they might have. It’s okay to ask about motivation in general, but you might generate clearer data if you ask about one of the sub-factors that comprise motivation.

Here is a question I developed recently for my Smile-Sheet Workshop:

How motivated are you to IMPLEMENT PERFORMANCE-FOCUSED SMILE SHEETS in your organization?

CIRCLE ONLY ONE ANSWER. ONLY ONE!

A. I’m NOT INTERESTED IN WORKING TOWARD IMPLEMENTING this.

B. I will confer with my colleagues to SEE IF THERE IS INTEREST.

C. I WILL ADVOCATE FOR performance-focused smile sheet questions.

D. I WILL VIGOROUSLY CHAMPION performance-focused smile sheet questions.

E. Because I HAVE AUTHORITY, I WILL MAKE THIS HAPPEN.

More
Thoughts?

In this question, I’m focusing on people’s predilection to act. Here I’ve circumnavigated any issues in asking learners to divulge their internal motivational state, and instead I’ve focused the question on the likelihood that they will utilize their newly-learned knowledge in developing, deploying, and championing performance-focused smile sheets.

 

Final Word

It’s been humbling to work on smile sheet improvements over many years. My earlier mistakes are still visible in the digital files on my hard drive. I take solace in making incremental improvements — and in knowing that the old way of creating smile-sheet questions is simply no good at all, as it provides us with perversely-irrelevant information.

As an industry — and the learning industry is critically important to the world — we really need to work on our learning evaluations. Smile sheets are just one tool in this. Unfortunately, poorly constructed smile sheets have become our go-to tool, and they have led us astray for decades.

I hope you find value in my book (SmileSheets.com). More importantly, I hope you’ll participate along with some of the world’s best-run companies and organizations in developing improved smile sheet questions. Again, please email me with your questions, your question improvements, and alternatively, with examples of poorly-crafted questions as well.

Join The Debunker Club in Gathering Intel on Neuroscience-and-Learning

,

Neuroscience and Learning

The Debunker Club, formed to fight myths and misconceptions in the learning field, is currently seeking public comment on the possibility that so-called neuroscience-based recommendations for learning and education are premature, untenable, or invalid.

 

Click here to comment or review the public comments made so far…

 

Click here to join The Debunker Club…

 

Seeking CLO’s, Training and eLearning Managers, and Talent Development Executives to Interview

,

Send me your tired, your salaried, your harried Chief Learning Officers (and other Talent Development Executives) so that I can collect data on the state of our industry — focusing on a practical science-of-learning perspective.

Bigstock-Large-Group-of-Business-People-62228402

Areas Covered in this Research Effort:

  • what is your organization asking of the learning function?
  • how are your results measured? what matters to management?
  • what are the key struggles for your learning units?
  • research utilization — how utilized? who utilizes?
  • current strengths of learning professionals.
  • development needs of learning professionals
  • sources of knowledge for these learning professionals.
  • how does your organization deal with learning myths?
  • how do you separate good information from bad?
  • what vendors provide valuable research/data?
  • what vendors provide training that is science-based?
  • what do your learning units need to be most effective?

Organizations who contribute will be entitled to the full report. An executive summary will be available for free to others.

Send me names and contact information to info@work-learning.com.

 

Seeking Organizations for Research on Improved Smile Sheets

,

Now that my book publishing responsibilities are out of the way, I'm ready to begin a research effort designed to find out more ways to improve the effectiveness of our smile sheets.

If you're organization would like to participate in this effort — and be among the first to see the results of the research — check out the full announcement on my book's website: http://smilesheets.com/2016/05/18/3705/.