Getting Better Responses on Your Smile Sheets.

,

One of the most common questions I get when I speak about the Performance-Focused Smile-Sheet approach (see the book’s website at SmileSheets.com) is “What can be done to get higher response rates from my smile sheets?”

Of course, people also refer to smile sheets as evals, level 1’s, happy sheets, hot or warm evaluation, response forms, reaction forms, etc. They also refer to both paper-and-pencil forms and online surveys. Indeed, as smile sheets go online, more and more people are finding that online surveys get a much lower response rate than in-classroom paper surveys.

Before I give you my list for how to get a higher response rate, let me blow this up a bit. The thing is, while we want high response rates, there’s something much more important than response rates. We also want response relevance and precision. We want the questions to relate to learning effectiveness, not just learning reputation and learner satisfaction. We also want the learners to be able to answer the questions knowledgeably and give our questions their full attention.

If we have bad questions — one’s that use Likert-like or numeric scales for example — it won’t matter that we have high response rates. In this post, I’m NOT going to focus on how to write better questions. Instead, I’m just tackling how we can motivate our learners to give our questions more of their full attention, thus increasing the precision of their responding while also increasing our response rates as well.

How to get Better Responses and Higher Response Rates

  1. Ask with enthusiasm, while also explaining the benefits.
  2. Have a trusted person make the request (often an instructor who our learners have bonded with).
  3. Mention the coming smile sheet early in the learning (and more than once) so that learners know it is an integral part of the learning, not just an add-on.
  4. While mentioning the smile sheet, let folks know what you’ve learned from previous smile sheets and what you’ve changed based on the feedback.
  5. Tell learners what you’ll do with the data, and how you’ll let them know the results of their feedback.
  6. Highlight the benefits to the instructor, to the instructional designers, and to the organization. Those who ask can mention how they’ve benefited in the past from smile sheet results.
  7. Acknowledge the effort that they — your learners — will be making, maybe even commiserating with them that you know how hard it can be to give their full attention when it’s the end of the day or when they are back to work.
  8. Put the time devoted to the survey in perspective, for example, “We spent 7 hours today in learning, that’s 420 minutes, and now we’re asking you for 10 more minutes.”
  9. Ensure your learners that the data will be confidential, that the data is aggregated so that an individual’s responses are never shared.
  10. Let your learners know the percentage of people like them who typically complete the survey (caveat: if it’s relatively high).
  11. Use more distinctive answer choices. Avoid Likert-like answer choices and numerical scales — because learners instinctively know they aren’t that useful.
  12. Ask more meaningful questions. Use questions that learners can answer with confidence. Ask questions that focus on meaningful information. Avoid obviously biased questions — as these may alienate your learners.

How to get Better Responses and Higher Response Rates on DELAYED SMILE SHEETS

Sometimes, we’ll want to survey our learners well after a learning event, for example three to five weeks later. Delayed smile sheets are perfectly positioned to find out more about how the learning is relevant to the actual work or to our learners’ post-learning application efforts. Unfortunately, prompting action — that is getting learners to engage our delayed smile sheets — can be particularly difficult when asking for this favor well after learning. Still, there are some things we can do — in addition to the list above — that can make a difference.

  1. Tell learners what you learned from the end-of-learning smile sheet they previously completed.
  2. Ask the instructor who bonded with them to send the request (instead of an unknown person from the learning unit).
  3. Send multiple requests, preferably using a mechanism that only sends these requests to those who still need to complete the survey.
  4. Have the course officially end sometime AFTER the delayed smile sheet is completed, even if that is largely just a perception. Note that multiple-event learning experiences lend themselves to this approach, whereas single-event learning experiences do not.
  5. Share with your learners a small portion of the preliminary data from the delayed smile sheet. “Already, 46% of your fellow learners have completed the survey, with some intriguing tentative results. Indeed, it looks like the most relevant topic as rated by your fellow learners is… and the least relevant is…”
  6. Share with them the names or job titles of some of the people who have completed the survey already.
  7. Share with them the percentage of folks from his/her unit who have responded already or share a comparison across units.

What about INCENTIVES?

When I ask audiences for their ideas for improving responses and increasing response rates, they often mention some sort of incentive, usually based on some sort of lottery or raffle. “If you complete the survey, your name will be submitted to have chance to win the latest tech gadget, a book, time off, lunch with an executive, etc.”

I’m a skeptic. I’m open to being wrong, but I’m still skeptical about the cost/benefit calculation. Certainly for some audiences an incentive will increase rates of completion. Also, for some audiences, the harms that come with incentives may be worth it.

What harms you might ask? When we provide an external incentive, we might be sending a message to some learners that we know the task has no redeeming value or is tedious or difficult. People who see their own motivation as caused by the external incentive are potentially less likely to seriously engage our questions, producing bad data. We’re also not just having an effect on the current smile sheet. When we incentivize people today, they may be less willing next time to engage in answering our questions. They may also be pushed into believing that smile sheets are difficult, worthless, or worse.

Ideally, we’d like our learners to want to provide us with data, to see answering our questions as a worthy and helpful exercise, one that is valuable to them, to us, and to our organization. Incentives push against this vision.

 

Doing Research On Our Learning Products

,

The learning profession has been blessed in recent years with a steady stream of scientific research that points to practical recommendations for designers of learning. If you or your organization are NOT hooked into the learning research, find yourself a research translator to help you! Call me, for example!

That’s the good news, but I have bad news for you too. In the old days, it wasn’t hard to create a competitive advantage for your company by staying abreast of the research and using it to design your learning products and services. Pretty soon, that won’t be enough. As the research becomes more widely known, you’ll have to do more to get a competitive advantage. Vendors especially will have to differentiate their products — NOT just by basing them on the research — but also by conducting research (A-B testing at a minimum) on your own products.

I know of at least a few companies right now who are conducting research on their own products. They aren’t advertising their research, because they want to get a jumpstart on the competition. But eventually, they’ll begin sharing what they’ve done.

Do you need an example of a company who’s had their product tested? Check out this page. Scroll down to the bottom and look at the 20 or so research studies that have been done using the product. Looks pretty impressive right?

To summarize, there are at least five benefits to doing research on your own products:

  1. Gain a competitive advantage by learning to make your product better.
  2. Gain a competitive advantage by supporting a high-quality brand image.
  3. Gain a competitive advantage by enabling the creation of unique and potent content marketing.
  4. Gain a competitive advantage by supporting creativity and innovation within your team.
  5. Gain a competitive advantage by creating an engaging and learning-oriented team environment.

Definition of MicroLearning

, , ,

I’ve looked for a good definition of microlearning, but because I couldn’t find one, I’ve created my own.

Microlearning involves the use of:

“Relatively short engagements in learning-related activities—typically ranging from a few seconds up to 20 minutes (or up to an hour in some cases)—that may provide any combination of content presentation, review, practice, reflection, behavioral prompting, performance support, goal reminding, persuasive messaging, task assignments, social interaction, diagnosis, coaching, management interaction, or other learning-related methodologies.”

Microlearning has five utilization cases:

  1. Course Replacement
    Provides training content and learning support, often as a replacement for classroom training or long-form elearning.
  2. Course Augmentation
    Provides after-course or within-course streams of short learning interactions to reinforce, strengthen, or deepen learning.
  3. Retrieval Support
    Provides retrieval practice, spaced repetitions, and reminding to ensure knowledge and skills can be remembered when needed.
  4. Just-In-Time (Moment-of-Need) Learning
    Provides information when learners need it to perform a task they are working on.
  5. Behavioral Prompts
    Provides action nudges, task assignments, or performance support to directly prompt and support behavior.

If it’s not obvious, there are clearly overlaps in these five use cases, and furthermore, a single microlearning thread may utilize more than one of the methodologies suggested. For example, when using microlearning as a replacement for a standard elearning course, you might also consider retrieval support and behavioral prompts in your full learning design.

Will’s History of Spacing Out on Learning

,

As you know, if you’ve dabbled into my work for a few years, I’ve closely followed the research finding, “The Spacing Effect,” both from a research perspective and a practical perspective. Indeed, I was one of the first in the workplace learning field to recognize its practical significance, which I wrote about as early as 2002. In 2006 I published the research-to-practice report entitled Spacing Learning Over Time, which should — if there was justice in the world (LOL) or viable trade organizations (OUCH) — be enshrined in the Learning and Development Hall of Fame. Snickers are welcome. Taza Chocolate even better. A few years ago, still wanting to advocate for the practical use of the spacing effect, I began speaking about Subscription Learning at conferences and I developed a website (SubscriptionLearning.com) to encourage folks in the learning field to utilize the spacing effect in their learning designs. SubscriptionLearning.com is being retired in 2017, as it is no longer needed. Blog posts from the website are incorporated in this blog.

I am grateful to the enlightened organizations who have supported my work over the years and specifically to the individuals who continue to encourage the reading of the 2006 research report. Feel free to share yourself.

Now in 2017, I am grateful to another organization, Learning Technologies (in the UK) who is sponsoring me to speak on the spacing effect at their conference starting in a few weeks. As part of my efforts, I am developing a new presentation and I am updating my research compilation on the spacing effect. Stay tuned to this blog as I’m likely to share a few of my findings as I dig into the research.

Indeed, the research on spacing is some of the most interesting I’ve studied over the years. The first thing that fascinates is that there is so much damn research on the spacing effect, also referred to as spaced learning, distributed practice, and interleaving. In 1992, Bruce and Bahrick counted up the number of scientific studies on spacing and found over 300 articles at that time. Every year, there are more and more scientific articles published on spacing. By my rough count of journal articles cited on PsycINFO (a primary social-science database), over the last three years there have been 31 new articles published on the spacing effect (7 in 2014, 14 in 2015, and 10 in 2016).

One of the main reasons that so many research articles are published on the spacing effect is that the phenomenon is so intriguing. Why would spacing repetitions over time produce so much more remembering than giving the learners the exact same repetitions but simply massing them all at once or spacing them with less time in between? Freakin’ fascinating! So researchers keep digging into the complexities.

Harry Bahrick and Lynda Hall announced in 2005 that, “The spacing effect is one of the oldest and best documented phenomena in the history of learning and memory research.” And, just last year in a scientific review article, Geoffrey Maddox wrote, “Because of its robustness, the spacing effect has the potential to be applied across a variety of contexts as a way of improving learning and memory.”

Stay tuned, as I hope to be be spacing my research compilations over time…

Research

Bahrick, H. P., & Hall, L. K. (2005). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52, 566-577.

Maddox, G. B. (2016). Understanding the underlying mechanism of the spacing effect in verbal learning: A case for encoding variability and study-phase retrieval. Journal of Cognitive Psychology, 28(6), 684-706.

Thalheimer, W. (2006, February). Spacing Learning Events Over Time: What the Research Says. Available at: http://work-learning.com/catalog.html.

Stop the Madness — Animated Screen Writing in eLearning

, ,

Is anyone else getting completely annoyed watching someone’s hand draw and write on videos and elearning?

OMG! It’s beginning to drive me nuts! What the hell is wrong with us?

Here’s the thing. When this was new, it was engaging. Now it’s cliche! Now most people are habituated to it. What we’re doing now is taking one of our tools and completely overusing it.

Let’s be smarter.

Testing for Instructional Designers — A Common Mistake

, , ,

Somebody sent me a link to a YouTube video today — a video created to explain to laypeople what instructional design is. Most of it was reasonable, until it gave the following example, narrated as follows:

“… and testing is created to clear up confusion and make sure learners got it right.”


Something is obviously wrong here — something an instructional designer ought to know. What is it?

Scroll down for the answer…

Before you scroll down, come up with your own answer…

.

.

.

.

.

.

.

.

.

.

.

.

Answer: 

The test question is devoid of real-world context. Instead of asking a text-based question, we could provide an image and ask them to point to the access panel.

Better yet, we could have them work on a simulated real-world task and follow steps that would enable them to complete the simulated task only if they used the access panel as part of their task completion.

Better yet, we could have them work on an actual real-world task… et cetera…

Better yet, we might first ask ourselves whether anybody really needs to “LEARN” where the access panel is — or would they just find it on their own without being trained or tested on it?

Better yet, we might first ask ourselves whether we really need a course in the first place. Maybe we’d be better off to create a performance-support tool that would take them through troubleshooting steps — with zero or very little training required.

Better yet, we might first ask ourselves whether we could design our equipment so that technicians don’t need training or performance support.

.

.

.

Or we could ask ourselves existential questions about the meaning and potency of instructional design, about whether a career devoted to helping people learn work skills is worthy to be our life’s work…

Or we could just get back to work and crank out that test…

SMILE…

 

 

Improving Our Performance in Watching the Presidential Debates

, ,

I work in the learning-and-performance field. I watched the first two presidential debates. They were problematic to say the least. So, I thought to myself, “Hey Will, you’re a learning-and-performance professional. What might be done to improve them?” Actually, it was more like, “Damn, somebody’s got to make these debates better. They’re not creating the best outcomes. They’re not really educating us on the important issues for the presidential election. Our debates are not helping the democratic process. Indeed, they’re probably creating harm.”

24564574914_0cdd268f92_zPicture above by Donkey Hotey 2016 Creative Commons on Flickr

In looking at the debate process, there are several leverage points, including the following:

  1. The debate format.
  2. The questions asked.
  3. The candidates’ responses.
  4. The citizenry’s cognitive processing of the debate.
  5. The news media’s and social media’s messaging about the debate.
  6. The citizenry’s cognitive processing of the media messaging.

For most of these, we — the citizenry — have very little direct leverage to influence the process. Together, we can influence the social-media messages, but as individuals, we have very little influence there. Tears in the ocean.

But certainly the media and the candidates are acting with intentionality to influence the citizenry. Candidates act bold and confident because they know that people, in general, are suckers for those who display confidence. Candidates use certain words to influence — even if those words are too general to be meaningful (words like freedom, equality, strength, diversity). Candidates avoid talking about complicated issues, because they think we can’t understand. Candidates speak in terms of black and white, good and bad, either this or that — though the world is shaded in sepia tones — because they think we need certainty.

They act because they think we as citizens will react in predictable ways. But what if we changed and improved our responses to the debates? What if we as citizens improved our cognitive processing and instead of having knee-jerk reactions, we improved our thinking? What if instead of being persuaded by irrelevancies, we were persuaded by a clear understanding of the issues? Is there anything we can do to improve and deepen our thinking about the debates?

Yes! We can do a better job! Not a perfect job, not a brilliant job, but we can improve and deepen our thinking if we take some time to think about what we care about and what matters.

I’m sure my efforts here will be inadequate, but I offer a one-page checklist to spur your thinking. Take a look, try it out at Wednesday’s final Presidential debate and let me know how you’d improve it. Tell me what works and what doesn’t. Better yet, let me know what else we can do to improve the results of the debates.

 

Try the Presidential Debates Insight Checklist

 

Read My Article on LinkedIn

 

 

WHAT ELSE? WHAT DO YOU THINK?

Okay, I really feel like this effort is too inconsequential and too unlikely to make a difference, so I’d love to hear your ideas:

  • What’s wrong with this approach?
  • What could make it better?
  • What else could be done to improve the outcome of the debates — that is, creating a better informed citizenry.
  • Are there other tools in our learning-and-performance toolbox we might use?
  • What could make our debates great again? (sorry, couldn’t resist)

It’s Behavior Change Stupid! Transfer that!

,

Those of us in the learning professions are naturally enamored with the power of learning. This is all fine and good–learning is necessary for human survival and for our most enlightened achievements–but too narrow a focus on learning misses a key responsibility. Indeed, learning without sustained behavior change is like feeding a man who's planning to jump off a bridge. Nice, but largely besides the point. 

The bottom line is that we learning professionals must not only look to the science of learning, but also the science of behavior change.

My friend and colleague Julie Dirksen has been thinking about behavior change for years. Here's a recent article she wrote:

Here is another recent resource on behavior change:

It's good to keep this all in perspective. Science often moves slowly and in fits and starts. There is great promise in the many and varied research areas under study. You can see this most fully in the health-behavior-change field. There's a ton of research being done. Here's a quick list of research being done on behavior change:

  • Cardiac health
  • Obesity
  • Clean cooking
  • Asthma
  • HIV and STD prevention
  • College-student drinking
  • Cancer prevention
  • Child survival
  • Recycling
  • Use of hotel towels
  • Young-driver distraction
  • Hand washing
  • Encourage walking and cycling
  • Smoking cessation
  • Healthy pregnancy behaviors
  • Promoting physical activity

Okay, the list is almost endless.

One of the findings is not surprising. Lasting behavior change is very difficult. Think how hard it is to lose weight and keep it off, or stop an internet addition. So it's great that researchers are looking into this.

In the learning field, we have our own version of behavior-change research. It's called transfer. We've already learned a lot about how to get people to transfer what they've learned back to their jobs or into their lives. We're not done learning, of course.

One thing we do know is that training by itself is rarely sufficient to produce lasting change. Sometimes our learners will take what they've learned, put it immediately into practice, deepen their own learning and continue to learn and engage and use what they've learned over time. Too often, they forget, they get distracted, they get no support.

 

Summary

My four messages to you are these:

  1. Keep your eyes open for Behavior Change Research.
  2. Keep your eyes open for Transfer-of-Learning Research.
  3. Don't be a fool in thinking that training/learning is enough.
  4. We learning professionals have a responsibility to enable usable behavior change.

 

Personalized Learning with Video… by Consensus.

, ,

Research shows that one-on-one tutoring is generally highly effective — with a good tutor, of course. Similarly, John Anderson — the cognitive psychologist — along with others — developed Intelligent tutoring systems (ITS) that tailor content to learners based on cognitive models of what they know and don't know.

These learning approaches are beneficial because they personalize learning based on a diagnosis of learner knowledge. This is not a new concept. Indeed, the Socratic Method also took learners on a journey based on their responses. B. F. Skinner's operant conditioning provided reinforcement based on learner actions. Skinner's Programmed Learning and Fred Keller's Personalized System of Instruction are practical applications based on operant conditioning principles.

We've known for millennium that personalized learning is good — and we've even dabbled in scalable implementations like programmed learning — but for the most part we are still awaiting the promise of such personalization.

Now may be the time. New technologies are beginning to show promise. For example, subscription-learning threads based on personalized spacing schedules personalize learning nuggets based on learner responses. Still, one area where personalization hasn't been much in evidence is video. That may be changing.

Consensus Demo

Consensus allows its users to create videos tailored to each learner. Their focus is on sales and marketing — and particularly on creating demos — but the technology could be used for other needs as well. Consensus tailors content to users based on their interests, background, etc. When folks choose a topic as “Very Important,” users get the full video segment and get it first. When folks choose a topic as “Somewhat Important,” users get a summary video. When they choose “Not Important,” users don’t get any segments associated with those topics.

Consensus can also segment users based on their role in an organization, based on the size of their company, or based on other demographics as well.

They’ve got a great video that you can tailor to your needs. Worth watching! Click to check out Consensus.

 

Learning Executive: Using Credentialing to Accelerate Change!

,

In my ongoing research interviewing learning executives, I occasionally come across stories or ideas that just can't wait until the full set of data is collected.

This week, I interviewed a director of employee development and training at a mid-sized distribution company. She expressed many of the frustrations I've heard before in my consulting work with L&D (Learning and Development) leaders. For example:

  • Lack of good learning measurement, causing poor feedback to L&D stakeholders.
  • Too many task requirements to allow for strategic thinking in L&D.
  • While some SME's are great trainers, too many deliver poorly-designed sessions.
  • Lack of some sort of competency testing of learners.
  • Lack of follow-through after training, limiting likelihood of successful application to the job.

There were so many changes to make that it appeared overwhelming — as if making positive change was going to take forever.

Then she got an idea. Her organization had begun to train its customers (in addition to training its employees), and they began to search for ways to demonstrate the value and credibility of the customer-focused courses.

What this director realized was that accreditation might serve multiple purposes — if it provided a rigorous evaluation scheme; one that demanded living up to certain standards.

She found an accrediting agency that fit the bill. IACET, the International Association of Continuing Education and Training would certify her organization, but the organization would have to prove that it engaged in certain practices.

This turned out to be a game changer. The requirements, more often than not, propelled her organization in directions she had hoped they would travel anyway. The accreditation process had become a powerful lever in the director's change-management efforts.

Some of things that the accreditation required:

  • The L&D organization had to demonstrate training needs, not just take orders for courses.
  • They had to map learning evaluations back to learning objectives, ensuring relevance in evaluations.
  • They had to have objectives that tied into learning outcomes for each course.
  • Trainers had to be certified in training skills (aligned to research-based best practices).
  • Trainers had to be regularly trained to maintain their certifications.
  • Et cetera…

While before it was difficult for her to get some of her SME's to take instructional design seriously, now accreditation constraints propelled them in the right direction. Whereas before, SME's balked at creating tests of competence, now the accreditation requirements demanded compliance. Whereas before, her SME's could skip out on training on evidence-based learning practice, now they were compelled to take it seriously — otherwise they may lose their accreditation; thus losing the differentiation their training provides to customers.

The accreditation process was a catalyst, but it wouldn't work on it's own — and it's not a panacea. The director acknowledges that a full and long-term change management effort is required, but accreditation has helped her move the needle toward better learning practices.