I know I'm going completely against most training-industry practice in saying this, but it's the truth. Likert-like scales create poor data on smile sheets.

If you're using questions on your smile sheets with answer choices such as:

  • Strongly Agree
  • Agree
  • Neither Agree Nor Disagree
  • Disagree
  • Strongly Disagree

You're getting data that isn't that useful. Such questions will create
data that your stakeholders–and you too–won't be able to decipher very
well. What does it mean if we average a 4.2 rating? It may sound good,
but it doesn't give your learners, your stakeholders, or your team much
information to decide what to do.

Moreover, let's remember that our learners are making decisions with every smile-sheet question they answer. It's a lot tougher to decide between "Strongly Agree" and "Agree" than between two more-concrete answer choices. 

Sharon Shrock
and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion-Referenced Test Development,
offer the following wisdom: On using Likert-type
Descriptive Scales (of the kind that use response words such as “Agree,”
“Strongly Agree,” etc.):

“…the
resulting scale is deficient in that the [response words] are open to many
interpretations.”
(p. 188)

So why do so many surveys use Likert-like scales? Answer: It's easy, it's tradition, and surveys have psychometric advantages often because they are repeating the same concepts in multiple items and they are looking to compare one category to another category of response.

Smile sheets are different. On our smile sheets, we want the learners to be able to make good decisions, and we want to send clear messages about what they have decided. Anything that fuzzes that up, hurts the validity of the smile-sheet data.

When asked for a simple heuristic in how to use the spacing effect–the
finding that repetitions spaced in time are more effective in
supporting remembering than repetitions squished narrowly in time–I've
often told people that the ideal spacing interval is one that will equal
the retention interval one desires. If you want you learners to
remember for a month, give them one-month spaced repetitions. If you
can't do that, longer is better, and there seems to be something magical
about repeating something overnight.

But new research suggests that even short spacings of only
half-a-minute or so can have lasting benefits over non-spaced
repetitions.

Citation:

Rawson, K. A., & Dunlosky, J. (2012). Relearning Attenuates the Benefits and Costs of Spacing. Journal of Experimental Psychology: General. Advance online publication.

Subscription Learning Defined

Subscription learning, as its name implies, provides an intermittent stream of learning-related interactions to those who are subscribed. These learning-related interactions–called “nuggets”–can involve a great variety of learning-related events, including content presentation, diagnostics, scenario-based questions, job aids, reflection questions, assignments, discussions, etc.

Nuggets are short, usually presented in less than five minutes, delivered to recipients using push technology; either through email, text messaging, cell-phone alerts, desktop notifications, or some other form of prompting. Nuggets are intentionally scheduled over time to support learning, often utilizing research-based findings related to the spacing effect. Learners subscribe (or are subscribed) to one or more series of learning nuggets, called “threads.”

Learning threads can be predesigned, creating nuggets based on anticipated learner needs or they can be dynamically created based on learner performance. Threads can be an alternate way to present course-like information, they can augment other learning interventions, or they can provide an alternative to traditional learning approaches. In addition, subscription learning can go beyond learning delivery and be designed to create open communication channels, directly prompt performance, or provide hybrid experiences.

Visual Representation

Let’s compare a 90-minute elearning course to a commensurate subscription-learning thread. Typically, learners will engage a 90-minute elearning program in one sitting on one day, never to engage the learning material again. A standard e-learning course could be pictured as one big block of learning, whereas the same content in a subscription-learning thread could be presented as 18 five-minute learning nuggets.

Given the time-based reality that subscription-learning nuggets are spread over time, it’s probably best to present nuggets as typically distributed on a calendar.

Special Capabilities of Subscription Learning

Subscription learning provides affordances that standard elearning typically does not provide. Here’s a short list:

  • Relatively long spaced-repetition intervals to maximize remembering
  • Engagement in the learners’ work context over time
  • Opportunities for learners to practice real-world tasks over time
  • Increased likelihood that learning will be discussed with coworkers
  • Multiple learning contexts in terms of cognitive frames of mind
  • Heightened attention to learning content with fresh engagements
  • Potential to prompt on-the-job actions with multiple prompts
  • Potential to engage in social-media conversations that support learning
  • Potential to engage with senior leaders or other organizational experts
  • Potential to modify content in response to learning dynamics
  • Potential to modify content in response to organizational changes

Research-Based Benefits

From a learning standpoint, subscription learning has several advantages over typical training. First, it keeps learning fresh in the minds of learners. Forgetting is a natural cognitive process, but it is the scourge of most training programs. By utilizing a thread of short nuggets, key concepts stay accessible from within long-term memory. Second, subscription learning can be designed to utilize the spacing effect—one of learning science’s most studied phenomenon. The spacing effect is the research finding that repetitions of concepts spaced over time are a more potent way to support long-term memory than repetitions that are not spaced over time (or are not spaced as widely in time). Interestingly, while the spacing effect is one of the most studied learning factors it is also one of the most underutilized in training and education.

Note that while subscription learning is the learning approach most amenable to the spacing effect, subscription learning need not utilize the spacing effect. When a learning thread presents new content in each nugget–and never utilizes the power of repetition–we still have subscription learning, but we don’t have the spacing effect.

Macro vs. Micro Spacings

While immersing myself in the research and attempting to bridge the gap between the research and practical learning situations, I realized that a new terminology was needed. Spacings can be relatively long, say providing a content repetition after four days; or relatively short, for example providing content repetitions after a minute or two. In practice, both can work very effectively—but their relevance depends on factors like the topic, the learners’ knowledge level, importance, etc.

Micro spacings are utilized when learners have to learn many separate bits of knowledge. For example, in learning a language, learners are trying to learn tons of vocabulary—so waiting a week between content presentations would be too slow. In such a case, micro spacings present many learning opportunities within a short time frame. Micro spacings are particularly valuable for things like language learning, vocabulary, multiplication tables, or any learning situation where there is a lot of learning information—or there is a short time horizon for learning a smaller set of information.

Macro spacings are utilized when learners only have to learn a short list of learning points (say less than 20)—or when there is no urgency in learning and repetitions can be leisurely spread over time.

Of course, we shouldn’t forget the findings of the learning researchers. In general, the longer the spacing interval, the longer the information will be remembered. Indeed, micro spacings often require a hybrid spacing scheme, using multiple micro spacings in one sitting, but then utilizing macro spacing to ensure long-term remembering.

The diagram below is my attempt to show how micro spacings differ from macro-spacings—and how they can be combined.

 

(Subscription Learning—using elearning to send learners short nuggets of learning-interactions spread out over time—is a relatively new phenomenon. As such, it is subject to the same learning curve that all new technologies entail. As a learning researcher/consultant who has been heavily pushing the idea of subscription learning in keynotes and articles recently, I’m fascinated by the human side of this burgeoning field. It’s thrilling to see new subscription-learning applications
enter the learning field, but it’s even more interesting to explore the human aspect of invention and technology-dissemination.

Eric Blumthal, who along with his business partner at count5, Gordon Eilen, set out in 2005 to create a subscription-learning platform to help sales reps better use their valuable time. They’ve created something even more broadly applicable, but their story is instructive. Eric, who had years and years of experience as a sales guy—a former VP of sales—saw a need. Sales people would be pulled away from selling to learn a ton of information, but they’d soon forget it. What Eric saw was a broken model—and a waste of time and productivity—and a loss of sales! He thought—with little or no background in the science of learning—that what was needed was a “reinforcement strategy.” When he surveyed sales organizations, hardly any had a way to reinforce what was learned. They were all leaking knowledge!

What Eric and Gordon created—along with their team—was Q MINDshare, a subscription-learning reinforcement tool. But just like all innovations, Q MINDshare has morphed and improved over the years, now incorporating learning approaches that have proven to work with real employees. And they’ve expanded beyond sales people to provide value for customer service folks, field technicians, managers, supervisors, university students, and others.

Q MINDshare – How it Works

Q MINDshare’s authoring tool is designed for non-technical subject-matter experts. The tool provides many types of interactions, including presentations, questions, videos, job aids, documents, animations, etc. They also have scenario-based open-ended question responses and then ask the learners to grade themselves. Soon they will have the capability to let learners record what they are going to say. Nuggets can be tagged with variables—for example different learning goals—and information on the learner’s progress toward these variables can be displayed to learners and provided for work-learning professionals and stakeholders on the business side as well. As an example, a Fortune 20 client of count5 uses Q Mindshare with about a dozen variables tied to the learning objectives of their training program.

From the learner’s perspective, they’ll get a notification on their computer, or phone, or tablet, which prompts them to engage in a short nugget of learning interaction.

On a computer the notification looks like this on the bottom right of the screen:

On a smartphone or tablet, it looks like this:

While most people engage immediately, the system keeps track and feeds learners new nuggets only when they’re ready. Learners typically get some sort of question to test their understanding, and then several weeks later they get a follow-up question on the same topic. The schedule of questions depends on how well the learners do the first time they answer the question.

What Eric and Gordon have Learned

Eric, Gordon, and company have learned a ton of stuff over the years. Eric first had the idea for a reinforcement tool way back in 2002 or 2003. They went to market with their system in 2005, and have been making refinements based on real-world application ever since. The following will describe some of what they learned.

Email Push Technology Doesn’t Work That Well

One of the most important things they’ve found is that people are so desensitized to emails and are so likely to ignore them that they had to invent a more potent push-technology delivery method. Specifically, they have found that they get 10-20% engagement with email pushes, but 90-95% engagement with their embedded push technology. They call this a “Cut-Thru™ technology” and refer to it as “The Q channel.”

Eric makes sure to preach the importance of using the Q channel ONLY for the really important stuff—to the organizations that count5 works with. For example, at one client, Q MINDshare is deployed to about 5,000 folks (soon to be about 25,000), but only about a dozen employees have privileges to send messages on the channel. These gatekeepers learn that keeping the channel clear of detritus is a key to keeping it valuable for employees. Indeed some companies have begun to use Q MINDshare for strategic messaging from their senior management—so that the messages get through.

The PC is still King

When people get notifications—pushed nuggets—through their PC’s, their engagement level with those nuggets is higher compared with those who get nuggets pushed to their smartphones and tablets. However, those who get nuggets pushed to both their PC and their smartphones show the highest engagement. Eric and I surmised that the PC is the most-work-focused device, whereas smartphones and tablets are often used for non-work activities. Of course, the technology landscape is always changing so the future may bring different patterns—and different user groups may have different profiles.

Prompting Attention

While there are a number of interactive methods in use through Q MINDshare, even simple presentations of information are paid more attention because the learners know that they might be asked a question about the information later.

System Design Can Lead to Continuous Improvement of the Learning

Learners can favorite any item of content in a learning interaction. They can also rate each item using a three-star scale combined with their comments. Both of these tools not only boost learner engagement, but they also help count5 learn what works best and what needs to be improved.

In many instances, when work-learning professionals review the results for one of their Q-channel threads, they often will spontaneously create follow-up learning or create job aids on information that the learners don’t seem to be getting. This is phenomenal! It’s a system that prompts us—as work-learning professionals—to get good feedback on what we’re doing so that we can build continuous cycles of improvement into our practice. Using feedback to improve our learning products is freakin’ revolutionary, but it should be common practice.

The following graphic shows one of the reports that are available. Note how high the participation rates are and how feedback and second chance opportunities lead to better proficiency.

Avoiding Chapterization Produces better Learning

As Eric and count5 have deployed subscription learning, they’ve learned that interleaving topics produces better results than teaching one topic at a time (and never coming back to reinforce them). In the learning field, we too often use such a “chapterization” strategy, when we should be using an interleaf strategy like that which count5 has found most effective. Interestingly, the research on learning suggests the same thing, showing once again that if you’re getting good feedback on real results of your learning interventions, you may find out what works without ever looking at the research. And of course, even if you’re following research-based practices (HIGHLY RECOMMENDED!) it’s still wise to verify the research recommendation with your own content and learners.

Augmenting Traditional Training and More

Originally, count5 started out to augment traditional training with follow-up reinforcement, but now Eric is pushing the concept of using Q MINDshare to provide stand-alone threads to convey learning content. His clients have also used it for prework as well.

Creating Significant Learning (and Financial) Benefits

A huge financial services company was going through the chaos of mergers and their salespeople were not selling enough title insurance. The company provided the usual PowerPoint-dense training program and followed that up with two typical elearning programs for each of their sales reps. But in addition, they also decided to try Q MINDshare as an after-training learning reinforcement device.

Not everyone at the company was completely sold on the idea, so they decided to do what the smartest companies do. They provided Q MINDshare reinforcement thread for half the sales people, but not for the other half. Then they examined the results. Their learning folks came up with a competency test that was given to all their sales folks at the end of the Q MINDshare trial. The good news for Eric and count5 was that the sales reps that used Q MINDshare performed better on the competency test by 40%—and 80% when the easiest questions were eliminated from the analysis. More remarkable were the actual sales results. Each Q MINDshare rep sold more than $200,000 more than the non-Q MINDshare reps! And this wasn’t just a few learners. There were over 200 reps in the comparison.

But even as these numbers were giving Eric thrills, there were still questions from within the sales organization. One day, as Eric was regaling the Senior VP of sales about the results, they began focusing on the biggest sale of all—a plus-million-dollar sale by Lisa, one of the best sales people in the organization. The VP decided to call her up and ask her to come immediately to his office. When she arrived—surprised if not befuddled by the sudden call to attention—the VP asked her straight-up, did Q MINDshare make the difference in her getting her million-plus deal. She said, with some humble gravitas, that she was the one responsible for the sale—that she had long been one of the most productive sales people. She also told the story of what happened.

When she’d gone to the three-day training and then taken the standard elearning courses, she didn’t feel very confident that she could sell title insurance competently to her clients. She even made a conscious decision to avoid emphasizing it with her best clients. But then, when she began to get the Q MINDshare nuggets, her sense of competence rose. Perhaps just as importantly, after about a month of engaging the Q thread, and specifically after she had engaged a Q nugget earlier one day, she was talking on the phone to one of her biggest clients and decided to broach the idea of title insurance. The rest is the history of a million-dollar sale. As you might imagine, Eric Blumthal—sitting there in the VP’s office listening to this story—was floating with delight, seeing that years of hard work and innovation had paid off big time.

Will’s Summary

Eric and Gordon of count5 are engaged in a cycle of innovation and creativity—specifically aimed at figuring out what works and what doesn’t in subscription learning. Their instincts about what might work have always been good, but now they are incorporating the latest research on learning (and spacing and feedback in particular) into their product designs. Q MINDshare is a potent subscription-learning product, worthy of our admiration. Eric and Gordon are years ahead of the field in many respects because they‘ve done the hard work of trial-observe-improve, they’ve done it time and time again, and their work shows the results.

 

 

 

This Sign-Up is specifically for those interested in Subscription Learning.

I promise NEVER to share your information.

                                                                       = Will Thalheimer

 







* Email
* First Name
* Last Name
Phone
Business
* Zip
* Country
* = Required Field

Below are the slides I promised.

 

Here is my mailing list sign up:


 

Slides:

Click here to download the slides from the October 15th session.

 

Thank you for your interest in my work!

= Will Thalheimer

NOTE: This was the first blog post on my SubscriptionLearning.com blog (all posts from that blog are now incorporated into this blog).

=======================

It was about 7 years ago when I was researching the spacing effect that I first got the idea for subscription learning. So, for the first post on the subscription-learning blog, I’m going to explain the “Spacing Effect” research finding.

The basic finding is this: When a learning point is repeated, it will create better remembering if it is repeated after a wider period of time than if it is repeated after a shorter period of time. For example, repeating something after four days will create higher levels of memory retrieval than repeating it after two days. Similarly, a repetition interval of one day will produce more remembering than a repetition interval of half a day. One hour will be better than half an hour. 10 minutes will be better than five minutes, and so forth.

The spacing effect is one of the best ways to support long-term remembering. Indeed, in my list of the 12 most important learning factors (the Decisive Dozen) I’ve found that spacing is one of three learning factors that is particularly potent in supporting remembering.

As Harry Bahrick and Linda Hall (Learning Researchers) put it in the Journal of Memory and Language:

“The spacing effect is one of the oldest and best documented phenomena
in the history of learning and memory research.”

Of course, as with any research on human beings, there are variations, contingencies, and boundary conditions on the spacing effect. Still, it is a very robust finding. And, it should be one of the most important tools in any learning professional’s toolbox.

I wrote a research-to-practice report on spacing back in 2006 and included over 100 research citations. And the research continues to accumulate year after year because the phenomenon is so fascinating. Today there would be dozens of additional studies that could be highlighted. I’ll provide a short list below.

Here is what is so fascinating: Why is it that providing the same exact information to learners, but in one case separating the repetitions more widely in time, will produce profound differences in how well people remember? Why are wider spacings so much more potent than more-narrowly-spaced repetitions?

Different reasons have been proposed by researchers:

  • Wider spacings require more cognitive effort.
  • Wider spacings produce more varied retrieval routes through memory.
  • Wider spacings promote more retrieval failure, and thus encourage better learning practices.

Let me be clear. Subscription learning–as defined–does not require the use of the spacing effect, but compared with other forms of learning, it is most suited to enabling the spacing effect.

Some Recent Research on The Spacing Effect

  • Mulligan, N. W., & Peterson, D. J. (2013). The Spacing Effect and Metacognitive Control. Journal of Experimental Psychology: Learning, Memory, and Cognition. Advance online publication.
  • Goossens, N. A. M. C., Camp, G., Verkoeijen, P. P. J.
    L., Tabbers, H. K., & Zwaan, R. A. (2012). Spreading the words: A
    spacing effect in vocabulary learning. Journal of Cognitive Psychology, 24(8), 965-971.
  • Logan, J. M., Castel, A. D., Haber, S., &
    Viehman, E. J. (2012). Metacognition and the spacing effect: The role of
    repetition, feedback, and instruction on judgments of learning for
    massed and spaced rehearsal. Metacognition and Learning, 7(3), 175-195.
  • Vlach, H. A., & Sandhofer, C. M. (2012).
    Distributing learning over time: The spacing effect in children’s
    acquisition and generalization of science concepts. Child Development, 83(4), 1137-1144.
  • Callan, D. E., & Schweighofer, N. (2010). Neural
    correlates of the spacing effect in explicit verbal semantic encoding
    support the deficient-processing theory. Human Brain Mapping, 31(4), 645-659.
  • Smith, T. A., & Kimball, D. R. (2010). Learning from feedback: Spacing and the delay–retention effect.
  • Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(1), 80-95.
  • Son, L. K. (2010). Metacognitive control and the spacing effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(1), 255-262.

This list will be updated periodically.

WILL’S NOTE:  Previously, I had given up trying to keep this list current because it was nearly impossible to research and catalog the “gazillions” of new providers entering this category. I’m going to make another effort to keep this list, but I need your help through the contact form below. THANKS!

 

To link to their websites, move your mouse above the name.

Subscription-Learning Authoring Tools — General Purpose

  • Ed (subscription and mobile-learning authoring tool)
  • KnowledgeGuru (subscription and game-based learning authoring tool)
  • Moblrn (subscription learning authoring tool)
  • Cameo (subscription-learning authoring tool, scenario-based questions, email, tracks learning)
  • Q MINDshare (subscription learning authoring tool)
  • Mindsetter (subscription-learning authoring tool)
  • Memrise (subscription-learning authoring tool, crowdsourcing, mnemonics, gamification)
  • Mindmarker (subscription learning authoring tool)
  • SimWriter (scenario-based simulation authoring tool)
  • Zunos (learning and knowledge-management authoring tool)
  • DailyBitsOf (subscription-learning authoring tool and course catalog)
  • Cerego (subscription-learning authoring tool and course catalog)
  • Trivie (subscription learning authoring tool focused on retrieval practice)
  • Flip Training (subscription-learning authoring tool)

Subscription-Learning Authoring Tools — Drill and Practice

Subscription-Learning Developers

  • Axonify (custom developer, utilizes spacing effect)
  • NexLearn (custom developer of simulations)
  • Wranx (subscription-learning provider)
  • GamEffective (gamified and performance-focused engagements with subscription learning).

Subscription-Learning Single-Focus Applications

Subscription-Learning Repositories of Courses

  • MyTools2Learn (subscription learning course repository)
  • Cerego (flashcard authoring tool and course catalog)
  • DailyBitsOf (subscription-learning authoring tool and course catalog)
  • Memrise (subscription-learning authoring tool, crowdsourcing, mnemonics, gamification)
  • Ed (subscription and mobile-learning authoring tool)
  • EdX (MOOC available as subscription and mobile learning)
  • Coursera (MOOC available as subscription and mobile learning)
  • Class Central (a repository and rating compilation of university courses)
  • https://gohighbrow.com/

Subscription-Learning After-Training Follow-up Tools

Microlearning — Without Threaded Content

Hello. This is Will Thalheimer! Here to advocate for and explore the idea of SUBSCRIPTION LEARNING.

 

What is Subscription Learning?

Subscription learning, as its name implies, provides an intermittent stream of learning-related interactions to those who are subscribed. These learning-related interactions–called “nuggets”–can involve a great variety of learning-related events, including content presentation, diagnostics, scenario-based questions, job aids, reflection questions, assignments, discussions, etc. Nuggets are short, usually presented in less than five minutes. Nuggets are intentionally scheduled over time to support learning, often utilizing research-based findings related to the spacing effect. Learners subscribe (or are subscribed) to one or more series of learning nuggets, called “threads.” Learning threads can be predesigned, selecting nuggets based on anticipated learner needs or they can be dynamically created based on learner performance.

 

Why is This Important?

When most of us think about learning or elearning we think in terms of providing learners with courses–large learning blocks that require learners to sit for an hour or two at a time. While courses have a long history and certainly provide benefits, we shouldn’t be hogtied by the traditional view. Indeed, the science of learning suggests that short learning events provided over time are likely to be an even more potent method. And, in today’s accelerated world, many of our learners don’t have time to devote to long periods of learning.

 

What this Website Will Attempt to Do

My hope is to generate converts to the subscription-learning approach, provide vetted information of value to learning professionals, and help everyone learn from our collective successes and mistakes. I’m also open to suggestions, so feel free to email me at will {dot} thalheimer {at} work-learning.com.