MOOC’s are Ineffective–Except for One Thing!

, ,

MOOC's don't have to suck. The 4% to 10% completion rates may be the most obvious problem, but too many MOOC's simply don't use good learning design. They don't give learners enough realistic practice, they don't set work in realistic contexts, they don't space repetitions over time.

But after reading this article from Thomas Friedman in the New York Times, you will see that there is one thing that MOOC's do really well. The get learning content to learners.

Really, go ahead. Read the article…

 

Why is "Exposure" one of the Decisive Dozen learning factors?

Many people have wondered why I included "Exposure" as one of the most important learning factors. Why would exposing learners to learning content rank as so important? Friedman's article makes it clear in one example, but there are billions of learners just waiting for the advantage of learning.

I got the idea of the importance of exposing learners to valid content by noticing in many a scientific experiment that learners in the control group often improved tremendously–even though they were almost always outclassed by those who were in the treatment groups.

By formalizing Exposure as one of the top 12 learning factors, we send the message that while learning design matters, giving learners valid content probably matters more.

And yes, that last sentence is as epically important as it sounds…

It also should give us learning experts a big dose of humility…

 

MOOC's will get better…

Most MOOC's aren't very well designed, but over time, they'll get better.

 

 

Learners’ Often Use Poor Learning Strategies — From a Research Review

,

I just read the following research article, and found a great mini-review of some essential research.

  • Hagemans, M. G., van der Meij, H., & de Jong, T. (2013). The effects of a concept map-based support tool on simulation-based inquiry learning. Journal of Educational Psychology, 105(1), 1-24. doi:10.1037/a0029433

Experiment-Specific Findings:

The article shows that simulations—the kind that ask learners to navigate through the simulation on their own—are more beneficial when learners are supported in their simulation playing. Specifically, they found that learners given the optimal learning route did better than those supplied with a sub-optimal learning route. They also found that concept maps helped the learners by supporting their comprehension. They also found that learners who got feedback on the correctness of their practice attempts were motivated to correct their errors and thus provided themselves with additional practice.

Researchers’ Review of Learners’ Poor Learning Strategies

The research Hagemans, van der Meij, and de Jong did is good, but what struck me as even more relevant for you as a learning professional is their mini review of research that shows that learners are NOT very good stewards of their own learning. Here is what their mini-review said (from Hagemans, van der Meij, and de Jong, 2013, p. 2:

  • Despite the importance of planning for learning, few students engage spontaneously in planning activities (Manlove & Lazonder, 2004).  
  • Novices are especially prone to failure to engage in planning prior to their efforts to learn (Zimmerman, 2002).  
  • When students do engage in planning their learning, they often experience difficulty in adequately performing the activities involved (de Jong & Van Joolingen, 1998; Quintana et al., 2004). For example, they do not thoroughly analyze the task or problem they need to solve (Chi, Feltovich, & Glaser, 1981; Veenman, Elshout, & Meijer, 1997) and tend to act immediately (Ge & Land, 2003; Veenman et al., 1997), even when a more thorough analysis would actually help them to build a detailed plan for learning (Veenman, Elshout, & Busato, 1994).  
  • The learning goals they set are often of low quality, tending to be nonspecific and distal (Zimmerman, 1998).
  • In addition, many students fail to set up a detailed plan for learning, whereas if they do create a plan, it is often poorly constructed (Manlove et al., 2007). That is, students often plan their learning in a nonsystematic way, which may cause them to start floundering (de Jong & Van Joolingen, 1998), or they plan on the basis of what they must do next as they proceed, which leads to the creation of ad hoc plans in which they respond to the realization of a current need (Manlove & Lazonder, 2004).  
  • The lack of proper planning for learning may cause students to miss out on experiencing critical moments of inquiry, and their investigations may lack systematicity.
  • Many students also have problems with monitoring their progress, in that they have difficulty in reflecting on what has already been done (de Jong & Van Joolingen, 1998).
  • Regarding monitoring of understanding, students often do not know when they have comprehended the subject matter material adequately (Ertmer & Newby, 1996; Thiede, Anderson, & Therriault, 2003) and have difficulty recognizing breakdowns in their understanding (Ertmer & Newby, 1996).
  • If students do recognize deficits in their understanding, they have difficulty in expressing explicitly what they do not understand (Manlove & Lazonder, 2004).
  • One consequence is that students tend to overestimate their level of success, which may result in “misplaced optimism, substantial understudying, and, ultimately, low test scores” (Zimmerman, 1998, p. 9).

The research article is available by clicking here.

Final Thoughts

This research, and other research I have studied over the years, shows that we CANNOT ALWAYS TRUST THAT OUR LEARNERS WILL KNOW HOW TO LEARN. We as instructional designers have to design learning environments that support learners in learning. We need to know the kinds of learning situations where our learners are likely to succeed and those where they are likely to fail without additional scaffolding.

The research also shows, more specifically, that inquiry-based simulation environments can be powerful learning tools, but ONLY if we provide the learners with guidance and/or scaffolding that enables them to be successful. Certainly, some few may succeed without support, but most will act suboptimally.

We have a responsibility to help our learners. We can't always put it on them…

What is Subscription Learning

,

Subscription Learning Defined

Subscription learning, as its name implies, provides an intermittent stream of learning-related interactions to those who are subscribed. These learning-related interactions–called “nuggets”–can involve a great variety of learning-related events, including content presentation, diagnostics, scenario-based questions, job aids, reflection questions, assignments, discussions, etc.

Nuggets are short, usually presented in less than five minutes, delivered to recipients using push technology; either through email, text messaging, cell-phone alerts, desktop notifications, or some other form of prompting. Nuggets are intentionally scheduled over time to support learning, often utilizing research-based findings related to the spacing effect. Learners subscribe (or are subscribed) to one or more series of learning nuggets, called “threads.”

Learning threads can be predesigned, creating nuggets based on anticipated learner needs or they can be dynamically created based on learner performance. Threads can be an alternate way to present course-like information, they can augment other learning interventions, or they can provide an alternative to traditional learning approaches. In addition, subscription learning can go beyond learning delivery and be designed to create open communication channels, directly prompt performance, or provide hybrid experiences.

Visual Representation

Let’s compare a 90-minute elearning course to a commensurate subscription-learning thread. Typically, learners will engage a 90-minute elearning program in one sitting on one day, never to engage the learning material again. A standard e-learning course could be pictured as one big block of learning, whereas the same content in a subscription-learning thread could be presented as 18 five-minute learning nuggets.

Given the time-based reality that subscription-learning nuggets are spread over time, it’s probably best to present nuggets as typically distributed on a calendar.

Special Capabilities of Subscription Learning

Subscription learning provides affordances that standard elearning typically does not provide. Here’s a short list:

  • Relatively long spaced-repetition intervals to maximize remembering
  • Engagement in the learners’ work context over time
  • Opportunities for learners to practice real-world tasks over time
  • Increased likelihood that learning will be discussed with coworkers
  • Multiple learning contexts in terms of cognitive frames of mind
  • Heightened attention to learning content with fresh engagements
  • Potential to prompt on-the-job actions with multiple prompts
  • Potential to engage in social-media conversations that support learning
  • Potential to engage with senior leaders or other organizational experts
  • Potential to modify content in response to learning dynamics
  • Potential to modify content in response to organizational changes

Research-Based Benefits

From a learning standpoint, subscription learning has several advantages over typical training. First, it keeps learning fresh in the minds of learners. Forgetting is a natural cognitive process, but it is the scourge of most training programs. By utilizing a thread of short nuggets, key concepts stay accessible from within long-term memory. Second, subscription learning can be designed to utilize the spacing effect—one of learning science’s most studied phenomenon. The spacing effect is the research finding that repetitions of concepts spaced over time are a more potent way to support long-term memory than repetitions that are not spaced over time (or are not spaced as widely in time). Interestingly, while the spacing effect is one of the most studied learning factors it is also one of the most underutilized in training and education.

Note that while subscription learning is the learning approach most amenable to the spacing effect, subscription learning need not utilize the spacing effect. When a learning thread presents new content in each nugget–and never utilizes the power of repetition–we still have subscription learning, but we don’t have the spacing effect.

Macro vs. Micro Spacings

While immersing myself in the research and attempting to bridge the gap between the research and practical learning situations, I realized that a new terminology was needed. Spacings can be relatively long, say providing a content repetition after four days; or relatively short, for example providing content repetitions after a minute or two. In practice, both can work very effectively—but their relevance depends on factors like the topic, the learners’ knowledge level, importance, etc.

Micro spacings are utilized when learners have to learn many separate bits of knowledge. For example, in learning a language, learners are trying to learn tons of vocabulary—so waiting a week between content presentations would be too slow. In such a case, micro spacings present many learning opportunities within a short time frame. Micro spacings are particularly valuable for things like language learning, vocabulary, multiplication tables, or any learning situation where there is a lot of learning information—or there is a short time horizon for learning a smaller set of information.

Macro spacings are utilized when learners only have to learn a short list of learning points (say less than 20)—or when there is no urgency in learning and repetitions can be leisurely spread over time.

Of course, we shouldn’t forget the findings of the learning researchers. In general, the longer the spacing interval, the longer the information will be remembered. Indeed, micro spacings often require a hybrid spacing scheme, using multiple micro spacings in one sitting, but then utilizing macro spacing to ensure long-term remembering.

The diagram below is my attempt to show how micro spacings differ from macro-spacings—and how they can be combined.

 

Why is the Subscription-Learning Notion Important?

To put it simply, subscription learning is a radical new paradigm in elearning. It enables us to think about elearning in new ways–in ways that challenge our old mental models of elearning.

Elearning is still a relatively young field, having its start in the 1960’s during the advent of the computer age and gradually gaining a critical mass after the internet became a mass phenomenon. Because it’s a young field, we are still learning how to think about elearning. With each new paradigm, we think more deeply, more fully about what elearning is–and can be. Below is my categorization of the most important elearning paradigms as of 2014.

eLearning Paradigms 2014

  • Content Presenter (enables content to be presented to learners)
  • Comprehension Tester (enables learners’ knowledge to be tested–and feedback provided)
  • Practice Provider (enables learners’ decision-making to be tested–and feedback provided)
  • Performance Supporter (enables performers to be prompted toward action)
  • Reminder (enables learners or performers to be reminded to learn and/or take action)
  • Social Augmentation Provider (enables learners to learn from and with each other)
  • Gamification Provider (provides motivational incentives and behavioral prompts to action)
  • Mobile Learning Provider (provides learning and/or performance support through mobile technology)
  • Data Utilizer (enables data collection and data-based interventions)
  • Video Provider (enables video to be utilized in various ways)
  • Learning Organizer (provides organizational structure around learning opportunities)
  • Personalizer (enables content or prompting to be individualized or tailored)
  • Learning-Delivery Augmenter (enables easy delivery of content or prompting)
  • Context-Based Triggerer (enables content or prompting to be delivered depending on context)
  • Cost Saver (enables learning to be delivered at a lower cost)

I’m sure that I’m missing some elearning paradigms. You might have noticed that I’m only listing elearning memes that have a positive connotation. I am not mentioning such things as boring, trivial, poorly-designed. Also, some of the list may not be true, or may not always be true. For example, I’ve recently read research that shows that elearning is not often a cost saver. The bottom line, however, is that the list above represents a good number of the ways in which we tend to think about elearning.

Here’s the thing: The paradigms listed above represent the dominant mental models we use when we think about elearning. As Thomas Kuhn wrote many years ago, paradigms are a double-edge sword. On the one hand, they help us think. On the other hand, they put boundaries on what we think. For us in the learning field, we get both benefits and costs from our elearning paradigms. They help us consider ways that we might design or utilize elearning. On the darker side, they constrict our thinking. One of the reasons we created the eLearningManifesto was to get the field to think beyond some of its weaker paradigms.

 

The Subscription-Learning Meme

I am pushing the idea of subscription learning not just because it is aligned with the learning research on the spacing effect, but also because it gives us a completely new way of thinking about elearning. It opens elearning to new possibilities.

Where we often think of elearning content delivery as requiring relatively long events of 30 minutes or more, subscription learning lets us think of much shorter events spaced over time. Where we often think of performance support as being delivered through a single-focus system at a time of known need, subscription learning can prompt a series of thoughts or actions even when learners don’t know they need to know. Where we think that learners have to seek their own learning nuggets, subscription learning can push learning to learners to better support learning as a process.

Rethinking Instructional Objectives

,

Clark Quinn (blog, website, Twitter)  recently cited some of my thinking about instructional objectives in the instructional technology forum of AECT (ITFORUM). I wrote a long email to Clark in response, thanking him, and going into more detail. I am reprising my response to Clark here:

In a recent post to this list, Clark Quinn rightly notes that objectives for learners and objectives for instructional designers need not be identical. Indeed, as both Clark and I have previously noted, the probably shouldn’t be identical.

Here’s the thinking: Objectives are designed to guide behavior. So, how can it be that identically-worded objectives can adequately guide the behavior of two disparate groups of individuals (learners and instructional designers)? It just doesn’t make any sense!!

And indeed, Hamilton (1985) found that presenting learners with learning objectives in the way Mager suggested, PRODUCES NO BENEFITS AND MAY BE HARMFUL. Here’s what Hamilton wrote:

“[An instructional] objective that generally identifies the information to be learned in the text will produce robust effects. Including other information (per Mager’s, 1962, definition) will not significantly help and it may hinder the effects of the objectives”

(Hamilton, 1985, p. 78).

Objectives are not only designed to change behavior for a particular set of individuals, but they are also designed with particular purposes in mind—or they should be.

So, when we talk of instructional objectives, we also need to think about what purpose we have for them.

The quote above from Hamilton is focused on how well learning objectives focus the attention of learners. Interestingly, this is the only area in which extensive research has been done on learning objectives. You might be surprised to know that learning objectives help learners focus on the information targeted by learning objectives, but actually diminish their attention on information in the learning materials not targeted by learning objectives. For example, in two experiments using specific objectives, Rothkopf and Billington (1979) found that when focusing objectives were provided to learners, performance on material related to the objectives improved by 49% and 47% over situations when focusing objectives were not used. However, the material not related to the learning objectives was learned 39% and 33% WORSE than it would have been if no learning objectives were used!

These types of instructional objectives—presented to learners prior to subsequent learning—I call “focusing objectives” because they are designed for the purpose of focusing learner attention on critical learning material. As the Hamilton (1985) review pointed out, it does NOT help to add Mager’s criterion information to focusing objectives, because it doesn’t help learners focus on the critical material.

NOW, here’s an important point (I say to focus your attention):  We don’t necessarily need to use focusing objectives with learners if we have other means to focus their attention!! We can use a relevant, gripping story. We can do a shout-out (example, “Here’s an important point…”). We can have them attempt to answer a relevant scenario-based question and struggle with it. Etcetera.

Here’s another important point: Focusing objectives are only one type of objective we might want to utilize. I have a whole list, and I’m sure you can think of more of them.

Instructional Objectives for Learners:

  1. Table-of-Contents Objectives
    To give learners a big picture sense of what will be taught.
  2. Performance Objectives
    To let learners know what performance will be expected of them.
  3. Motivation Objective
    To ensure learners know why they might be motivated to engage the learning or application of the learning.
  4. Focusing Objective
    To guide learner attention to the most critical information in the learning material.

Instructional Objectives for Developers:

  1. Instructional-Design Objective
    To guide developers toward the ultimate goal of the learning intervention.
  2. Evaluation Objective
    To guide developers (and other stakeholders) to the ultimate measurable outcomes that the learning intervention will be measured by.
  3. Situation Objectives
    To guide developers to the situations that learners must be prepared for.
  4. Organization Objective
    To guide developers to the organizational effects targeted by the instruction.

 

Questions:

So, here’s some questions for you:

Is it okay to use the word understand in an “instructional-design objective”?

How about in a “focusing objective”?

Answer: It’s okay to use the word understand in a focusing objective—because it does not hurt the learner in setting them up to focus attention on critical concepts. But it is NOT okay to use the word understand in an instructional-design objective—because the word “understand” doesn’t have enough specificity to guide instructional design.

My point in asking these questions is to show that over-simplistic notions about instructional objectives are likely to be harmful to your instructional designs.

As usual, the research helps us see things we wouldn’t otherwise have seen.

Hope this helps!!

= Will

 

Will’s Note:

References

Hamilton, R. J. (1985). A framework for the evaluation of the effectiveness of adjunct questions and objectives. Review of Educational Research, 55, 47-85.

Mager, R. (1962). Preparing Instructional Objectives. Palo Alto, CA: Fearon Publishers.

Rothkopf, E. Z., & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive processing model from inspection times and eye movements. Journal of Educational Psychology, 71(3), 310-327.

CEO’s Do NOT Successfully Transfer Their Skills to Other Companies!

In a recent, NY Time Article, cited a paper (Executive Superstars, Peer Groups and Over-­‐Compensation –Cause, Effect and Solution) by Charles M. Elson and Craig K. Ferrere who compile research from others (see citations below) that basically shows that CEO's do NOT successfully transfer their skills from one company to another. CEO's who are hired from within tend to do a much better job than CEO's brought in from the outside.

We in the learning-and-performance field should take note. The reason this is true is because it takes real-world expertise, gained through long bouts of experience in a particular set of situations, to give people the tools they need to be experts.

This we should all know. If you want to be good in calculus as it applies to astronomy, you ought to get practice in applying calculus to astronomy. It is not enough to just learn calculus and learn astromony.

And certainly it is not helpful to learn Latin to learn math, or even to learn other languages. Anybody who has been to graduate school in learning should have learned about how unlikely it is for learning to transfer from one domain to the other. Thorndike anyone?

Donald Clark has a nice blog post on Thorndike and the uselessness of learning latin.

So, from both a training and on-the-job learning perspective, we should be giving our learners extensive opportunities to learn their areas of expertise.

This doesn't mean they should never dabble in other domains. In fact, creativity research shows that "domain-spanners" are more likely to be creative than those who only learn with a narrow focus. HOWEVER, people who are only domain-spanners (and don't know a field really well) tend to just be good at coming up with ideas–but not generally good ones. Expertise matters!

Those of you in the United States who are thinking about our election in terms of this lack of transfer from one job situation to another could have a field day…or a depression.

 

Some of Elson and Ferrere's citations showing that CEO's skills don't transfer well:

Richard A. Cazier and John M. McInnis, Do Firms Contract Efficiently on Past Performance When Hiring External CEOs? (Working Paper, 2010) (studying 192 external successions from 1993-2005 they find a negative correlation between excess compensation and future performance;; paradoxically the superstars underperformed).

Gregory L. Nagel, William G. Hardin III, The Transferability of CEO Skills, (Working Paper, 2007)

Mark R. Huson, Paul H. Malatesta and Robert Parrino, Managerial succession and firm performance, 74 J. FIN. ECON. 237, 237 (2004) (finding that “the appointment of outside successors [is] not significantly related to post-turnover performance changes in [their] regression analysis.”)

Chuck Lucier, Steven Wheeler, and Rolf Habbel, The Era of the Inclusive Leader, 47 STRATEGY+BUSINESS (2007) (finding that “experienced CEOs”, though hypothesized to bring experience in dealing with stakeholders and shareholders actually underperformed).

James S. Ang and Gregory L. Nagel, The Financial Outcome of Hiring a CEO from Outside the Firm, (Working Paper, 18, 34, 2011) (using a structural self-selection model to estimate counterfactual performance that would have been obtained if the firm hired an insider rather than an outsider and vice-versa, at all target levels of performance internal hires stochastically dominate outsiders).

Is Your Training Course Likely to Boost Performance? Free Course Review Template.

, ,

Despite decades of advocacy by our best trade associations, our wisest gurus, and our most practical researchers, most organizations today still rely on training courses that have little impact in promoting on-the-job performance.

As I mentioned in a recent article, we as learning professionals continue to fail in five major ways. You can access that article by clicking here.

I used to think that this was just a failure of knowledge, but in most of the organizations in which I’ve consulted, there are at least a few learning-and-performance professionals who understand that training alone is not enough. Part of the problem is the dead weight of tradition–the “old normal” continues to blind us to new possibilities. The enlightened few have a hard time pushing back against the gravitational pull of this mass hypnosis.

I recently had a new insight–a way of looking at this problem that I think might enable organizations to break out of their bad habits. The solution is that we have to gain control of the leverage points we have to push for change. We have to change the levers that warp and control our thinking. The big lever is learning measurement. I’ve been pushing this for years as our most important leverage point. If we measured better, we’d get better feedback, which would push us to create better learning interventions.

But learning measurement isn’t our only lever and changing your learning measurement practices is not always easy politically. Beside learning measurement, I’ve compiled a whole list of other leverage points that  really matter. In fact, it was only recently that I had this incredible insight (one I maybe should have had 10 years ago), that we ought to figure out all the levers we have at our disposal and change them to help push our organizations toward a performance orientation. I’d like to reveal one of those levers today.

One of the things we do in our organizations is review our training courses from time to time–either intentionally or by osmosis and feeling. Well, instead of using the wrong metrics, why not use methods that we know–based on our understanding of learning-and-performance–are likely to be good indicators of whether our training course will support actual on-the-job performance.

The Course Review Template is something that can be used on any training course–classroom training or e-learning.It includes a set of questions that are indicators of how performance-based your training course is. Each rubric in this tool is inspired by research or proven practices which I’ve learned in my 25+ years in the workplace learning field.

I should give you a warning. You’re unlikely to be happy with what you find. If I bet each of you one dollar for each training course of yours that doesn’t support performance, I’d be a millionaire overnight.

But to be fair, I’m going to let you try out the tool yourself. It’s free. Use it. And, let me know how your training courses rate. Are they likely to improve on-the-job performance or not?

Click to Download the Course Review Template

After you review a course, post your results at the following link, and when we get enough responses, we’ll let you compare your results to others.

Click to Post Your Course Review Results — SORRY, we’re done collecting data in a survey format.

Maybe I’m having a momentary bout of delusional cognition, but I’m thinking right now that this simple Course Review Template might just revolutionize our ability to simply review our courses to see how performance focused they are.

Such a grandiose statement will provoke eye rolls in some, so let me stipulate a few things. First, this is a first draft, so the Course Review Template is going to be imminently improvable. Second, the Course Review Template is NOT a precision instrument. It is not psychometrically derived, the numbers it assigns to each rubric are best guesses, and there was no super-committee here–just me. Third, the rubrics themselves are subject to interpretation. Instead of over-complicating the form and making it unusable, I decided to keep it simple and make it less precise. Finally, course reviews are just one of the levers you’ll need to completely transition from a course-focus to a performance-focus.

The bottom line is that we have to try some innovate new things to push our organizations to a performance focus. The old ways have not worked. The Course Review Template–or something like it–is worth a try. And seriously, I think it could revolutionize the way your organization views its training courses.

NOTE 2017: While this is the original blog post, it now includes the latest version of the Course Review Template. A later post that introduced the improvements is available here.

 

The Five Failures of Workplace Learning Professionals

,

To view this as a PDF, click here.

————————-

To improve, we must know our biggest failings.

In the training and development field, our five biggest failures are as follows:

  1. We forget to minimize forgetting and improve remembering.
  2. We don’t provide training follow-through.
  3. We don’t fully utilize the power of prompting mechanisms.
  4. We don’t fully leverage on-the-job learning.
  5. We measure so poorly that we don’t get good feedback to enable improvement.

1. Minimizing Forgetting, Improving Remembering

It is not enough to help people understand new concepts or even to motivate them to utilize those concepts. If they don’t remember concepts when they encounter situations in which those concepts would be useful, then previous understanding and motivation is for naught.

There are three powerful mechanisms that support long-term remembering, (a) aligning the learning and performance contexts, (b) providing retrieval practice, and (c) utilizing spaced repetitions. Most of our learning interventions do a poor job of providing these mechanisms—resulting in training that may create awareness but doesn’t support remembering or performance improvement.

We need to give our learners more realistic practice using scenarios and simulations. We also need to space repetitions of learning over time—much more than we do now. Instead of trying to teach everything at a basic awareness level, we need to cover less content—but not just present it—instead giving our learners opportunities for deliberate practice.

2. Training Follow-Through

Providing training but no effort to ensure that learners will apply what they’ve learned is the height of professional malpractice. If we assume that learners remember what they’ve learned (which as we just saw is not a given), learners still must (a) remain motivated to apply what they’ve learned, (b) feel that there is some benefit to applying the learning, (c) have the resources and time to put their learning into practice, (d) get feedback and guidance to improve their performance, and (e) be prepared to overcome obstacles and frustrations in applying the learning.

Note how the first two failures create an additive effect—both significantly lessen the likelihood of on-the-job application of the learning. If learners don’t remember, they’re not going to apply what they’ve learned. If learners don’t receive after-training follow-through support, they are unlikely to provide the continuous and persistent focus needed to apply the learning in a way that creates sustainable success.

To reach a credible level of training follow-through we need to (a) engage our learners managers to enlist their support, (b) provide reminders to apply the learning, (c) provide relearning opportunities for that which has been forgotten, (d) enable additional learning to improve and elaborate on the performance, (e) ensure our learners have the resources and time they need to apply the learning and integrate it into their behavioral repertoire, (f) provide coaching support to guide the learning-and-performance process, (g) ensure the learners are incentivized either tangibly with money or perks or intrinsically by aligning efforts with personal values and sense-of-identity, and (h) encourage persistence even in the face of obstacles and frustration.

3. Prompting Mechanisms

Prompting mechanisms rely on one particularly powerful foible of the human cognitive architecture—that our working memories are triggered easily by environmental stimuli. Prompting mechanisms include things like job aids, performance support tools, signage, intuitive cues in our tools and equipment, and some forms of management oversight. They work because they prompt certain strands of thinking, and thus performance. For example, a job aid that lists 5 key interview goals, 10 key interview questions and their rationales automatically triggers in the interviewer a certain way of thinking about interviewing. For example, an interview template might remind its user that interviews are more telling if interviewees are asked to perform a work task or describe how they would perform a work task. Without such a prompt, the interviewer might focus only on how well they think the person would fit into the work culture, etc.

While we are aware of these prompting mechanisms, we are not aggressive enough in their use. If we utilized prompting mechanisms more often with our training and more often as a replacement for training, we’d create better outcomes. If we went looking for grassroots prompting mechanisms already being used and helped spread their use, we’d be more effective. If we evaluated learning facilitators on their use of prompting mechanisms, we’d be more likely to encourage the use of prompting mechanisms. If we asked learners in training to practice with prompting mechanisms, we’d see more being used on the job—and our learners would remember more of what they learned.

4. On-the-Job Learning

We as learning professionals tend to focus almost exclusively on the creation and delivery of training interventions even when we know that our learners are doing a great deal of their learning on the job without any training. Employees learn through trial-and-practice, getting help from others, through social media, by reading task instructions, by using help systems, and so forth. While we have much less direct influence on on-the-job learning than on training, we do have some influence and we ought to use it if we are serious about getting results.

Often the biggest impact we can have is by accessing managers and encouraging them to actively promote learning. Managers can improve learning in their direct reports by (a) making it a point to monitor their employees’ competencies and guide them toward learning opportunities, (b) being approachable and available for questions and advice, (c) creating a culture of learning and information sharing, (d) encouraging data-driven decision-making instead of opinion-driven decision-making, (e) utilizing an experimental mindset, for example by encourage pilot-testing and rapid prototyping, and (f) giving direct reports time for learning and exploration.

We can also have an influence on on-the-job learning by creating and maintaining social-media mechanisms that can be tailored to particular needs. For example, wikis can be used by project teams to get input from various parties and blogs can be used by senior folks to lay out a compelling vision.
We can encourage better on-the-job learning by improving people’s ability to coach their fellow employees. Too often people asked to coach others do a poor job because they just don’t know what good coaching looks like.

We can utilize diagnostic tools to help people in the organization see things about themselves—or about the organization—that they might not otherwise see. For example, if the organization engages in an effort to improve coaching ability, those being coached can be asked to take a short diagnostic survey on how well their coach is doing in coaching them. If an organization wants to change its culture to one that is more flexible and creative, we can utilize a diagnostic to track progress. We can also use a diagnostic to get the organization talking about specifics—so that employees know what behaviors represent the past culture and which represent the new culture.

There are, of course, other things we can do to directly influence on-the-job learning. In addition, we can change our brand by stopping our tendency to be order takers for training. By changing the way we define our role, we can encourage the business side to be fuller partners in organizational learning.

5. Measurement and Feedback to Spur Improvement

We as learning professionals suck at measurement, creating a vacuum of information that pushes us to make poor decision after poor decision in our learning designs. By only seeking learner opinions about the learning, we encourage a bias toward entertainment and engagement and away from content validity, remembering, and application. By measuring only when the learners are in the training context, we don’t learn whether the learning intervention would generate remembering in a work context that is not like the training situation. By measuring only during the learning event, we measure the learning intervention’s ability to create understanding, but we do not measure the learning intervention’s ability to support long-term remembering. We also fail to examine whether any training follow-through is utilized. By utilizing only low-level questions in our tests of learning, we fail to measure the ability of our learners to make decisions that relate to workplace performance. In short, we don’t get the feedback we need to make good learning decisions.

Maintaining ourselves in a state of permanent darkness, we continue to make terrible decisions in regard to learning design, development, and deployment. We design primarily for engagement and understanding, while ignoring remembering, motivation, and application. We hire and promote trainers and training companies who get great ratings but who don’t help learners remember or apply what they’ve learned. Because our measurement is focused only on training, we fail to engage our business partners to ensure that they are adequately supporting learning application—we also never learn what obstacles and leverage points face our learners when they go to apply the learning in their jobs. We build e-learning programs that encourage learners to focus on low-level trivia instead of focusing on the main points. By abstaining from diagnostics, we leave employees blind to conditions from which they might benefit. Poor measurement enables the first four failures.

The bottom line on measurement is that measurement should provide us with valid feedback. Unfortunately, because we haven’t taken the human learning system into account in our measurement designs—and in our measurement models—we are getting biased information and drawing inappropriate conclusions from poor data.

The Five Failures are Fixable

We as learning professionals—as a whole—though working honorably and with good intentions, are too often failing to maximize our impact. Our job is work-performance improvement. We can start by improving our own work performance.

But instead of focusing on everything—which will certainly overwhelm us—we should focus on the things that really matter. We should focus on our five failures. Instead of following willy-nilly prescriptions that pop like fads from a popcorn popper—we should focus on five things that are fundamental—and inspired by the learning research. We should focus on the five failures.

In this brief article, I have provided strong hints about how to rethink and redirect each of the five failures. While such a brief synopsis is certainly not sufficient to enable you to completely redesign your learning efforts, it should, I hope, motivate you to get started.

————————-

To view this as a PDF, click here.

 

Learning and Forgetting Curves — In Depth On Video

I've been sharing the ideas behind the learning and forgetting curves for years.

In this new video, I go into depth about the powerful implications of these curves.

 

 

Question Types and Methods

, ,

Many of us are inclined to see audience response systems only as a way to deliver multiple-choice and true-false questions. While this may be true in a literal sense, such a restricted conception can divert us from myriad possibilities for deep and meaningful learning in our classrooms.

The following list of 39 question types and methods is provided to show the breadth of possibilities. It is distilled from 85 pages of detailed recommendations in the white paper, Questioning Strategies for Audience Response Systems: How to Use Questions to Maximize Learning, Engagement, and Satisfaction, available free by clicking here.

NOTE from Will Thalheimer (2017): The report is focused on audience-response systems — and I must admit that it is a bit dated now in terms of the technology, but the questions types are still a very potent list.

1. Graded Questions to Encourage Attendance

Questions can be used to encourage attendance, but there are dangers that must be avoided.

2. Graded Questions to Encourage Homework and Preparation

Questions can be used to encourage learners to spend time learning prior to classroom sessions, but there are dangers that must be avoided.

3. Avoiding the Use of One Correct Answer (When Appropriate)

Questions that don’t fulfill a narrow assessment purpose need not have right answers. Pecking for a correct answer does not always produce the most beneficial mathemagenic (learning creating) cognitive processing. We can give partial credit. We can have two answers be equally acceptable. We can let the learners decide on their own.

4. Prequestions that Activate Prior Knowledge

Questions can be used to help learners connect their new knowledge to what they’ve already learned, making it more memorable. For example, a cooking teacher could ask a question about making yogurt before introducing a topic on making cheese, prompting learners to activate their knowledge about using yogurt cultures before they begin talking about how to culture cheese. A poetry teacher could ask a question about patriotic symbolism, before talking about the use of symbols in modern American poetry.

5. Prequestions that Surface Misconceptions

Learners bring naïve understandings to the classroom. One of the best ways to confront misconceptions is to bring them to the surface so that they can be confronted straight on. The Socratic Method is a prime example of this. Socrates asks a series of prequestions thereby unearthing misconceptions and leading to a new improved understanding.

6. Prequestions to Focus Attention

Our learners’ attention wanders. In an hour-long session, sometimes they’ll be riveted to the learning discussion, sometimes they’ll be thinking of other ideas that have been triggered, and sometimes they’ll be off in a daze. Prequestions (just like well-written learning objectives) can be use to help learners pay attention to the most important subsequent learning material. In fact, in one famous study, Rothkopf and Billington (1979) presented learners with learning objectives before they encountered the learning material. They then measured learning and eye movements and found that learners actually paid more attention to aspects of the learning material targeted by  the learning objectives. Prequestions work the same way as learning objectives—they focus attention.

7. Postquestions to Provide Retrieval Practice

Postquestions—questions that come after the learning content has been introduced—can be used to reinforce what has been learned and to minimize forgetting. This is a very basic process. By giving learners practice in retrieving information from memory, we increase the probability that they’ll be able to do this in the future. Retrieval practice makes perfect.

8. Postquestions to Enable Feedback

Feedback is essential for learners and instructors. Corrective feedback is critical, especially when learners have misunderstandings. Providing retrieval practice with corrective feedback is especially important as learners are struggling with newly-encountered material, difficult material, and when their attention is likely to wander—for example when they’re tired after a long-day of training, when there are excessive distractions, or when the previous material has induced boredom.

9. Postquestions to Surface Misconceptions

We already talked about using prequestions to surface misconceptions. We can also use postquestions to surface misconceptions. Learners don’t always understand concepts after only one presentation of the material. Many an instructor has been surprised after delivering a “brilliant” exposition to find that most of their learners just didn’t get it.

10. Questions Prompting Analysis of Things Presented in Classroom

One of the great benefits of classroom learning is that it enables instructors to present learners with all manner of things. In addition to verbal utterances and marks on a white board, instructors can introduce demonstrations, videos, maps, photographs, illustrations, learner performances, role-plays, diagrams, screen shots, computer animations, etcetera. While these presentations can support learning just by being observed, questions on what has been seen can prompt a different focus and a deeper understanding.

11. Using Rubric Questions to Help Learners Analyze

In common parlance, the term “rubric” connotes a set of standards. Rubrics can be utilized in asking learners questions about what they experience in the classroom. Rubric questions, if they are well designed, can give learners practice in evaluating situations, activities, and events. Such practice is an awesome way to engage learners and prepare them for critical thinking in similar future situations. In addition, if rubrics are continually emphasized, learners will integrate their wisdom in their own planning and decision-making.

12. Questions to Debrief an In-Class Experience

Classrooms can also be used to provide learners with experiences in which they themselves participate. Learners can be asked to take part in role plays, simulations, case studies, and other exercises. It’s usually beneficial to debrief those exercises, and questions can be an excellent way to drive those discussions.

13. Questions to Surface Affective Responses

Not all learning is focused on the cold, steely arithmetic of increasing the inventory of knowledge. Learners can also experience deep emotional responses, many of which are relevant to the learning itself. In topics dealing with oppression, slavery, brutality, war, leadership, glory, and honor, learners aren’t getting the full measure of learning unless they experience emotion in some way. Learners can be encouraged to explore their affective responses by asking them questions.

14. Scenario-Based Decision-Making Questions

Scenario-based questions present learners with scenarios and then ask them to make a decision about what to do. These scenarios can take many forms. They can consist of short descriptive paragraphs or involved case studies. They can be presented in a text-only format or augmented with graphics or multimedia. They can put the learner in the protagonist’s role (“What are you going to do?”) or ask the learner to make a decision for someone else (“What should Dorothy do?”). The questions can be presented in a number of formats—as multiple-choice, true-false, check-all-that-apply, or open-ended queries.

15. Don’t Show Answer Right Away

There’s no rule that you have to show learners the correct response right after they answer the question. Such a reflexive behaviorist scheme can subvert deeper learning. Instructors have had great success in withholding feedback. For example, Harvard professor Mazur’s (1997) Peer Instruction method requires learners to make an individual decision and then try to convince a peer to believe the same decision—all before the instructor weighs in with the answer.

By withholding feedback, learners are encouraged to take some responsibility for their own beliefs and their own learning. Discussions with others further deepen the learning. Simply by withholding the answer, instructors can encourage strategic metacognitive processing, thereby sending learners the not-so-subtle message that it is they—the learners—who must take responsibility for learning.

16. Dropping Answer Choices

There are several reasons to drop answer choices after learners have initially responded to a question. You can drop incorrect answer choices to help focus further discussions on more plausible alternatives. You can drop an obviously correct choice to focus on more critical distinctions. You can drop an unpopular correct choice to prompt learners to question their assumptions and also to highlight the importance of examining unlikely options. Each of these methods has specific advantages.

17. Helping Learners Transfer Knowledge to Novel Situations

“Transfer” is the idea that the learning that happens today ought to be relevant to other situations in the future. More specifically, transfer occurs when learners retrieve what they’ve learned in relevant future situations. As we’ve already discussed, the easiest and often the most potent way to promote transfer is to provide learners with practice in the same contexts—retrieving the same information—that they’ll be required to retrieve in future situations. But questions can also be used to prepare learners to retrieve information in situations that are not, or cannot, be anticipated in designing the learning experience.

18. Making the Learning Personal

By making the learning personal, we help learners actively engage the learning material, we support mathemagenic cognitive processing, and we make it more likely that they’ll think about the learning outside of our classrooms, further reinforcing retention and utilization. Questions can be designed to relate to our learners’ personal experiences, thus bolstering learning.

19. Making the Material Important

Sometimes we can’t make the material directly personal or provide realistic decisions for learners to make, but we can still use questions to show the importance of the topic being discussed.

20. Helping Learners Question Their Assumptions

One of our goals in teaching is to get learners to change their thinking. Sometimes this requires learners to directly confront their assumptions. Questions can be written that force learners to evaluate the assumptions they bring to particular topic areas.

21. Using the Devil’s Advocate Tactic

In a classroom, when we play the devil’s advocate, we argue ostensibly to find flaws in the positions put forth. The devil’s advocate tactic can be used in a number of different ways. You can play the devil’s advocate yourself, or utilize your learners in that role. From a learning standpoint, when someone plays the devil’s advocate, learners are prompted to more fully process the learning material.

22. Data Slicing

Data slicing is the process of using one factor to help make sense of a second factor. So for example, through the use of our audience response systems, we might examine how our learner’s socio-economic background affects their opinion of race relations. Data slicing can be done manually or automatically. It is particularly powerful in the classroom for demonstrating how audience characteristics may play a part in their own perceptions or judgments.

23. Using Questions for In-class Experiments.

For some topics, in-class experimentation—using the learners as the experimental participants—is very beneficial. It helps learners relate to the topic personally. It also highlights how scientific data is derived. For example, in a course on learning, psychology, or thinking; learners could be asked to remember words, but could—unbeknownst to them—be primed to think about certain semantic associates and not others.

24. Prompting Learners to Make Predictions

Prediction-making can facilitate learning in many ways. It can be used to provide retrieval practice for well-learned information. It can be used to deepen learners’ understandings of boundary conditions, contingencies, and other complications. It can be used to engender wonder. It can be used to enable learners to check their own understanding of the concepts being learned.

25. Utilizing Student Questions and Comments

Our learners often ask the best questions. Sometimes a learner’s question hints at the outlines of his or her confusion—and the confusion of many others as well. Sometimes learners want to know about boundary conditions. Students can also offer statements that can improve the learning environment. They may share their comfort level with the topic, add their thoughts in a class discussion, or ar
gue a point because they disagree. All of these interactions provide opportunities for a richer learning environment, especially if we—as instructors—can use these questions to generate learning.

26. Enabling Readiness When Learners are Aloof or Distracted

Let’s face it. Not all of our learners will come into our classrooms ready to learn. Some will be dealing with personal problems. Some will be attending because they have to—not because they want to. Some will be distracted with other stress-inducing responsibilities. Some will think the topic is boring, silly, or irrelevant to them. Fortunately, experienced instructors have discovered tricks that often are successful. Audience response technology can help.

27. Enabling Readiness When Learners Think They Know it All

Some learners will come to your classroom thinking they already know everything they need to know about the topic you’re going to discuss. There are two types of learners who feel this way—those who are delusional (they actually need the learning) and those who are quite clearheaded (they already know what they need to know). Using the right questions and gathering everyone’s responses can help you deal with both of these characters.

28. Enabling Readiness When Learners are Hostile

In almost every instructor’s life, there will come a day when one, two, or multiple learners are publicly hostile. Experienced instructors know that such hostility must be dealt with immediately—not ignored. Even a few bad apples can ruin the learning experience and the satisfaction of the whole classroom. Fortunately, there are ways to fend off the assault.

29. Using Questions with Images

Using images as part of the learning process is critical in many domains. Obvious examples are art appreciation, architecture, geology, computer programming, and film. But even for the least likely topics, such as poetry or literature, there may be opportunities. For example, a poetry teacher may want to display poems to ask learners about the physical layout of poems. Images should not be thrown in willy-nilly. They should be used only when they help instructors meet their learning goals. Images should not be used just to make the question presentation look good. Research has shown that placing irrelevant images in learning material, even if those images seem related to the topic, can hurt learning results, distracting learners from focusing on the main points of the material . One easy rule: Don’t use images if they’re not needed to answer the question.

30. Aggregating Handset Responses for a Group or Team

Some handset brands enable responses of individual handsets to be aggregated. So for example, an instructor in a class of 50 learners might break the learners into 10 teams, with five people on a team. All 50 learners have a handset, but the responses from each team of five learners are aggregated in some way. This aggregation feature enables some additional learning benefits. Teamwork can be rewarded and competition between teams can add an extra element of motivation. Using aggregation scoring allows the instructor to encourage out-of-class activities where learners within a team help each other. Obviously, this will only work if the learning experience takes place over time. In such cases, aggregation can be used to build a learning community. Learners can be assigned to the same team or rotated on different teams, depending on the goals of instruction. Putting learners on one team encourages deeper relationships and eases the logistics for out-of-class learning. Rotating learners through multiple teams enables a greater richness of multiple perspectives and broader networking opportunities. It’s a tradeoff.

31. Using One Handset for a Group or Team

Although one of the prime benefits of handsets is that every learner is encouraged to think and respond, handsets don’t have to be used only in a one-person one-handset format. Sometimes a greater number of audience members show up than expected. Sometimes budgets don’t allow for the purchase of handsets for every learner. Sometimes learners forget to bring their handsets. In addition, sometimes there are specific interactions that are more suited to group responding. When a group of learners has to make a single response, there has to be a mechanism for them to decide what response to make. Several exist, each having their own strengths and weaknesses.

32. Using Questions in Games

As several sales representatives have told me, one of the first things instructors ask about when being introduced to a particular audience response system is the gaming features. This excitement is understandable, because almost all classroom audiences respond energetically to games. Our enthusiasm as instructors must be balanced, however, with knowledge of the pluses and minuses of gaming. Just as with grading manipulations, games energize learners toward specific overt goals—namely scoring well on the game. If this energy is utilized in appropriate mathemagenic activity, it has benefits. On the other hand, games can be highly counterproductive as well.

33. Questions to Narrow the Options in Decision Making

Sometimes the audience in the room must make decisions about what to do. For example, a senior manager running an action-learning group may want to take a vote about which project to pursue given a slate of 15 possible projects. A professor in an upper-level seminar course might give students a vote in deciding which of the 10 possible topics to discuss in the final three weeks of the course. A supervisor might want her employees to narrow down the candidates for employee of the year. A primary school teacher might want to give her students a choice of field trip options. Audience response systems can be used in two ways to do this, single round voting and double round voting.

34. Questions to Decide Go or No Go

Sometimes it’s beneficial to give our learners a chance to decide whether they’re ready to go on to the next topic. You might ask, “Are we ready to go ahead?” Or, “Are we ready to go ahead, or do I need to clarify this a bit more?” Using an audience response system has distinct advantages over handraising here because most learners are uncomfortable asking for additional instruction, even when they need it.

35. Perspective-Taking Questions

There are some topics that may benefit by encouraging learners to take perspectives of others in answering questions. In other words, instead of only asking our learners to express their opinions, we can ask them to take a guess as to the opinions of others. For example, we might ask our learners to guess the opinion of both rich and poor people to affirmative action, the importance of education, etc.

36. Open-Ended Questions

Some people think that audience response systems lack potential because they only enable the use of multiple-choice questions. In contrast, the research on learning suggests to me that (a) multiple choice questions can be powerful on their own, and (b) variations of multiple-choice questions add to this power, and (c) open-ended questions can be valuable in conjunction with multiple-choice formats, for example by letting learners think first on their own, providing student ideas, providing more authentic retrieval practice, etc.

37. Matching

Matching questions are especially valuable if your learning goal is to enable learners to distinguish between closely related items. The matching format can also be useful for logistical reasons in asking more than one question at a time. Although the matching question has its uses, it is often overused by instructors who are simply trying to use non-multiple-choice questions. Often, the matching format only helps learners reinforce relatively low-level concepts, like definitions, word meaning, simple calculations, and the like. While this type of information is valuable, it’s not clear that the classroom is the best place to reinforce this type of knowledge.

38. Asking People to Answer Different Questions

Some audience response systems enable learners to simultaneously answer different questions. In other words, Sam might answer questions 1, 3, 5, 7, and 9, while Pat answers questions 2, 4, 6, 8, and 10. This feature provides an advantage only when it’s critical not to let (a) individual learners cheat off other learners, or (b) groups of learners overhear the conversations of other groups of learners. The biggest disadvantage to this tactic is that it makes post-question discussions particularly untenable. In any case, if you do find a unique benefit to having learners answering different questions simultaneously, it’s likely to be for information that is already well learned—where in-depth discussions are not needed.

39. Using Models of Facilitated Questioning

In the paper that details these 39 question types and methods, I attempted to lay bare the DNA of classroom questioning. I intentionally stripped questioning practices down to their essence in the hope of creating building blocks that you, my patient readers, can utilize to build your own interactive classroom sessions. For example, I talked specifically about using prequestions to focus attention, to activate prior knowledge, and to surface misconceptions. I didn’t describe the myriad permutations that pre- and postquestions might inhabit for example, or any systematic combinations of the many other building blocks I described. While I didn’t describe them, many instructors have developed their own systematic methods—or what I will call, “Models of Facilitated Questioning.” For example, in the paper I briefly describe Harvard Professor Eric Mazur’s “Peer Instruction” method and the University of Massachusetts’s Scientific Reasoning Research Institute and Department of Physics’ “Question-Driven Instruction” method.

Click to download the full report.

Learning Landscape Model

,

One of the features of the book is that it will provide a comprehensive model of workplace learning and performance. This model can be used in many phases of our work, from design through to evaluation.

Or click to watch a larger version (with more viewing control) directly on YouTube: Learning Landscape Model.