The Learning Show, “Don’t Forget Forgetting,” Now with a Lesson Plan

The Learning Show Episode, "Don’t Forget Forgetting," has been available on the web for several months. About 1400 people have accessed the program. In fact, some training departments seem to be sending their whole staff to view it, based on the large numbers from some organizations.

Now, thanks to the initiative and generous contribution of Jim Wright of the Instructional Development and Research department at Thompson Rivers University (TRU), you can get your department or work team involved in an organized effort to utilize these critical learning-design principles.

Check out Jim’s thoughtful accompanying "lesson plan" to Episode 1 of the Learning Show by clicking on the link below.

The document is made available to you, for your personal use, courtesy of Thompson Rivers University.

Download DontForget_LessonPlan.pdf.

Thanks Jim !!!

Learning Research Quiz Results 2002-2007

Since 2002, I have offered the Learning Research Quiz on the Work-Learning Research website. Over 1200 people have taken that quiz. 991 people answered all 15 questions. There are 15 questions on the quiz. A blind monkey randomly guessing the answers would score about 24% (there are three to seven answer choices per question).

The questions are scenario-based questions presenting relatively realistic learning-design situations. The answer choices are plausible—there are no throwaway answer choices. The questions are thus not easy to answer. They are specifically designed to measure people’s understanding of fundamental learning principles like repetition, spacing, feedback, retrieval practice, instructional objectives, etc.

People who answered the questions are self-selected from visitors to the Work-Learning Research website so we can’t really be sure who this sample represents. In general, it is my experience that most people who come to the website are experienced learning professionals who are passionate about learning, training, teaching, and/or instructional design.

Here are the results for the 991 people who answered all 15 questions:

If 24% is a completely random score, what should we expect from those who take the quiz? Would you expect them to do twice as well (around 50% correct?) or do better, say 60%, 70%, 80%, 90%, or 100%?

On average our 991 participants scored 32% correct, barely better than chance responding!!

Scoring 1% to 25% correct were 279 respondents.
Scoring 25% to 50% correct were 610 respondents.
Scoring 50% to 75% correct were 101 respondents.
Scoring 75% to 100% correct was 1 respondent, who scored 80% correct.

Association affilitation had little impact on average correct:

ISPI members scored 36% correct.
ASTD members scored 33% correct.
Other trade association members 28% correct.
Research association members 30% correct.
Non-affilitated respondents scored 31% correct.

Age of respondents had little impact on average correct:

Under 21 years of age scored 26%
21-25 scored 28%
26-35 scored 32%
36-45 scored 33%
46-55 scored 30%
56-65 scored 35%
66-75 scored 33%

Education degree had little impact on average correct:

"Ph.D. in psychology, learning, etc." scored 35% correct.
"Ph.D. in other discipline" scored 34% correct.
"Masters in psychology, learning, etc." scored 33% correct.
"Masters in other discipline" scored 32% correct.
"Bachelors" scored 31% correct.
"Other" scored 29% correct.

Job Title had little impact on average correct:

"Trainer" scored 32% correct.
"Instructional Designer" scored 33% correct.
"Performance Consultant" scored 33% correct.
"Human Performance Technologist" 38% correct.
"E-Learning Specialist" scored 29% correct.
"Learning Technology Developer" scored 33% correct.
"Teacher" scored 31% correct.
"Professor (with little research activity)" scored 33% correct.
"Learning Researcher (professors too)" scored 31% correct.
"Manager of Instructional Development" scored 34% correct.
"Manager of Training" scored 34% correct.
"Student" scored 28% correct.
"Other" scored 29% correct.

Date of Taking the Quiz had little impact on average correct:

2002 scored 30%
2003 scored 33%
2004 scored 31%
2005 scored 33%
2006 scored 30%
2007 scored 33%

Conclusion

Because the sample of respondents is difficult to define, any conclusions must remain speculative. Still, the results suggest a massive competence gap. The 32% average score—and the stubborn lack of improvement regardless of experience, education, and age—suggests that most people in the learning-and-performance field are unprepared for roles as designers of learning, at least as far as their ability to apply knowledge of learning research.

Links

Learning Show — Don’t Forget Forgetting

Welcome to The Learning Show — Don’t Forget Forgetting discussion page.

To see the episode, click here.

Description: This episode provides a provocative look at the importance of “forgetting” in designing learning interventions. My intention is that this will be a wake-up call for our industry. Too many of us forget to design our learning interventions to minimize the ravages of forgetting. In this 25-minute online presentation, I briefly cover the research and rationale, and then examine the many implications—from design to deployment to assessment. This episode presents fundamental concepts in learning that all of us ought to know and utilize in our work as learning-and-performance professionals.

To access this episode on your mobile device (the quality is diminished), copy and paste this hyperlink: http://www.brainshark.com/worklearningresearch/mobile/164722759.wmx.

To comment, click the comment link below.

Ethics in Internet Selling and Advertising

Newspapers and magazines have recognized for many years that there must be a clear distinction between content and advertising in the perceptions of readers. These entities not only have clear standards about these distinctions, but they also enforce a clear delineation between their sales departments and their news departments so that their sales efforts cannot influence (bias) their news reporting.

It is similar in the TV news business. Even as recently as last year, news organizations have been chastened with public outrage when Bush Administration video propoganda—created to look like real news broadcasts—was used by several news organizations as if it was an actual independent news clip. See the following article for details, for example (access costs money), THE MESSAGE MACHINE: How the Government Makes News; Under Bush, a New Age of Prepackaged News. By DAVID BARSTOW AND ROBIN STEIN; ANNE E. KORNBLUT CONTRIBUTED REPORTING FOR THIS ARTICLE. NY Times: March 13, 2005

News organizations don’t follow these ethical practices only because they believe in their sanctity. They follow these practices because the general public becomes outraged when news is seen as biased by financial interests or the influence of powerful elites.

Unfortunately, on the web—where the general public is being sorted into smaller and smaller subgroups—it is easier for organizations to deceive these micro-publics into believing that commercial messages come from independent unbiased sources.

With this in mind, I offer the following minimum standards of ethics for internet sellers and advertisers:

  1. Almost everyone who views your advertising should understand immediately that it is an advertisement.
  2. It is more ethical to offer independent content than sponsored content.
  3. It is more ethical to offer both independent and sponsored content than it is to offer sponsored content alone.
  4. It is more ethical to offer sponsored content and independent content if the financial benefits of the sponsored content are required to enable the production of the independent content.
  5. Readers/viewers must be able to immediately recognize which information is sponsored and which information is independent.
  6. Almost everyone who views sponsored content (web pages, white papers, articles, awards, best-of lists, etc.) should understand immediately that the information is being presented to sell or persuade the readers or viewers.
  7. The development of independent content should not be influenced by sponsoring entities.
  8. Authors must clearly acknowledge any relationships that may bias their work.

What else am I missing?

What do I have wrong?

What evidence of this do you see in the Workplace Learning-and-Performance Field?

What else, specifically, do we have to worry about? Any of the following (I’m just brainstorming)?

  • product placements in e-learning?
  • trade organization behaviors?
  • helping our learners distinguish truth from sales pitches?
  • conferences that offer sponsorships?
  • webinars that are sales pitches masquerading as information?
  • gurus that have vendor money in their pockets?
  • online communities based on sponsorship money?
  • white papers?
  • award programs?
  • vendor research?

Fair-Assessment Quick-Audit

I’m a strong advocate for assessment. If we don’t assess our learning interventions, (a) we as instructional designers don’t learn ourselves, (b) we don’t have valid data to give ourselves feedback, and (c) we can’t possibly improve our learning designs.

If we’re going to validly assess our learning interventions, we have to understand human learning and beware of biasing our results. Some of the things we have to watch out for:

  1. Testing our learners only when information is top-of-mind.
  2. Testing our learners in the learning context.
  3. Testing our learners unfairly with biasing pretests.
  4. Testing our learners with stupid, irrelevant questions.
  5. Using Level 1 smile-sheet data exclusively.
  6. Measuring with post-hoc metrics.

To help folks in the field avoid some of these pitfalls, I’ve developed the Fair-Assessment Quick-Audit courtesy of LearningAudit.com and Work-Learning Research.

Download The_Fair-Assessment_Quick-Audit.pdf

Some of these criteria are critical, especially because we in the field tend to do the exact opposite of what is fair and valid.

See this (click here) recent post that describes the mistakes we’re making in assessment.

Internet Changes Sex-Advice Column

Dan Savage, noted syndicated sex-advice columnist (warning: graphic discussions), said recently on the radio show Infinite Mind that through the years he’s seen a change in the questions people are asking him. No longer are they asking him simple questions about what a particular sex act is, how to do it, etc. He’s says the internet has changed what people want to know about. They can quickly google the basic stuff online. Now they want to know more about the deeper stuff, the relationship stuff, etc.

Hmmm.

Is there a parallel for employee-training situations?

Do we need to embrace deeper, more emotional, and/or more penetrating learning methods?

  • Coaching?
  • Mentoring?
  • In-depth Articles?
  • Questions and Answers from Experts?
  • Group Discussions
  • Collaboration?
  • Management Involvement?
  • Online Communities?
  • Interpersonal Network Analysis?

Does this relate to some content and not others?

  • IT Training
  • Soft Skills
  • Onboarding
  • Business Acumen
  • Ethics and Safety

Or, should we make sure we do a new round of needs assessment to explore how our learners are getting their information, what information they’re getting already, what information they still need?

Or, should we just remember to ask ourselves, "What can we do to make training/learning sexy?"

m-Learning Tsunami

I was in a video conference today with the Chicagoland Learning Leaders, and out of my mouth came the following sentence (I’m paraphrasing):

"The m-Learning tsunami is coming."

I know where this phrasing came from. It came from a research report I wrote on e-learning a number of years ago. Here’s what I wrote back in 2002:

E-learning is churning the learning-and-performance field like a great tsunami hovering high above, a dark blue wave threatening everything that has come before. As those on the shore—trainers, instructional designers, performance consultants—run furiously in all directions, everybody wants to know:

1. Can this e-learning stuff work?
2. For what uses is e-learning most appropriate?
3. Will e-learning replace traditional learning approaches?
4. How can e-learning be designed to be effective?

Replace the "e" with an "m" and the same questions still apply.

The point I made today in the video conference was that m-learning is on its way, that its going to become its own force of nature and that we’re going to feel a lot of pressure to ride the wave.

Just like all learning technologies, we need to remember the fundamentals. Especially relevant:

  1. First determine the learning outcomes you hope for (ignoring the m-learning technology).
  2. Then determine the best way to reach those goals (considering m-learning as one of a hundred options).
  3. When implementing a new learning technology, rapidly develop some prototypes.
  4. Remember to base all your designs so that the learning intervention aligns with the human learning system. Utilize research-based instructional design.
  5. Test your rapid prototypes in a meaningful way (measure learners ability to make decision, measure changes in job performance, measure business results).
  6. Keep what works, rolling it out in a bigger way. Keep testing to see if the roll-out works as well.
  7. Lather, rinse, repeat.

Yes, it’s obvious. The m-Learning Tsunami is coming.

  • The installed base of MP3 players is growing exponentially
  • Cell phones are becoming usable for learning
  • PDA’s and small Tablet PC’s are being deployed
  • m-Learning is one of the hottest conference sessions right now
  • Vendors are investing in R&D, and more vendors are focusing on m-Learning.

And yes, we’ll all be swept up in it. Many of us will drown. Some of us will have the ride of our lives.

Most of us will do best to remember the fundamentals.

m-Learning is a bunch of different tools. We have to learn when and how to use them—and when not to use them.

We need to pay attention to WHAT’S NEW, but remember WHAT’S TRUE.

Emerging E-Learning Technologies Reports

Dr. Gary Woodill, Senior Researcher at Brandon-Hall Research has recently released the second in his three-part series on emerging e-learning technologies. These reports briefly cover the emerging technologies and methods, and then—for each topic—provide an exhaustive set of references and web links.

I have to admit to being amazed at the breadth and scope of these reports. These reports won’t be for everyone—they’re much too packed with reference-like information—but for someone like me who likes to see what I’ve failed to learn as of yet, they’re wonderful.

I’d recommend these reports for folks who are responsible for their company’s Learning R&D efforts, for those interested in brainstorming new e-learning design options, and for academics and researchers who want to see what the new areas are.

These reports aren’t perfect, of course. I wouldn’t expect such an exhaustive research effort to be flawless.

The reports don’t provide an organizing framework for understanding how these technologies might interface with the flesh-and-blood machinery of human learners. The danger is that readers who willy-nilly act to employ these technologies may not get the learning benefits they might expect—or, they may create negative outcomes.

Some technologies were not described at all, although they may be in the third report. One e-learning technology I’m passionate about wasn’t mentioned—learning-follow-through software, which facilitates training transfer by encouraging learners to implement their after-training goals, by getting managers involved in the process, by recording actual work follow-through, and by keeping the learning conversations going after the formal learning events are finished.

I’ve written about this technology before, both in a software review and a book review.

This software is a paradigm buster—one of e-learning’s killer aps—but these reports don’t mention the technology at all. See Friday5s from the Fort Hill Company and ActionPlan Mapper from ZengerFolkman as the two exemplars of this technology.

To reiterate, the weaknesses of these reports are minor in comparison with their strengths. For those who really want to delve into what’s out there, Gary Woodill has provided a great service in these reports.

  1. Emerging E-Learning: New Approaches to Delivering Engaging Online Learning Content
  2. Emerging E-LearningTechnologies: Tools for Developing Innovative Online Training

One of the reasons information like this is so critical is that too many of us associated with e-learning have not really pushed ourselves to develop models beyond the page-turner, the multiple-stupid interaction page clicker, the game show, the webinar, and the branching simulation.

We need to be more creative—testing our ideas of course in the crucible of the real world—but really working on alternative models. Compilations of ideas (like these reports provide) are very helpful in this regard, again with the caveat that these technologies absolutely have to be designed with the human learner in mind and tested for learning effectiveness.

Web 3.0 and Learning

The NY Times had a nice article in last Sunday’s edition—on the front page above the fold—on the concept of Web 3.0, which may have implications for our field.

To give you a sense of what Web 3.0 is, here are some quotes from the article:

  • "[The Web 3.0] goal is to add a layer of meaning on top of the existing Web that would make it less of a catalog and more of a guide — and even provide the foundation for systems that can reason in a human fashion."
  • "But in the future, more powerful systems could act as personal advisers in areas as diverse as financial planning, with an intelligent system mapping out a retirement plan for a couple, for instance, or educational consulting, with the Web helping a high school student identify the right college."
  • "[The holy-grail of Web 3.0 developers] is to build a system that can give a reasonable and complete response to a simple question like: ‘I’m looking for a warm place to vacation and I have a budget of $3,000. Oh, and I have an 11-year-old child.’"

The goal then is to be able to analyze the information from the web and come up with quick and meaningful responses to queries people ask.

Advertisement Work-Learning Research

Hmmm. That’s sort of a learning application in a way.

And, if we can create such systems, why couldn’t we ask a query like, "I want to become a CLO at a socially responsible company, and I’m currently an instructional designer with an undergraduate degree in humanties and an MBA, plus 5 years experience as a leadership trainer. What do I have to do reach my goal and what do I have to learn?"

What would Web 3.0 mean for our field?

Training Media Review Authoring Tools

Training Media Review reviews are now available on both a subscription basis ($229 per year) or a daily basis ($25).

The current issue reviews authoring tools such as Lectora, Captivate, Articulate, Camtasia, and ViewLet Builder. You can check out all these reviews for only $25 by buying a one-day subscription.