Last week I spoke at one of the major conferences in the workplace learning-and-performance industry to a room filled with about 150 people. I am not including the name of the conference organizer because I don't want to single them out. They are not alone in the going-paperless practice.

Unfortunately, two of the conference decisions—(1) not to provide participants with paper handouts—(2) not to allow electronic handouts to include sufficient pages to provide images of the slides that are shown—may have hurt learning results. Perhaps ironic in a conference devoted to help people build better learning interventions.

Setting the Scene of My Presentation

I spoke on the topic of Situation-Based Learning Design and the session seemed to be well-received. I got the following emails afterward:

  • "I found your session to be amazing!"
  • "One of the highlights of the conference for me."

One of my friends, who sat in the audience, told me afterward that he knew the session was good because (1) hardly anybody left even after 90 minutes, and (2) many people were furiously taking notes.

But was it a successful learning experience? Isn't that the key question?

I could have been entertaining. People could have liked the session. They could have taken lots of notes. But probably the most important reason they came to the conference was so that they could go back and do something better and to think differently because they had attended.

I used some learning-design techniques to help them remember—to help them integrate the new learning with their current conceptions—but was this enough?

I think something was missing because of the paperless practices, but before I get to that, let me recount some of the design tactics I used specifically to ensure remembering and application to their work.

  1. Content was verified (by research) to be effective in supporting good work practices around learning design. Some stuff you learn at conferences can actually make you less effective.
  2. Participants were provided with realistic learning-design decisions to make before topics were introduced. By activating prior knowledge and situating the learning in situations similar to those folks would face in their work as learning professionals, the learning would become more usable and more memorable.
  3. Examples of real-world e-learning courses were provided to again link realistic context cues to the learning points.
  4. Learners were encouraged to specifically engage in trigger setting (what researchers call implementation intentions) to increase the likelihood that they would take action in implementing what they learned
  5. Participants were presented with spaced repetitions of key learning
    points to support long-term remembering. Repetitions were achieved by
    providing (a) prequestions, (b) research explanations, (c) supporting
    metaphors, (d) real-world examples, (e) follow-up practice, (f) similar
    topics that reinforced points from previous topics, (g) a final
    summary, and (h) a reminder to link the learning to their future
    situations.

Good stuff to be sure, but was anything missing?

I think so.

The biggest thing that is missing is super-strong links back to the workplace. Even though I use great long-term retrieval support (better than most conference sessions provide I would think), having our learners rely only on the learning-remembering link when transportable cues are available is committing learning-to-performance malpractice. If learners could take their annotated slide images back to the workplace, they would remember more, share more, and apply more of the session content.

The Problem of Going Paperless.

Note-taking can prompt learners to engage in deep processing of their learning. By organizing the learning content during note-taking, learners will be more likely to fully understand what they're learning and remember it later. Note-taking has other benefits as well. Learners can go back and study their notes at a later time, providing themselves with spaced repetitions—a proven aid to long-term remembering. Finally, learners who take notes may be more likely to attempt to share what they've learned with others when they return to work. The process of preparing to teach others, provides further deep processing of the learning material and makes it more likely that the learning content will be remembered and utilized on the job.

When a conference goes paperless, it is likely to curtail note-taking. When I asked my 150 learners how many of them had paper copies or were looking at the handouts on their laptops, less than half raised their hands.

Just because participants can print out the handouts beforehand, doesn't mean that they will. This is a lame excuse for conference organizers.

Taking Notes with Slide Images versus Taking Notes on Plain Paper

The conference organizer only allowed us 8 pages of handouts. I could only fit 12 of my slides into these 8 pages, because some other things were required as well. For me, this meant that more than 85% of my slides were NOT provided to my learners.

Ironically, one of the research streams I talked about in my session was the importance of context in learning and remembering. Specifically, research is very clear that if stimuli are available in both the learning context and the retrieval context, people will be better able to remember what they learned. The contextual cues in the retrieval situation remind the learners of what they learned.

Slide images are such stimulus cues. If learners can transport slide images (and any notes they've added to those images) from the learning situation (my session) to their work situation, they will remember more of what they learned. So, conference organizers who deprive their participants (aka delegates) of slide images and concomitant note-taking are hurting learning.

This is compounded because learners will also be less likely to take back their learning to their coworkers. Just yesterday I did a web meeting with a prospective client and last night I got a request from that prospective client to get copies of my slides. They said they took notes, but they wanted the slides to ensure they didn't miss anything and also to prepare to persuade others in their organization.

Finally, if you've ever looked at a copy of somebody else's slide deck, you know how hard it is to make sense of it without the context provided by aural narration or annotations. People need to be able take notes so that they will later have the full context of the slides. Research also shows that labels, annotations, and notes must be geographically close to the objects of those textual elements—so that limited cognitive resources are not overtaxed, hurting learning. Bottom line here: Notes near the paper on slide images are more effective than notes taken without slide images.

The Push To Paperless

The paperless movement is bound to continue. It cuts into profits for conference organizers to print handouts. It goes against the rising tide of the green environmentally-friendly zeitgeist (which I am very sympathetic to).

If everyone brought their laptop to each session and had the slides available on their screens, they could take notes using Adobe Acrobat's commenting features, or perhaps some other program. What is needed is the ability to put a comment or note where you want it to be, and draw simple diagrams in case there is some visual meaning that should be captured. Note that PowerPoint, even the latest 2007 version, has no such capability. 

One must wonder if the environmental costs of laptop energy-use might outweigh the environmental costs of paper printing, shipping, etc.

If organizers expect participants to print their own copies, than there doesn't appear to be an environmental benefit. Just the costs are shifted to the conference participant.

I have found that I am sometimes able to annotate my own slides so that my audience members don't have to—this would create a minor learning benefit perhaps—but it doesn't really solve the underlying issues.

Academic conferences often have session-leaders write short papers that conference participants can access for later reading. This might be a way to transfer more of the learning content back to the workplace—but could we really make this cultural shift. Also, such short papers would provide a strictly post-learning intervention. It wouldn't fully compensate for the loss of context and note-taking.

Summing it Up

When conferences go paperless:

  1. They either hurt learning, remembering, and on-the-job application,
  2. Or, they create minor environmental improvements.
  3. With current state of installed technology usage, they can't create good learning impacts and good environmental impacts (most people don't have a way to take notes on their slides on their computer).
  4. Conference organizers save themselves a heap of money, increasing their profits, which probably doesn't trickle down to conference participants in the form of lower registration fees, especially if participants have to print and transport handouts to the conference and back by themselves.
  5. You as a learner have to make a conscious decision to support your conference learning, and you can do this by getting a PDF of the slides beforehand and either printing them out or using Adobe Acrobat to take notes on the slides on your laptop.
  6. If you as a conference goer want to maximize learning, you can contact conference organizers and complain, or vote with your dollars by not attending conferences that don't support your learning.
    • Tell them: You want slide images.
    • Tell them: You want a way to take notes on those slide images.

Scientists may have found how exercise induces calm–and how it may help people in stressful situations perform better.

Check out this article from the NY Times…

Given that lots of us are stressed these days–and even the relaxed folks have to occasionally make a big stressful presentation, etc.–enabling people to exercise might give them a competitive advantage.

I got an email today from someone asking me about a term I created called “Evaluation Objectives.” I realize that I have not actually written anything for public consumption on this, SO this blog post will suffice until my book on workplace learning is released. Apologies if the following is not completely clear.

The basic idea is that we ought to have evaluation objectives rather than learning objectives in the traditional sense.

Specifically, we need to decouple our learning objectives from our evaluation objectives so that what we evaluate is directly relevant. Of course our evaluation objectives and learning objectives have to
be linked, but not necessarily with a one-on-one correspondence.

AN EXAMPLE

Suppose you want to train managers to be better at championing change efforts.

Traditionally, we might have objectives like:

The learner will be able to describe how people tend to resist change.

Or, put in a more performance-oriented fashion, a traditional objective might read:

The learner will engage in activities that lessen colleagues’ resistance to change.

Examples of evaluation objectives might be as follows:

The learner will initiate a change effort within one month after the training ends and be successful in getting 75% of his/her colleagues to sign a public statement of support for the effort.

OR, if real-world compliance cannot be assessed, an evaluation objective might be something like:

2. In the “Change-Management Simulation” the learner will score 65 points out of a total possible of 90.

OR, if a simulated performance can’t be created, an evaluation objective might focus on ratings by employees.

3. Two months after the training ends, the learners’ colleagues will rate them on average at least 4.5 (of 6 levels) on the multi-rater 360-degree change-management scale on each of the 5 indices.

OR, if this can’t be done, an evaluation objective might focus on a series of scenario-based questions.

4. On the 20-question scenario-based quiz on change management given two weeks after the course ended, the learner will get at least 17 correct.

NOTE: More than one evaluation objective can be used for any learning intervention.

THE POINT:

Evaluation objectives are NOT tied to individual learning points that have to be learned, though of course they are linked because both should be relevant to the overarching goals of the learning program.

THE BIG BENEFIT:

When objectives focus on the big picture, as compared to when there is a one-to-one correspondence between learning objectives and evaluation items, (1) they are more relevant, (2) the learners are more likely to see them as valuable and worth achieving, (3) organization stakeholders are more likely to see the evaluation results as having face validity, (4) the evaluation results will give us additional pertinent information on how to improve our learning interventions.

Dear President Obama,

You're a technophile I have heard. So, I have an improvement to suggest for the FDA, particularly how it deals with food-safety issues.

Here's what the FDA does now.

In the age of web technology, the FDA's methodology is just plain laughable.

I propose a webpage with a database that would enable citizens to submit food-safety alerts.

This should be damn simple. The post office has a list of all addresses in the country. Why can't the FDA create a list of all foods sold in the U.S. plus a list of all food sellers (grocery stores, restaurants, etc.).

Consumers who suspect they have some bad food could go online and within a few clicks select their product and where they bought it from. They could describe the issue, etc.

In the background, the system would monitor products for unusual activities (larger than normal number of alerts) and create an alerting response when something looks wrong.

If the FDA doesn't have the wherewithal to design and create such a system. I would be glad to take this on with my strategic partner Centrax Corporation (they build high-premium e-learning and web programs and could whip this up no problem).

Seriously, the FDA could save lives very simply and at a relatively low cost. Let's just do it.

Thank you Mr. President for considering this.

Please let me know what I'm supposed to do with the yogurt in my refrigerator that tastes bad. If you think I'm going to call one of those numbers, you just don't get it.

–A worried citizen/consumer

Update Thursday April 16th

Yesterday I decided I should make those calls. I called the yogurt manufacturer and went to their website and I called my regional FDA hotline person (who called me back today, a day later). Stoneyfield Farm has posted the following recall information (their phone complaint line was horribly implemented with long wait times and no one has gotten back to me from their online complaint system):

Londonderry, NH – April 3, 2009 Stonyfield Farm is
conducting a voluntary recall of Fat Free Plain Quarts in Stonyfield
Farm branded containers limited to specific dates. The products are
being recalled because they may contain a presence of food grade
sanitizer.

Affected products are limited to Stonyfield Farm 32 ounce Fat Free
Plain yogurt UPC # 52159 00006 carrying one of the following product
codes printed along the cup bottom that start with the following date
codes:
· May 06 09 Time stamped 22:17 thru 23:59 (limited to these specific time stamps only)
· May 07 09 All time stamps

Approximately 44,000 quarts were distributed to retail accounts nationally.

We have received several reports of people noticing an off-taste
when eating the product. We have received no reports of illness of any
kind after consuming the product.

The issue was a result of human error in not following our Company's
standard operating procedures. Stonyfield has taken all the necessary
corrective action to prevent this from occurring again.

Consumers are advised not to consume the product and to return
opened and unopened containers to the store where it was purchased.
Anyone returning these products will be reimbursed for the full value
of their purchase.

Customers with questions should contact Stonyfield Farm Consumer Relations at 1-800-Pro-Cows (776-2697) or visit our website at www.stonyfield.com.

This is listed on their website when I checked today. I didn't notice it yesterday (they have a very busy home page), but it probably was there.

Note to Stonyfield Farm: 

I am not satisfied with your announcement stating, "We have received several reports of people noticing an off-taste
when eating the product. We have received no reports of illness of any
kind after consuming the product."

THAT IS NOT GOOD ENOUGH!! You should (1) tell us what we ingested, (2) get health experts to provide us with some expert guidance on what symptoms or dangers we might be subject to.

More:

I just called Stonyfield Farm Consumer Hotline again (and actually got through to them today) and the guy said it was a Food-Grade Sanitizer, FDA approved, organic, etc. He told me ingesting it wouldn't hurt me, but I'm not convinced. I told him I wanted to know what it was I ingested. He wouldn't or couldn't tell me. I asked him if I ate a whole container whether it would hurt me…He said no.

Hey Stonyfield. You can do better…

Last year I wrote at length about my efforts to improve my own smile sheets. It turns out that this is an evolving effort as I continue to learn from my learners and my experience.

Check out my new 2009 version.

You may remember that one of the major improvements in my smile sheet was to ask learners about the value and newness of EACH CONCEPT TAUGHT (or at least each MAJOR concept). This is beneficial because people respond more accurately to specifics than to generalities, they respond better to concrete learning points than to the vague semblance of a full learning experience.

What I forgot in my previous version was the importance of getting specific feedback on how well I taught each concept. Doh!

My latest version adds a column for how well each concept is taught. There is absolutely no more room to add any columns (I didn't think I could fit this latest one in), so I suppose this may allow diminishing returns on any more improvements.

Check it out and let me know what you think.

Okay, I made these suggestions on Twitter today, but because it's so ephemeral, I RT them here:

  1. A Better Twitter Query: "What's happening for you?
  2. Use "MT" when you are suggesting a link to your own stuff.

1.
The Better Query encompasses the original query (still in use today): "What are you doing?" BUT it also conveys the way a majority of my Twitter contacts use Twitter, to convey what they find exciting, useful, notable. Sure, we have to go back to the 60's and 70's "What's happening man?" but those times weren't all bad.

2.
MT=Me Tweet. The MT idea helps people know whether the Tweeter is plugging their own work. This is useful in many ways. If you really like what someone you're following has to say in their longer off-Twitter conveyances, then you'll want to go there (and vice versa). It also enables the Tweeter to follow common interpersonal traditions by enabling spam-warnings. For example, in normal conversations we might say the following while looking apologetic, "Well, I know there are lots of perspectives on this, but here's my thoughts…" thus lubricating the social dialogue. Finally, some smart Twitter programmer will come up with a way to measure MT's and then self-promoters can be labeled as such.

Please RT (Re-Tweet) these ideas if you like them.

(1290 characters) about 9 tweets.

Robert Gagne's 1st event of instruction was "Gain Attention." Michael Allen's company, Allen Interactions, has been saying for years, "No More Boring e-Learning." We've all heard the stories of how often e-learning turns learners off. And yet, there is still a whole lot of boring e-learning out there.

An article from the eLearning Guild helps us avoid the trap, specifically by helping us start our learning interventions in ways that grab attention. Paul Clothier interviews Carmen Taran, author of the book Better Beginnings.

Dan Balzer and Susan Manning offer an excellent Podcast on the topic. You can find the link to the Guild article at their web page as well.

Nicholas Kristoff, Op-Ed Columnist for the New York Times has a problem. He wants to tell people about the genocide in Darfur but people tune out after a while.

Click here to see his latest attempt to keep our attention.

Click here to see his recent work on this topic with George Clooney.

Advice for Learning Professionals

As a learning professional, have you ever utilized "celebrity" to grab your learners' attention? I'm not necessarily talking about movie stars. What about a well-respected person in your company? Your CEO?

Of course, it's not always easy to get this right, but it's a tool we ought to have in our toolbox.

By the way, which link did you click first above? They're both the same…

Since 1998 when I started Work-Learning Research, I've been trying to spread the word about research-based principles of learning. I naively thought that good information would resonate so much that it would change practices industry wide.

I've largely failed in that endeavor up to this point.

That's okay. I've learned a simple truth about influencing others. It's hard.

Take two recent examples outside the learning field. Antibacterial soap and vitamins. It has been widely reported for about five years or so that using antibacterial soap is generally counterproductive. It has also been reported over the last two years or so that taking vitamins may produce no benefits and in some cases can be harmful (see today's article on vitamins).

I've sent many articles on these topics to my family and close friends. Many people just can't incorporate the new information into their old mindsets. We've learned for so long that germs are bad and vitamins are good that we think from those points of view. New information is deflected before it can become part of our new thinking.

As learning professionals, we know that "Telling Ain't Training" and "Training Ain't Performance" (thanks Harold), but we often forget that long-held views are not easily overcome. We need to be more careful and more energetic in confronting them. It's not our learners' fault when they don't make the turn. We have to make it our fault. We have to take responsibility.

Google has a nice blog post out on its use of eye movement research.

I remember getting a tour of Fidelity a few years ago and learning that their eye movement studies on web browsing showed that people were beginning to ignore big dark chunks of graphics because they thought they were advertisements.

My dissertation advisor, Ernie Rothkopf did a classic study (with Billington) in 1979 using eye movement data to test whether learners actually paid more attention (had more and higher-quality eye movements) toward information in the learning material that was targeted by learning objectives than to information that was not so targeted. It turned out that learning objectives worked to boost learning because they prompted learners to pay more attention to the objective-relevant material and less attention to the rest of the information.

See: Rothkopf, E. Z., & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive processing model from inspection times and eye movements. Journal of Educational Psychology, 71, 310-327.

To answer the question I posed above. Yes, more of us should be using eye-movement research to support us as we do e-learning design.

And by the way, as web pages change their strategies to gain our attention, our learners may change their strategies to avoid things deemed irrelevant. Moreover, as our learners see more and more of our company’s e-learning, their eyes may learn where to go…In fact, a lot of them already have a well-learned capacity to find the NEXT key through a swarm of bees.