Neuroscience and Learning

The Debunker Club, formed to fight myths and misconceptions in the learning field, is currently seeking public comment on the possibility that so-called neuroscience-based recommendations for learning and education are premature, untenable, or invalid.

 

Click here to comment or review the public comments made so far…

 

Click here to join The Debunker Club…

 

Original post appeared in 2011. I update it here.

Updated Article

When companies think of evaluation, they often first think of benchmarking their performance against other companies. There are important reasons to be skeptical of this type of approach, especially as a sole source of direction.

I often add this warning to my workshops on how to create more effective smile sheets: Watch out! There are vendors in the learning field who will attempt to convince you that you need to benchmark your smile sheets against your industry. You will spend (waste) a lot of money with these extra benchmarking efforts!

Two forms of benchmarking are common, (1) idea-generation, and (2) comparison. Idea-generation involves looking at other company’s methodologies and then assessing whether particular methods would work well at our company. This is a reasonable procedure only to the extent that we can tell whether the other companies have similar situations to ours and whether the methodologies have really been successful at those other companies.

Comparison benchmarking for training and development looks further at a multitude of learning methods and results and specifically attempts to find a wide range of other companies to benchmark against. This approach requires stringent attempts to create valid comparisons. This type of benchmarking is valuable only to the extent that we can determine whether we are comparing our results to good companies or bad and whether the comparison metrics are important in the first place.

Both types of benchmarking require exhaustive efforts and suffer from validity problems. It is just too easy to latch on to other company’s phantom results (i.e., results that seem impressive but evaporate upon close examination). Picking the right metrics are difficult (i.e., a business can be judged on its stock price, its revenues, profits, market share, etc.). Comparing companies between industries presents the proverbial apple-to-orange problem. It’s not always clear why one business is better than another (e.g., It is hard to know what really drives Apple Computer’s current success: its brand image, its products, its positioning versus its competitors, its leaders, its financial savvy, its customer service, its manufacturing, its project management, its sourcing, its hiring, or something else). Finally, and most pertinent here, it is extremely difficult to determine which companies are really using best practices (e.g., see Phil Rosenweig’s highly regarded book on The Halo Effect) because companies’ overall results usually cloud and obscure the on-the-job realities of what’s happening.

The difficulty of assessing best practices in general pales in comparison to the difficulties of assessing its training-and-development practices. The problem is that there just aren’t universally accepted and comparable metrics to utilize for training and development. Where baseball teams have wins and losses, runs scored, and such; and businesses have revenues and profits and the like; training and development efforts produce more fuzzy numbers—certainly ones that aren’t comparable from company to company. Reviews of the research literature on training evaluation have found very low levels of correlation (usually below .20) between different types of learning assessments (e.g., Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008).

Of course, we shouldn’t dismiss all benchmarking efforts. Rigorous benchmarking efforts that are understood with a clear perspective can have value. Idea-generation brainstorming is probably more viable than a focus on comparison. By looking to other companies’ practices, we can gain insights and consider new ideas. Of course, we will want to be careful not to move toward the mediocre average instead of looking to excel.

The bottom line on benchmarking from other companies is: be careful, be willing to spend lots of time and money, and don’t rely on cross-company comparisons as your only indicator.

Finally, any results generated by brainstorming with other companies should be carefully considered and pilot-tested before too much investment is made.

 

Smile Sheet Issues

Both of the meta-analyses cited above found that smile sheets were correlated with an r = 0.09, which is virtually no correlation at all. I have detailed smile-sheet design problems in detail in my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. In short, most smile sheets focus on learner satisfaction, and fail to focus on factors related to actual learning effectiveness. Most smile sheets utilize Likert-like scales or numeric scales that offer learners very little granularity between answer choices, opening up responding to bias, fatigue, and disinterest. Finally, most learners have fundamental misunderstandings about their own learning (Brown, Roediger & McDaniel, 2014; Kirschner & van Merriënboer, 2013), so asking for their perceptions with general questions about their perceptions is too often a dubious undertaking.

The bottom line is that traditional smile sheets are providing almost everyone with meaningless data in terms of learning effectiveness. When we benchmark our smile sheets against other companies’ smile sheets we compound our problems.

 

Wisdom from Earlier Comments

Ryan Watkins, researcher and industry guru, wrote:

I would add to this argument that other companies are no more static than our own — thus if we implement in September 2011 what they are doing in March 2011 from our benchmarking study, then we are still behind the competition. They are continually changing and benchmarking will rarely help you get ahead. Just think of all the companies that tried to benchmark the iPod, only to later learn that Apple had moved on to the iPhone while the others were trying to “benchmark” what they were doing with the iPod. The competition may have made some money, but Apple continues to win the major market share.

Mike Kunkle, sales training and performance expert, wrote:

Having used benchmarking (carefully and prudently) with good success, I can’t agree with avoiding it, as your title suggests, but do agree with the majority of your cautions and your perspectives later in the post.

Nuance and context matter greatly, as do picking the right metrics to compare, and culture, which is harder to assess. 70/20/10 performance management somehow worked at GE under Welch’s leadership. I’ve seen it fail miserably at other companies and wouldn’t recommend it as a general approach to good people or performance management.

In the sales performance arena, at least, benchmarking against similar companies or competitors does provide real benefit, especially in decision-making about which solutions might yield the best improvement. Comparing your metrics to world-class competitors and calculating what it would mean to you to move in that direction, allows for focus and prioritization, in a sea of choices.

It becomes even more interesting when you can benchmark internally, though. I’ve always loved this series of examples by Sales Benchmark Index:
http://www.salesbenchmarkindex.com/Portals/23541/docs/why-should-a-sales-professional-care-about-sales-benchmarking.pdf

 

Citations

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-357.

Brown, P. C., Roediger, H. L., III, & McDaniel, M. A. (2014). Make It Stick: The Science of Successful Learning. Cambridge, MA: Belknap Press of Harvard University Press.

Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183.

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93, 280-295.

Connie Malamed is The eLearning Coach, an intriguing podcaster, and the author of two fantastic books on visual design. Here I interview her in regards to her most recent book, Visual Design Solutions.

 

Here is the book:

Here is Connie:

1.

Will:
Connie, in your book, Visual Design Solutions: Principles and Creative Inspiration for Learning Professionals, your goal seems to be to help learning professionals utilize effective visuals to improve their learning outcomes. Indeed, you dedicate the book to “the hard-working creative learning professionals who want to make a difference.”

Tell me about your hope for the book and the importance visual design has for learning professionals?

 

Connie:
My three goals in writing this book were to: 1) prove that it is possible to improve one’s visual design skills without being an artist, 2) demonstrate the benefits of using visuals to enhance and amplify learning, and 3) raise awareness about the importance of aesthetics in a learner’s experience.

For those with normal vision, the brain processes more sensory information from the eyes than from any other sense. So learning professionals should expect that the visual aspect of instructional materials would be of great importance to comprehension, retention and the user experience. The good news is that anyone can become more competent in visual design by learning, applying and practicing the foundation principles.

 

2.

Will:
Many learning professionals enter the field with little or no background or experience in graphic arts, visualization principles, or aesthetics—and yet you declare in your book that “you do not need drawing talent to work as a visual designer.” First, let me ask you, “Why not?” Second, let me ask you what key skills people do need to be effective at the visual aspects of learning design?

 

Connie:
Early in my career, I met an excellent designer who didn’t know how to draw. He told me he wasn’t an illustrator. I was shocked. Since then, I’ve met and read about many designers who do not illustrate. Visual design involves the arrangement of images and text in graphic space. To be able to do this, one doesn’t need to render with a pen or pencil. Of course, it’s always nice to brainstorm ideas with a pencil and sketchpad, but visual concepts can be communicated using geometric shapes and stick figures.

The skills that I think people need for visual design competence can be learned. Here is my list:

  • An understanding of how to think about and solve visual problems.
  • Foundation principles of design, such as use of white space, establishing a visual hierarchy and appreciating typography.
  • Awareness of design in the world around you to see how others have solved visual problems.

 

3.

Will:
I noticed in your book that you begin with lots of research supporting the benefits of using visuals. But certainly visuals can also be used in such a way that causes harm. What are some of the problems involved in using visuals? What are some of the most common mistakes learning professionals make?


Connie:

Right. Like anything else, it takes thoughtfulness to come up with an effective visual design. I think common problems are: cluttering a layout with too many flourishes, using irrelevant graphics that are distracting, and splitting attention so that the visuals and text or activity are not well integrated. One way to avoid common mistakes is to get your work critiqued by peers, sponsors and potential users.

 

4.

Will:
In the book, you emphasize white space, and yet I’ll bet this is one of the hardest things for us instructional designers to get. Just as we tend to cram content into our curriculums and don’t leave enough time for learning, I’ve often seen designs that cram content into our visuals without thinking about white space. What is white space and what are the top three things learning designers should realize about it?

 

Connie:
White space is also known as negative space. It’s the area in a visual that does not contain any images or text. It even includes the area in between letters. Here are three tips about working with white space.

  1. Think of white space as another element, just like text and images. These three building blocks of design have to work together to create a clear communication. Without white space, you can’t have form and without form, you can’t have white space.
  2. As you design, if you begin to focus on the shape of the white space, you begin to bring it into the foreground perceptually. Become conscious of the white space and make sure the shape is pleasing and that it’s not broken up into tiny little pieces.
  3. White space gives a viewer’s eyes a place to rest and allows a design to breathe. So don’t be stingy with your white space. Let your designs have some spaciousness.

 

 

 

5.

Will:
I put a ton of time into creating PPT slides for my workshops and presentations—and I’ve developed some beliefs over the years that may or may not be true. Would you comment and critique my visual-design prejudices?

  1. Never use clip art; photos are cheap, easily available, and convey more credibility.
  2. Using a transparency fade (for example when you take a photo but gradually fade one side of it into the background) is good looking, adds credibility, and enables room for pertinent text.
  3. It is better to have one major learning point per slide with a nice supporting visual, than to offer many ideas on the same slide.
  4. Using objects with some gradient is almost always preferred to no gradient.


Connie:

  1. It depends on what you mean by clip art. Although you want to avoid the silly smiling characters, there are wonderful collections of illustrated and simplified vector drawing that you can use to represent concepts and objects. You can make an entire presentation or eLearning course using this minimalist and distilled style.
  2. Adding a transparency fade is a good way to be able to add text. But it’s not the only way and if it looks very feathered, it could look dated. Other ways to add text are to use a large 1024×768 photo (the size of the entire slide) and then overlay a slightly transparent rectangle on part of the photo where you want the text. Place the text within that rectangle and ensure there is enough contrast to read it.
  3. I think you’re probably right about one point per slide. Don’t tell anyone but sometimes I might put three related points on a slide. Maybe I’m being lazy.
  4. A gradient can give an object a 3D appearance, but it’s not always necessary. The flat design trend has moved away from full gradients and you will now probably see more designs with flat or solid-looking objects. I don’t think there’s a wrong or a right way to fill in objects, but as trends change, a viewer’s idea of what is aesthetically pleasing will change.

 

6.

Will:
You mention perspective in your book. First, tell us what it is. Second, because you recommend it “to add realism to a story or scenario,” could you tell us if there is a secret to searching for photos with perspective in photo databases (so we don’t have to search through an endless array of photos)?

 

Connie:
Perspective is a way to trick the eyes into perceiving three dimensions on what is really a flat surface. I don’t know of any great way to find images with perspective other than to type that word into your photo database. Such as “street perspective,” for example.

7.

Will:
You mentioned distilled graphics. What are they and when should we use them?

Connie:
Distilled graphics are simplified, schematic or iconic visuals that represent objects or concepts. We perceive and understand them quickly, similar to the images on road signs. I think it’s a good idea to use these when you want to get an idea across quickly. Also, using a distilled graphic like a silhouette works like a visual suggestion of what it represents without getting into the detail. Another suggestion is to use distilled graphics instead of bullet points, placed near the text to represent the concept or fact. It’s tough to explain without a visual accompaniment!

 

8.

Will:
Connie, maybe you can help me. When I look for photos, sometimes I find myself spending half an hour or more just to find an image I deem acceptable. Am I nuts? Please help me! How long should I spend looking for an image?

Connie:
Will, I’m going to guess that you are a little nuts, but not because you take so long to find photos. But yes, searching for photos is one of the most time-consuming aspects of this career. Most stock photo sites that weren’t specifically made for eLearning seem to have an advertising/marketing focus. The photographers still have that mindset, where rather than showing people in realistic situations, they show people smiling at the camera or cheering. There’s not enough diversity in the image choices either. I got so frustrated one day, I sat down and wrote an article about this problem: 21 Reasons Why Stock Photo Sites Make Me Cry.

 

9.

Will:
I have a new favorite font, but while it’s in Microsoft Office, when some of the online meeting tools convert it, they replace my beautiful artsy font with a boring font often of the wrong size. Are there any ways to do a work around? Can I search-and-replace fonts for example?

Connie:
For those situations, you could make a second version of the presentation for online meeting tools and change the font to a similar but more common one in your template. Even though it might be boring, it everything will line up the way you want it to. If the online meeting tool is hosted by a professional association or a company, you can see if they would be willing to install the font on the hosting computer.

As to replacing fonts, I’ve always replaced the font in the Slide Master and that usually works. You may have to choose the Master Layout again in your slide though.

 

10.

Will:
Connie, I love the section in your book on creating a visual hierarchy. As you describe it, visual hierarchies send unconscious signals to our brains that prompt us to look at certain parts of the visual before other parts. I didn’t really know this until I read your book. Thank you! I’m a big believer in PowerPoint (or KeyNote, etc.) of revealing aspects of our visuals one at a time, which is probably cheating to get a similar effect to hierarchies—but sometimes there would be a huge benefit in having a visual hierarchy. Please educate us on how to create a visual hierarchy, and tell us why it’s so important.

 

Connie:
A visual hierarchy indicates where the viewer should look first, second and perhaps third. Make your most important element the first thing that people will look at. You can do this through contrast.  Make it larger, place it in the upper left or top of the screen, make the element more colorful or brighter or add movement (if appropriate to the learning). There are other ways, but that’s a good place to start.

 

11.

Will:
I notice you added humor into your book. I laughed out loud when you told me that to transform myself into an “expert designer” that I’d need to wear all black. LOL. Your book is extremely helpful—and one doesn’t even have to change their wardrobe.

What’s your most important message for us learning professionals? Besides reading your book, what else can we do to be more effective? And, are there any methods you’ve seen for getting evaluation feedback on our visual designs?

Connie:
There are a lot of things learning professionals can do in addition to reading my book, dressing in all black and getting piercings in weird places:

  • Analyze the visual design in your environment and see what works and what doesn’t. Think about what the designer was trying to achieve. Notice the layout, typography, color palette and focal point. This means studying the design of websites, apps, magazines, brochures, posters, books, catalogs, packaging, billboards, subway ads, store interiors, videos, icons and junk mail. You get the idea.
  • Start an online collection (via Pinterest, bookmark sites, etc.) of designs that you like. Then use these for inspiration the next time you are stuck.
  • Read design books. Although most design books focus on advertisements and branding, they still offer a lot of sound principles and inspiration.

 

12.

Will:
Finally, what’s the best way for people to get your book?

Connie:
Thanks for asking. My book is available on Amazon, Barnes and Noble, and the ATD online bookstore, where it is discounted for members. I hope your readers understand that there are around 130 color graphics, which makes it a little more on the expensive side due to printing costs.

 

Will’s Note:
You can view the book on Amazon
by clicking the image below: