It’s time to publicly vilify NTL Institute for Applied Behavioral Science for propagating the myth that learners remember 10% of what they read, 20% or what they see visually, etc. They continue to claim that they did this research and that it is accurate.

The research is NOT accurate, nor could it be. Even a casual observer can see that research results that end neatly in 5’s or 0’s (as in 5%, 10%, 20%, 30%) are extremely unlikely. To see a complete debunking of this hoax, click here.

Normally, I choose not to name names when it comes to the myths in our field. We all make mistakes, right? But NTL continues to harm our field by propagating this myth. Here is the document (Download NTL’s email)–the one they send to people who inquire about the percentages. At least five separate people have sent me this document after contacting NTL on their own initiative.

I have talked to NTL staff people and emailed them (over a year ago), and even with my charming personality, I have failed to persuade them of the problems they are causing.

The people who write me about this are outraged (and frankly confused) that an organization would propagate such an obvious falsehood. Are you?

Here are claims that NTL makes in its letter that are false:

NTL: We know that in 1954 a similar pyramid with slightly different numbers appeared on p. 43 of a book called Audio-Visual Methods in Teaching, published by the Edgar Dale Dryden Press in New York.

Why false? There are NO numbers on page 43 of Edgar Dale’s book.

NTL: We are happy to respond to your inquiry about The Learning Pyramid. Yes, it was developed and used by NTL Institute at our Bethel, Maine campus in the early sixties when we were still part of the National Education Association’s Adult Education Division.

Very Intriguing: How could NTL have developed the pyramid in the 1960’s, when a similar version was published by Edgar Dale in 1954? Professor Michael Molenda of Indiana University has found some evidence that the numbers first appeared in the 1940’s. Maybe NTL has a time machine.

NTL: Yet the Learning Pyramid as such seems to have been modified and always has been attributed to NTL Institute.

No. It wasn’t attributed to NTL by Dale. Dale thought it was his. And again, Dale did not use any numbers. Just a cone.

Okay, so now half of you hate NTL, and the other half of you hate me for being the “know-it-all kid” from 7th grade. Well, I’ll take the heat for that. But still, is this the kind of field you want to work in?

And what is the advantage for NTL to continue the big lie?

Here’s what NTL should write when people inquire:

Thanks for your inquiry to the NTL Institute. Yes, we once utilized the “Learning Pyramid” concept in our work, starting in the 1960’s. However, we can no longer locate the source of the original information and recent research tends to debunk those earlier recommendations. We apologize for any harm or confusion we may have caused.

Okay, here’s another example of the same incorrect information that plagues our field. This is from a company named Percepsys:

 

Hopefully, sometime soon, the webpage on their site won’t work because the vendor will smarten up and remove this misinformation. NOTE from 2017: The company appears to be no longer in business.

It’s NOT TRUE that people remember 10% of what they read, 20% of what they hear, etc. Moreover, if you know anything about learning, you’d know it would be impossible to pin down the amount of remembering. It depends on the materials, the learners, the duration of learning, the type of learning activities, the consistency between the learning situation and the retrieval situation, and the length of retention among other things. Finally, while this information (the 10%, 20%, 30% information) is often attached to Dale’s Cone, Dale never actually had any numbers on his cone.

For the best review of the history of this misdirection, if I must say so myself, is here.

I will give $1000 (US dollars) to the first person or group who can prove that taking learning styles into account in designing instruction can produce meaningful learning benefits.

I’ve been suspicious about the learning-styles bandwagon for many years. The learning-style argument has gone something like this: If instructional designers know the learning style of their learners, they can develop material specifically to help those learners, and such extra efforts are worth the trouble.

I have my doubts, but am open to being proven wrong.

Here’s the criteria for my Learning-Styles Instructional-Design Challenge:

  1. The learning program must diagnose learners’ learning styles. It must then provide different learning materials/experiences to those who have different styles.
  2. The learning program must be compared against a similar program that does not differentiate the material based on learning styles.
  3. The programs must be of similar quality and provide similar information. The only thing that should vary is the learning-styles manipulation.
  4. The comparison between the two versions (the learning-style version and the non-learning-style version) must be fair, valid, and reliable. At least 70 learners must be randomly assigned to the two groups (with at least 35 minimum in each group completing the experience). The two programs must have approximately the same running time. For example, the time required by the learning-style program to diagnose learning styles can be used by the non-learning-styles program to deliver learning. The median learning time for the programs must be no shorter than 25 minutes.
  5. Learners must be adults involved in a formal workplace training program delivered through a computer program (e-learning or CBT) without a live instructor. This requirement is to ensure the reproducability of the effects, as instructor-led training cannot be precisely reproduced.
  6. The learning-style program must be created in an instructional-development shop that is dedicated to creating learning programs for real-world use. Programs developed only for research purposes are excluded. My claim is that real-world instructional design is unlikely to be able to utilize learning styles to create learning gains.
  7. The results must be assessed in a manner that is relatively authentic–at a minimum level learners should be asked to make scenario-based decisions or perform activities that simulate the real-world performance the program teaches them to accomplish. Assessments that only ask for information at the knowledge level (e.g., definitions, terminology, labels) are NOT acceptable. The final assessment must be delayed at least one week after the end of the training. The same final assessment must be used for both groups. It must fairly assess the whole learning experience.
  8. The magnitude of the difference in results between the learning-style program and the non-learning-style program must be at least 10%. (In other words, the average of the learning-styles scores subtracted by the average of the non-learning-styles scores must be more than 10% of the non-learning-styles scores). So for example, if the non-learning-styles average is 50, then the learning-styles score must be equal to 55 or more. This magnitude is to ensure that the learning-styles program produces meaningful benefits. 10% is not too much to ask.
  9. The results must be statistically significant at the p<.05 level. Appropriate statistical procedures must be used to gauge the reliability of the results. Cohen’s d effect size should be equal to .4 or more (a small to medium effect size according to Cohen, 1992).
  10. The learning-style program cannot cost more than twice as much as the non-learning-style program to develop, nor can it take more than twice as long to develop. I want to be generous here.
  11. The results can be documented by unbiased parties.

To reiterate, the challenge is this:

Can an e-learning program that utilizes learning-style information outperform an e-learning program that doesn’t utilize such information by 10% or more on a realistic test of learning, even it is allowed to cost up to twice as much to build?

$1,000 says it just doesn’t happen in the real-world of instructional design. $1,000 says we ought to stop wasting millions trying to cater to this phantom curse.

The Bloom is Off the Vine

I just came across this nifty little piece on Bloom’s Taxonomy, written by Brenda Sugrue for ISPI’s Performance Express.

It’s a nice critique on the validity and usefulness of Bloom’s Taxonomy for Instructional Design.

Read it here.

I tend to agree with Brenda’s Critique. For a long time I’ve been suspicious of Blooms.

 

==================

In case that link ever goes away, I’m repeating her piece here:

Problems with Bloom’s Taxonomy
by Brenda Sugrue, PhD, CPT

I did a 99-second critique of Bloom’s taxonomy at the 2002 ISPI conference, and it generated more unsolicited feedback than any other presentation I have made. The response varied from those who completely agreed with me and abandoned Bloom many years ago to those who are still true believers and avid users. In those 99 seconds, I criticized the taxonomy but did not have time to present more valid alternatives. This article summarizes the criticisms and presents two alternative strategies for classifying objectives in order to design appropriate instruction and assessment.

Invalidity
Bloom’s taxonomy is almost 50 years old. It was developed before we understood the cognitive processes involved in learning and performance. The categories or “levels” of Bloom’s taxonomy (knowledge, comprehension, application, analysis, synthesis, and evaluation) are not supported by any research on learning. The only distinction that is supported by research is the distinction between declarative/conceptual knowledge (which enables recall, comprehension, or understanding) and procedural knowledge (which enables application or task performance).

Unreliability
The consistent application of Bloom’s taxonomy across multiple designers/developers is impossible. Given any learning objective, it might be classified into either of the two lowest levels (knowledge or comprehension) or into any of the four highest levels (application, analysis, synthesis, or evaluation) by different designers. Equally, there is no consistency in what constitutes instruction or assessment that targets separate levels. A more reliable approach is to separate objectives and practice/assessment items into those that elicit or measure declarative/conceptual knowledge from those that elicit or measure task performance/procedural knowledge.

Impracticality
The distinctions in Bloom’s taxonomy make no practical difference in diagnosing and treating learning and performance gaps. Everything above the “knowledge” level is usually treated as “higher-order thinking” anyway, effectively reducing the taxonomy to two levels.

The Content-by-Performance Alternative
Recent taxonomies of objectives and learning object strategies distinguish among types of content (usually facts, concepts, principles, procedures, and processes) as well as levels of performance (usually remember and use). This content-by-performance approach leads to general prescriptions for informational content and practice/assessment such as those presented in Figure 1.

Figure 1. Prescriptions for Information and Practice Based on Content-Performance Matrix.

Content Type

Information to Present
(Regardless of Level of Performance

Practice/Assessment
(Depending on Level of Performance)

Remember

Use

Fact the fact recognize or recall the fact recognize or recall during task performance
Concept the definition, critical attributes, examples, non-examples recognize or recall the definition or attributes Identify, classify, or create examples
Principle/
Rule
the principle/rule, examples, analogies, stories recognize, recall, or explain the principle decide if the principle applies, predict an event, apply the principle to solve a problem
Procedure list of steps, demonstration recognize, recall, or reorder the steps perform the steps
Process description of stages, inputs, outputs, diagram, examples, stories recognize, recall, or reorder the stages identify origins of problems in the process; predict events in the process; solve problems in the process

The Pure Performance Alternative
A more radical approach would be to have no taxonomy at all, to simply assume that all objectives are at the use level (that is, “performance” objectives) and that learners will practice or be assessed on the particular performance in representative task situations. If there are “enabling” sub-objectives, those too can be treated as performance objectives without further classification. If, for example, a loan officer needs to be able to distinguish among types of mortgages and describe the pros and cons of each type of mortgage as an enabling skill for matching house buyers with mortgages, then we design/provide opportunities to practice categorizing mortgages and listing their pros and cons before we practice on matching buyers to mortgages. If a car salesperson needs to be able to describe the features of different car models as an enabling skill for selling cars, then we design/provide opportunities to practice describing the features of different cars before we practice on selling cars.

References
Bereiter, C., & Scardamalia, M. (1998). Beyond Bloom’s taxonomy: Rethinking knowledge for the knowledge age. In A. Hargreaves, A. Lieberman, M. Fullen, & D. Hopkins, (Eds.), International handbook of educational change. Boston: Kluwer Academic.

Merrill, M.D. (1994). Instructional design theory. Englewood Cliffs, NJ: Educational Technology Publications.

Moore, D.S. (1982). Reconsidering Bloom’s Taxonomy of educational objectives, cognitive domain. Educational Theory, 32(1) 29-34.

CPP, Inc., known formerly as Consulting Psychologists Press, announces that it is offering research grants for research on the Myers-Briggs Type Indicator.

This may seem commendable, but their research-grant program is biased. Here are the facts:

  1. CPP makes money by selling MBTI implementations, consulting, and paraphernalia.
  2. The MBTI (Myers-Briggs) is widely discredited by researchers. It is considered neither reliable nor valid. For example, see Pittenger, D. J. (2005). Cautionary Comments Regarding the Myers-Briggs Type Indicator. Consulting Psychology Journal: Practice and Research, 57, 210-221.
  3. The research grant program is biased toward research findings that support the MBTI. Here are some details:
    • CPP, a biased party, selects the grantees.
    • One of the criteria for selection is “advancement of the MBTI assessment.”
    • Money is distributed only for research reports selected by CPP for the “Best Paper Awards.”
  4. Instead of these regrettable procedures, CPP should form a body of unbiased reviewers, have criteria that don’t push toward a confirmatory bias, distribute money for good proposals not “favorable” results, and form an unbiased committee to select the best papers.

This Research Grant Program (as outlined in the publicly available materials produced by CPP) is clearly designed to produce results that support CPP’s financial interests and resurrect the flagging image of the MBTI. Statements in the proposal requiring researchers to “conform to the Americal Psychological Association’s Ethical Principles of Psychologists” do little to overcome the biases built into the program. As the materials make clear, the intention is to provide comfort to CPP’s clients. How else are we to interpret the following statement in CPP’s research-grant announcement?

“Abstracts from the papers will be used by CPP to communicate results with its customers.”

This type of biased research program is completely unacceptable. Not only does it have the potential to create biased information and lead to suboptimal or dangerous recommendations, but it also casts a shadow on fair-and-balanced research that might be used to guide learning-and-performance agendas.

If you’d like to share your thoughts with CPP, it appears that the person to write is available through this email address.

Publication Note

This article was originally published on the Work-Learning Research website (www.work-learning.com) in 2002. It may have had some minor changes since then. It was moved to my WillAtWorkLearning Blog in 2006, and has now been moved here in late 2017.

Updated Research

Even after more than a decade, this blog post still provides valuable information explaining the issues — and the ramifications for learning. However, further research has uncovered additional information and has been published in a scientific journal in 2014. You can read a review of that research here.

Introduction

People do NOT remember 10% of what they read, 20% of what they see, 30% of what they hear, etc. That information, and similar pronouncements are fraudulent. Moreover, general statements on the effectiveness of learning methods are not credible—learning results depend on too many variables to enable such precision. Unfortunately, this bogus information has been floating around our field for decades, crafted by many different authors and presented in many different configurations, including bastardizations of Dale’s Cone. The rest of this article offers more detail.

My Search For Knowledge

My investigation of this issue began when I came across the following graph:

The Graph is a Fraud!

After reading the cited article several times and not seeing the graph—nor the numbers on the graph—I got suspicious and got in touch with the first author of the cited study, Dr. Michelene Chi of the University of Pittsburgh (who is, by the way, one of the world’s leading authorities on expertise). She said this about the graph:

“I don’t recognize this graph at all. So the citation is definitely wrong; since it’s not my graph.”

What makes this particularly disturbing is that this graph has popped up all over our industry, and many instructional-design decisions have been based on the information contained in the graph.

Bogus Information is Widespread

I often begin my workshops on instructional design and e-learning and my conference presentations with this graph as a warning and wake up call. Typically, over 90% of the audience raises their hands when I ask whether anyone has seen the numbers depicted in the graph. Later I often hear audible gasps and nervous giggles as the information is debunked. Clearly, lots of experienced professionals in our field know this graph and have used it to guide their decision making.

The graph is representative of a larger problem. The numbers presented on the graph have been circulating in our industry since the late 1960’s, and they have no research backing whatsoever. Dr. JC Kinnamon (2002) of Midi, Inc., searched the web and found dozens of references to those dubious numbers in college courses, research reports, and in vendor and consultant promotional materials.

Where the Numbers Came From

The bogus percentages were first published by an employee of Mobil Oil Company in 1967, writing in the magazine Film and Audio-Visual Communications. D. G. Treichler didn’t cite any research, but our field has unfortunately accepted his/her percentages ever since. NTL Institute still claims that they did the research that derived the numbers. See my response to NTL.

Michael Molenda, a professor at Indiana University, is currently working to track down the origination of the bogus numbers. His efforts have uncovered some evidence that the numbers may have been developed as early as the 1940’s by Paul John Phillips who worked at University of Texas at Austin and who developed training classes for the petroleum industry. During World War Two Phillips taught Visual Aids at the U. S. Army’s Ordnance School at the Aberdeen (Maryland) Proving Grounds, where the numbers have also appeared and where they may have been developed.

Strange coincidence: I was born on these very same Aberdeen Proving Grounds.

Ernie Rothkopf, professor emeritus of Columbia University, one of the world’s leading applied research psychologists on learning, reported to me that the bogus percentages have been widely discredited, yet they keep rearing their ugly head in one form or another every few years.

Many people now associate the bogus percentages with Dale’s “Cone of Experience,” developed in 1946 by Edgar Dale. It provided an intuitive model of the concreteness of various audio-visual media. Dale included no numbers in his model and there was no research used to generate it. In fact, Dale warned his readers not to take the model too literally. Dale’s Cone, copied without changes from the 3rd and final edition of his book, is presented below:

Dale’s Cone of Experience (Dale, 1969, p. 107)

You can see that Dale used no numbers with his cone. Somewhere along the way, someone unnaturally fused Dale’s Cone and Treichler’s dubious percentages. One common example is represented below.

The source cited in the diagram above by Wiman and Meierhenry (1969) is a book of edited chapters. Though two of the chapters (Harrison, 1969; Stewart, 1969) mention Dale’s Cone of Experience, neither of them includes the percentages. In other words, the diagram above is citing a book that does not include the diagram and does not include the percentages indicated in the diagram.

Here are some more examples:

 

 

The “Evidence” Changes to Meet the Need of the Deceiver

The percentages, and the graph in particular, have been passed around in our field from reputable person to reputable person. The people who originally created the fabrications are to blame for getting this started, but there are clearly many people willing to bend the information to their own devices. Kinnamon’s (2002) investigation found that Treichler’s percentages have been modified in many ways, depending on the message the shyster wants to send. Some people have changed the relative percentages. Some have improved Treichler’s grammar. Some have added categories to make their point. For example, one version of these numbers says that people remember 95% of the information they teach to others.

People have not only cited Treichler, Chi, Wiman and Meierhenry for the percentages, but have also incorrectly cited William Glasser, and correctly cited a number of other people who have utilized Treichler’s numbers.

It seems clear from some of the fraudulent citations that deception was intended. On the graph that prompted our investigation, the title of the article had been modified from the original to get rid of the word “students.” The creator of the graph must have known that the term “students” would make people in the training / development / performance field suspicious that the research was done on children. The creator of Wiman and Meierhenry diagram did four things that make it difficult to track down the original source: (1) the book they cited is fairly obscure, (2) one of the authors names is spelled wrong, (3) the year of publication is incorrect, (4) and the name Charles Merrill, which was actually a publishing house, was ambiguously presented so that it might have referred to an author or editor.

But Don’t The Numbers Speak The Truth?

The numbers are not credible, and even if they made sense, they’d still be dangerous.

If we look at the numbers a little more closely, they are highly unconvincing. How did someone compare “reading” and “seeing?” Don’t you have to “see” to “read?” What does “collaboration” mean anyway? Were two people talking about the information they were learning? If so, weren’t they “hearing” what the other person had to say? What does “doing” mean? How much were they “doing” it? Were they “doing” it correctly, or did they get feedback? If they were getting feedback, how do we know the learning didn’t come from the feedback—not the “doing?” Do we really believe that people learn more “hearing” a lecture, than “reading” the same material? Don’t people who “read” have an advantage in being able to pace themselves and revisit material they don’t understand? And how did the research produce numbers that are all factors of ten? Doesn’t this suggest some sort of review of the literature? If so, shouldn’t we know how the research review was conducted? Shouldn’t we get a clear and traceable citation for such a review?

Even the idea that you can compare these types of learning methods is ridiculous. As any good research psychologist knows, the measurement situation affects the learning outcome. If we have a person learn foreign-language vocabulary by listening to an audiotape and vocalizing their responses, it doesn’t make sense to test them by having them write down their answers. We’d have a poor measure of their ability to verbalize vocabulary. The opposite is also nonsensical. People who learn vocabulary by seeing it on the written page cannot be fairly evaluated by asking them to say the words aloud. It’s not fair to compare these different methods by using the same test, because the choice of test will bias the outcome toward the learning situation that is most like the test situation.

But why not compare one type of test to another—for example, if we want to compare vocabulary learning through hearing and seeing, why don’t we use an oral test and written one? This doesn’t help either. It’s really impossible to compare two things on different indices. Can you imagine comparing the best boxer with the best golfer by having the boxer punch a heavy bag and having the golfer hit for distance? Would Muhammad Ali punching with 600 pounds of pressure beat Tiger Woods hitting his drives 320 yards off the tee?

The Importance of Listing Citations

Even if the numbers presented on the graph had been published in a refereed journal—research we were reasonably sure we could trust—it would still be dangerous not to know where they came from. Research conclusions have a way of morphing over time. Wasn’t it true ten years ago that all fat was bad? Newer research has revealed that monounsaturated oils like olive oil might actually be good for us. If a person doesn’t cite their sources, we might not realize that their conclusions are outdated or simply based on poor research. Conversely, we may also lose access to good sources of information. Suppose Teichler had really discovered a valid source of information? Because he/she did not use citations, that research would remain forever hidden in obscurity.

The context of research makes a great deal of difference. If we don’t know a source, we don’t really know whether the research is relevant to our situation. For example, an article by Kulik and Kulik (1988) concluded that immediate feedback was better than delayed feedback. Most people in the field now accept their conclusions. Efforts by Work-Learning Research to examine Kulik and Kulik’s sources indicated that most of the articles they reviewed tested the learners within a few minutes after the learning event, a very unrealistic analog for most training situations. Their sources enabled us to examine their evidence and find it faulty.

Who Should We Blame?

The original shysters are not the only ones to blame. The fact that many people who have disseminated the graph used the same incorrect citation makes it clear that they never accessed the original study. Everyone who uses a citation to make a point (or draw a conclusion) ought to check the citation. That, of course, includes all of us who are consumers of this information.

What Does This Tell Us About Our Field?

It tells us that we may not be able to trust the information that floats around our industry. It tells us that even our most reputable people and organizations may require the Wizard-of-Oz treatment—we may need to look behind the curtain to verify their claims.

The Danger To Our Field

At Work-Learning Research, our goal is to provide research-based information that practitioners can trust. We began our research efforts several years ago when we noticed that the field jumps from one fad to another while at the same time holding religiously to ideas that would be better cast aside.

The fact that our field is so easily swayed by the mildest whiffs of evidence suggests that we don’t have sufficient mechanisms in place to improve what we do. Because we’re not able or willing to provide due diligence on evidence-based claims, we’re unable to create feedback loops to push the field more forcefully toward continuing improvement.

Isn’t it ironic? We’re supposed to be the learning experts, but because we too easily take things for granted, we find ourselves skipping down all manner of yellow-brick roads.

How to Improve the Situation

It will seem obvious, but each and every one of us must take responsibility for the information we transmit to ensure its integrity. More importantly, we must be actively skeptical of the information we receive. We ought to check the facts, investigate the evidence, and evaluate the research. Finally, we must continue our personal search for knowledge—for it is only with knowledge that we can validly evaluate the claims that we encounter.

Updated Research

Even after more than a decade, this blog post still provides valuable information explaining the issues — and the ramifications for learning. However, further research has uncovered additional information and has been published in a scientific journal in 2014. You can read a review of that research here.

Our Citations

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Dale, E. (1946, 1954, 1969). Audio-visual methods in teaching. New York: Dryden.

Harrison, R. (1969). Communication theory. In R. V. Wiman and W. C. Meierhenry (Eds.) Educational media: Theory into practice. Columbus, OH: Merrill.

Kinnamon, J. C. (2002). Personal communication, October 25.

Kulik, J. A., & Kulik, C-L. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58, 79-97.

Molenda, M. H. (2003). Personal communications, February and March.

Rothkopf, E. Z. (2002). Personal communication, September 26.

Stewart, D. K. (1969). A learning-systems concept as applied to courses in education and training. In R. V. Wiman and W. C. Meierhenry (Eds.) Educational media: Theory into practice. Columbus, OH: Merrill.

Treichler, D. G. (1967). Are you missing the boat in training aids? Film and Audio-Visual Communication, 1, 14-16, 28-30, 48.

Wiman, R. V. & Meierhenry, W. C. (Eds.). (1969). Educational media: Theory into practice. Columbus, OH: Merrill.