Learners’ Often Use Poor Learning Strategies — From a Research Review

,

I just read the following research article, and found a great mini-review of some essential research.

  • Hagemans, M. G., van der Meij, H., & de Jong, T. (2013). The effects of a concept map-based support tool on simulation-based inquiry learning. Journal of Educational Psychology, 105(1), 1-24. doi:10.1037/a0029433

Experiment-Specific Findings:

The article shows that simulations—the kind that ask learners to navigate through the simulation on their own—are more beneficial when learners are supported in their simulation playing. Specifically, they found that learners given the optimal learning route did better than those supplied with a sub-optimal learning route. They also found that concept maps helped the learners by supporting their comprehension. They also found that learners who got feedback on the correctness of their practice attempts were motivated to correct their errors and thus provided themselves with additional practice.

Researchers’ Review of Learners’ Poor Learning Strategies

The research Hagemans, van der Meij, and de Jong did is good, but what struck me as even more relevant for you as a learning professional is their mini review of research that shows that learners are NOT very good stewards of their own learning. Here is what their mini-review said (from Hagemans, van der Meij, and de Jong, 2013, p. 2:

  • Despite the importance of planning for learning, few students engage spontaneously in planning activities (Manlove & Lazonder, 2004).  
  • Novices are especially prone to failure to engage in planning prior to their efforts to learn (Zimmerman, 2002).  
  • When students do engage in planning their learning, they often experience difficulty in adequately performing the activities involved (de Jong & Van Joolingen, 1998; Quintana et al., 2004). For example, they do not thoroughly analyze the task or problem they need to solve (Chi, Feltovich, & Glaser, 1981; Veenman, Elshout, & Meijer, 1997) and tend to act immediately (Ge & Land, 2003; Veenman et al., 1997), even when a more thorough analysis would actually help them to build a detailed plan for learning (Veenman, Elshout, & Busato, 1994).  
  • The learning goals they set are often of low quality, tending to be nonspecific and distal (Zimmerman, 1998).
  • In addition, many students fail to set up a detailed plan for learning, whereas if they do create a plan, it is often poorly constructed (Manlove et al., 2007). That is, students often plan their learning in a nonsystematic way, which may cause them to start floundering (de Jong & Van Joolingen, 1998), or they plan on the basis of what they must do next as they proceed, which leads to the creation of ad hoc plans in which they respond to the realization of a current need (Manlove & Lazonder, 2004).  
  • The lack of proper planning for learning may cause students to miss out on experiencing critical moments of inquiry, and their investigations may lack systematicity.
  • Many students also have problems with monitoring their progress, in that they have difficulty in reflecting on what has already been done (de Jong & Van Joolingen, 1998).
  • Regarding monitoring of understanding, students often do not know when they have comprehended the subject matter material adequately (Ertmer & Newby, 1996; Thiede, Anderson, & Therriault, 2003) and have difficulty recognizing breakdowns in their understanding (Ertmer & Newby, 1996).
  • If students do recognize deficits in their understanding, they have difficulty in expressing explicitly what they do not understand (Manlove & Lazonder, 2004).
  • One consequence is that students tend to overestimate their level of success, which may result in “misplaced optimism, substantial understudying, and, ultimately, low test scores” (Zimmerman, 1998, p. 9).

The research article is available by clicking here.

Final Thoughts

This research, and other research I have studied over the years, shows that we CANNOT ALWAYS TRUST THAT OUR LEARNERS WILL KNOW HOW TO LEARN. We as instructional designers have to design learning environments that support learners in learning. We need to know the kinds of learning situations where our learners are likely to succeed and those where they are likely to fail without additional scaffolding.

The research also shows, more specifically, that inquiry-based simulation environments can be powerful learning tools, but ONLY if we provide the learners with guidance and/or scaffolding that enables them to be successful. Certainly, some few may succeed without support, but most will act suboptimally.

We have a responsibility to help our learners. We can't always put it on them…

Five Biggest Mistakes in Onboarding: What the Research Says…

This article first appeared in my Newsletter in the October 2013 issue. You can sign up for my newsletter by clicking here.

————————————————————————–

Onboarding is ubiquitous. Every organization does it. Some do it with great fanfare. Some make a substantial investment. Some just let supervisors get their new hires up to speed. Unfortunately, most organizations make critical mistakes in onboarding—mistakes that increase turnover, raise costs, weaken employee loyalty, and lower productivity.

Fortunately, recent research highlights onboarding best practices. If organizations would just use the wisdom from the research, they’d save themselves money, time, and resources—and employees in those companies would have to deal with many fewer headaches.

Key Outcomes

Recent reviews of the research suggest that there four key outcomes that enable onboarding success:

  1. New hires have to quickly and effectively learn their new job role.
  2. New hires have to feel a sense of self-efficacy in doing their job.
  3. New hires have to learn the organizational culture.
  4. New hires have to gain acceptance and feel accepted by their coworkers.

Enabling Factors

Recent research suggests that the following factors are helpful in ensuring onboarding success:

 

What New Hires Can Do

  1. Be proactive in learning and networking
  2. Be open to new ways of thinking and acting
  3. Be active in seeking information and getting feedback
  4. Be active in building relationships

What the Organization Can Do

  1. Ensure that managers take a very active and effective role
  2. Provide formal orientations that go beyond information dissemination
  3. Provide realistic previews of the organization and the job
  4. Proactively enable new hires to connect with long-tenured employees

 

Five Biggest Mistakes
(In Reverse Order of Importance)

5—Providing an Information Dump during Orientation

The research shows that employee orientations can facilitate onboarding. However, too many organizations think their orientations should just cram tons of information down the throats of employees. Even worse are orientations that have employees sit and listen to presentation after presentation. Oh the horror. New employees are excited to get going. Putting them into the prison of listening—even to great content—is a rudeness that shouldn’t be tolerated. The best orientations help build relationships. They get employees involved. They prepare new hires for how to learn and grow and network on their own. They help new hires learn the organization culture—both the good and the bad. They share the organization’s vision, passions, and its strategic concerns.

4—Thinking that Training is Sufficient

Training can be essential to help get new employees competent in their new roles, but it is NEVER sufficient on its own. Training should be supported by prompting mechanisms (like job aids), structures and support for learning on the job, reinforcement and follow-through, and coaching to provide feedback, set goals, and lend emotional support.

3—Forgetting the Human Side of Onboarding

New hires are human beings, and, just like the rest of us, they too are influenced by the dynamics of social interaction. They don’t just learn to do a job. They also learn to love and trust a company, a work unit, or a group of coworkers—or they don’t. In return, new hires are either trusted and respected by their coworkers or they’re not.  The research is very clear about this. One of the keys to successful onboarding is the strength of the relationships that are built in the first year of a person’s tenure. The stronger the bonds, the more likely it is that a person will stay and bring value to the organization.

2—Considering Onboarding as Something that Can Be Done Quickly

Some companies offer a one week orientation and then cut loose their new hires to sink or swim. Enlightened companies, on the other hand, realize that onboarding is like relationship-building—it takes time. It takes time to really learn one’s job well. It takes time to integrate into the organizational culture. It takes time to connect with people. Realistic estimates suggest that onboarding can take 6 months, 12 months, or even 18 months to fully integrate a person into a new organization.

1—Not Preparing Supervisors

Supervisors are the single most important leverage point for onboarding success. You’ve probably heard it said that people don’t quit their companies, they quit their supervisors. Well, the flip side can also be said. People don’t join a company, they join a supervisor and his/her workgroup. Unfortunately, most supervisors just have no idea about the importance of onboarding and how to do it correctly. Where best practices give supervisors training and an onboarding checklist, too many supervisors just wing it. The real tragedy is that the investment in onboarding training and a checklist for supervisors is quite small in the greater scheme of things.

Final Thoughts on Onboarding

As a workplace learning-and-performance consultant, when I’ve been called in to advise companies on their onboarding programs, I often see incredibly dedicated professionals who are passionate about welcoming new people into their organizations. Unfortunately, too many times, I see organizations that have the wrong mental models about what makes onboarding successful. It’s a shame that our old mental models keep us from effectiveness—when the research on onboarding now gives us sound prescriptions for making onboarding successful.

Small Spacings Work!

When asked for a simple heuristic in how to use the spacing effect–the
finding that repetitions spaced in time are more effective in
supporting remembering than repetitions squished narrowly in time–I've
often told people that the ideal spacing interval is one that will equal
the retention interval one desires. If you want you learners to
remember for a month, give them one-month spaced repetitions. If you
can't do that, longer is better, and there seems to be something magical
about repeating something overnight.

But new research suggests that even short spacings of only
half-a-minute or so can have lasting benefits over non-spaced
repetitions.

Citation:

Rawson, K. A., & Dunlosky, J. (2012). Relearning Attenuates the Benefits and Costs of Spacing. Journal of Experimental Psychology: General. Advance online publication.

Decisive Dozen — Research Support

, ,

In continuing my dual career–partly as a charismatic and wildly-effective learning consultant and keynote speaker, partly as a grizzled, hermitted researcher and agonizingly-slow book writer (still working on the same book for the last 16 years)–I have started talking publicly about my list of 12 factors that if training developers implemented, would propel their training to divine glory. I call this list “The Decisive Dozen.”

Click here to read a short introduction to the Decisive Dozen

Of course, I get questions. “How does he do it when so many others have perished?” “How does he get his hair to do that?” And my personal favorite, “How the hell does Thalheimer know these are the most critical learning factors?” It’s a fair question, and instead of releasing 12 chapters from my potentially-forthcoming hopefully-not-posthumorous book, I created a little research brief just to show that I’m not making this stuff up. I’m stealing directly from the world’s best learning researchers!

Click to download the Decisive Dozen Research Review

Thanks for your interest in my work!

The Decisive Dozen: A Brief Synopsis

,

I set out over 15 years ago to come up with a short list of the most important learning factors based on scientific research and practical real-world wisdom. I felt at the time, and I believe even more strongly now, that the learning field–particularly the workplace learning-and-performance field–is too strongly tempted to jump from one learning fad to another, while ignoring the learning factors that are most important.

My original goal was to create a list that had no more than seven learning factors. I did extensive reviews of a wide swath of the research on learning, memory, and instruction; often doing comparative effect-size analyses to determine what factors were most important. As the years went by and blurred into the second decade of work, I more and more looked at the research from a practical perspective, hoping to find the factors that were not just the most potent, but also the most leveragable by real-world instructional designers, trainers, teachers, e-learning developers, etc.In the end, I failed to find only seven factors, but found 12 that seem extraordinarily potent and leveragable.

Obviously, to pick the most important learning factors is a difficult endeavor–and one subject to a significant degree of human judgment, of which some of mine is surely faulty. Still, all along the way, I have not lost my strong belief that coming up with a short list of learning factors based on the world’s best scientific research from peer-reviewed refereed journals would be extremely helpful in keeping us focused on the factors that matter the most. At a minimum, I feel the list I have created will help most of us create remarkably more effective learning interventions.

In the current version of my book draft, I say the following:

If you put all 12 of these factors into practice, your learning interventions are
likely to be more effective than 95% of all workplace learning interventions
currently being utilized!!

This is quite a bold statement, I know. But I’m very comfortable in making it. In the book I provide a detailed footnote explaining the evidence behind the statement, and maybe an editor will convince me not to be so bold (maybe push me down to 93%, for example–SMILE), but I really do think that most learning interventions in the workplace learning field are lacking significant effectiveness.

Also in the book, each of the 12 factors has its own chapter, often dozens of pages long, backed up by dozens of research studies, and dozens of practical implications and recommendations. Dozens to the max! Obviously, the short descriptions in the synopsis below cannot even approach doing these topics justice. Still, they may provide you with a good framework to enable you to begin to see your learning designs in a different light.

The Decisive Dozen

Because I have shared the Decisive Dozen with clients and in keynote addresses and conference speeches for the last year or so, I decided that it was time to make the list officially public.

Click here to download a brief synopsis of the Decisive Dozen

I welcome your comments and feedback.

And, of course, for those who can’t wait for the book (I can’t blame you, I’m taking a long time, aren’t I?), I would be delighted to discuss with you how the Decisive Dozen might be helpful in guiding your organization to greater learning effectiveness.

Update: Now you can check out a research review supporting the Decisive Dozen.

Click here to get access to the free research review

Poor New York Times piece on Education

, ,

Here is the comment I sent to the NY Times in response to their focus on a supposed research study that purported to show that gifted kids are being underserved.

I'm a little over the top in my comments, but still I think this is worth printing because it demonstrates the need for good research savvy and it shows that even the most respected news organizations can make really poor research-assessment mistakes.

Egads!! Why is the New York Times giving so much "above-the-fold" visibility to a poorly-conceived research study funded by a conservative think tank with obvious biases?

Why isn't at least one of your contributors a research-savvy person who could comment on the soundness of the research? Instead, your contributors assume the research is sound.

Did you notice that the references in the original research report were not from top-tier refereed scientific journals?

In the original article from the Thomas Fordham Institute (a conservative-funded enterprise), the authors try to wash away criticisms about regression-to-the-mean and test-variability, but this bone against the obvious–and most damaging, and most valid–criticisms is not good enough.

If you took the top 10% of players in the baseball draft, the football draft, any company's onboarding class, any randomly selected group of maple trees, a large percentage of the top performers would not be top performers a year or two later. Damn, ask any baseball scout whether picking out the best prospects is a sure thing. It's not!

And, in the cases I mentioned above, the measures are more objective than an educational test, which has much higher variability–which would make more top performers leak out of the top ranks.

NY Times–you should be embarrassed to have published these responses to this non-study. Seriously, don't you have any research-savvy people left on your staff?

We have scientific journals because the research is vetted by experts.

How Much Do People Forget?

,

This blog post is excerpted from the full report, How Much Do People Forget? Click here to download the full report. You may also access the report—and many other reports—by going to my catalog page by clicking here.

Everybody Wants to Know—How Much Do People Forget?

For years, people have been asking me, “How much do people forget?” and I’ve told them, “It depends.” When I make this statement, most people scowl at me and walk away frustrated and unrequited. I also suspect that some of them think less of me—perhaps that I am just hiding my ignorance.

But I try. I try to explain the complexity of human learning. I explain that forgetting depends on many things, for example:

  • The type of material that is being learned
  • The learners’ prior knowledge
  • The learners’ motivation to learn
  • The power of the learning methods used
  • The contextual cues in the learning and remembering situations
  • The amount of time the learning has to be retained
  • The difficulty of the retention test
  • Etc.

More meaningful materials (like stories) tend to be easier to remember than less meaningful material (like nonsense syllables). More relevant concepts tend to be easier to remember than less relevant concepts. Learners who have more prior knowledge in a topic area are likely to be better able to remember new concepts learned in that area. More motivated learners are more likely to remember than less motivated learners. Learners who receive repetitions, retrieval practice, feedback, variety (and other potent learning methods) are more likely to remember than learners who do not receive such learning supports. Learners who are provided with learning and practice in the situations where they will be asked to remember the information will be better able to remember. Learners who are asked to retrieve information shortly after learning it will retrieve more than learners who are asked to retrieve information a long time after learning it.

I try to explain all this, but still people keep asking.

And then there are the statistics I keep hearing—that are passed around the learning field from person to person through the years as if they were immutable truths carved by Old Moses Ebbinghaus on granite stones. Here is some information so cited (as of December 2010):

  • People forget 40% of what they learned in 20 minutes and 77% of what they learned in six days (http://www.festo-didactic.co.uk/gb-en/news/forgetting-curve-its-up-to-you.htm?fbid=Z2IuZW4uNTUwLjE3LjE2LjM0Mzc).
  • People forget 90% after one month. (http://www.reneevations.com/management/ebbinghaus-curve/)
  • People forget 50-80% of what they’ve learned after one day and 97-98% after a month. (http://www.adm.uwaterloo.ca/infocs/study/curve.html)

Never mind that these immutable truths conflict with each other.

So, I will try one more time to convince the world that forgetting depends.

To accomplish this, I explored 14 research articles, examining 69 conditions to see how much forgetting occured, representing over 1,000 learners.

The following graph details the amount of forgetting for each of the 69 conditions:

 

Conclusions

This graph and the indepth analysis in the full article revealed four critical concepts in human learning—truths that every learning professional should deeply understand.

  1. The amount a learner will forget varies depending on many things. We as learning professionals will be more effective if we make decisions based on a deep understanding of how to minimize forgetting and enhance remembering.
  2. Rules-of-thumb that show people forgetting at some pre-defined rate are just plain false. In other words, learning gurus and earnest bloggers are wrong when they make blanket statements like, “People will forget 40% of what they learned within a day of learning it.”
  3. Learning interventions can produce profound improvements in long-term remembering. In other words, learning gurus are wrong when they say that training is not effective.
  4. Different learning methods produce widely different amounts of forgetting. We as learning professionals can be more effective if we take a research-based approach and utilize those learning methods that are most effective.

Telling Findings From the Research

  1. People in the reviewed experiments forgot from 0% to 94% of what they had learned. The bottom line is that forgetting varies widely.
  2. Even within a restricted time range, learners forgot at wildly differing rates. For example, in the 1-2 day range, learners forgot from 0 to 73%. Learners in the 2-8 year range forgot from 16% to 94%. The obvious conclusion here is that learning varies widely (and wildly) and cannot be predetermined (except perhaps by deities, of whom, I think, we have not even a few in the learning field). To be specific, when we hear statements like, “People will forget 60% of what they learned within 7 days,” we should ignore such advice and instead reflect on our own superiority and good looks until we are decidedly pleased with ourselves.
  3. Even when we looked at only one type of learning material, forgetting varied widely. For example, in Bahrick’s classic 1979 experiment where learners were learning English-Spanish word pairs, learners forgot from 12% to 63%. Even more remarkably, if we include those cases where learners actually remembered more on the second test than the first test, learners’ “forgetting” varied from -41% to 63%, a swing of 104 percentage points! Again, we must conclude that forgetting varies widely.
  4. Many of the experiments reviewed in this report showed clearly that learning methods matter. For example, in the Bahrick 1979 study, the best learning methods produced an average forgetting score of -29% forgetting, whereas the worst learning methods produced forgetting at 47%, a swing of 76% points. In Runquist’s 1983 study, the best learning method produced average forgetting at 34%, whereas all the other learning methods produced average forgetting of 78%. In Allen, Mahler, and Estes’ 1969 experiment, the learners given the best learning methods forgot an average of 2.3%, whereas the learners who got middling learning methods forgot an average of 14.3%, and learners given the worst learning methods forgot approximately 21.7%. The bottom line is that the learning methods we choose make all the difference!!

Check out the full report to learn more about the following:

  • What you should do as a learning professional (in light of these findings).
  • Whether the learning-curve notion still applies.
  • What wisdom each of the 14 research articles revealed.
  • The methodology used in the research.
  • The calculation of forgetting.

 

K-12 Classrooms: Research Review of Multigrade, Multiage, Combination Classrooms

I don’t usually delve into K-12 classroom-based research to any great extent. However, as my daughter’s elementary school is starting a “redesign” process—and I heard arguments on both sides of the multigrade-classroom issue—I thought perhaps I’d give a quick look at the research available.

I had hoped that this would be a quick review, where I would find one or two definitive research reviews in scientific refereed journals, but unfortunately, the research base is rather frail and unclear.

Still, I think the following review does provide some wisdom about how to think about multiage classrooms.

————————————————-

Extent of Review 

I did a moderately quick review—not an exhaustive review—of a couple dozen recent research articles on multigrade classrooms.

This review was conducted in September and October of 2010.

 

————————————————-

Major Conclusions (See More Specific Recommendations Further Below):

The research, although being too scant and too difficult to interpret to make definitive recommendations, generally suggests that multigrade classroom approaches are not likely to produce results that differ substantially from single-grade classrooms.

Specifically, it is likely that the quality of the learning methods utilized and the teacher’s performance in the classroom makes more of difference than whether a multigrade or single-grade approach is utilized.

Multigrade teaching is generally considered more difficult and onerous than single-grade teaching.

If a multigrade approach is utilized, then it should be utilized with due diligence—providing teacher support and development, utilizing team teaching, encouraging many diagnostic opportunities (so that learning can be tailored to learner’s current levels), and so forth.

————————————————- 

Context:    

Researchers and practitioners use many labels for multigrade education, including the following: multigrade, multiage, mixed-age, vertical grouping, combination, composite, double-grade, split-grade, dual-age, hyphenated, nongraded, etc. These terms are often confused, redundant, etc., making researching and thinking about related issues difficult. In this report, I will use the following terms:

  • Single-grade classrooms are comprised of one grade.
  • Multigrade classrooms are comprised of two (or more) grades.
    • Combination Multigrade classrooms are multigrade classrooms that are utilized for logistical reasons.
    • Pedagogic Multigrade classrooms are multigrade classrooms that are utilized for pedagogic (learning-related) reasons.

Sometimes multigrade classrooms are created because of logistical reasons such as declining or uneven enrollments (Veenman, 1996; Burns & Mason, 2002; Mulryan-Kyne, 2007).

Sometimes multigrade classrooms are created for philosophical and pedagogical rationale. There are strong advocacy groups for pedagogic multigrade classrooms. Some of this advocacy is at odds with the research—in the sense that they claim overwhelming benefits for multigrade classrooms when the research is more balanced and uncertain (I noticed this myself in reviewing the research, but also see Burns & Mason, 1997).

It can be helpful to view classes on a continuum from single-grade classes, to combination multigrade classes, to pedagogic multigrade classes (See Mason & Burns, 1997). While many different arrangements are possible, we can talk in generalities as follows:

  • Single-grade classes have students only from one grade.
  • Combination multigrade classes have students from two (or more) grades and one teacher who teaches them.
  • Pedagogic multigrade classes have students from two (or more) grades but are taught by a team of teachers. Pedagogic multigrade classes also tend to be more focused on providing learners individually-tailored learning content than either single-grade or combination multigrade classes.

Teachers in combination multigrade classes are less likely to receive the support and structure they need than teachers in pedagogic multigrade classes (Mason & Burns, 1997).

Multigrade classrooms are not aberrations, but are used throughout the world, and are likely to continue and grow in use in the future (Mulryan-Kyne, 2007).

————————————————-

Quality of Available Research:

Unfortunately, it seems that much of the available research is tainted by methodological weaknesses. Specifically, it appears that more experienced teachers tend to teach multi-age classrooms—so that when advantages appear in the educational results, those advantages may be due to teacher experience as opposed to the multi-age classroom itself. In addition, more affluent students are the ones who tend to be taught in multigrade classrooms—so that when research results show advantages, those could be due to socio-economic and educational advantages of parents as opposed to the multigrade classrooms.

Quotes from the Research:

“Effective research in the area of multiage education is still in its infancy.” From Kinsey (2001).

“The literature on multigrade teaching is relatively sparse, some of it anecdotal in nature and/or of poor quality.” From Mulryan-Kyne (2007).

“There is considerable evidence that principals, in an effort to reduce the burden on multigrade teachers, place more able, more independent, and more cooperative students in multigrade classes.” From Mason and Burns (1996). Note: This is relevant in that good results in comparison to single-age classrooms may be due to having better students in the class.

“No studies in which students were randomly assigned to experimental and control groups were found.” From Veenman (1995). Note: While such random assignment is the gold standard in research, it is difficult to implement in the classroom.

“Because of the lack of distinction between combination and multiage/nongraded classes and the omission of important studies and methodological considerations, it appears that researchers have drawn overly optimistic and erroneous conclusions about the effects of combination classes.” From Mason & Burns (1997).

Conclusion based on Quality of Available Research:

These methodological weaknesses make firm conclusions difficult. Tentative conclusions are still possible.

————————————————-

Teaching is Often More Difficult in Multigrade Classrooms:  

It appears that teaching in a multigrade classroom is more difficult than teaching in a single-age classroom. This conclusion comes from those on all sides of the debate, so it is a fairly strong conclusion.

Quotes from the Research:

“It is commonly stated in the literature that multigrade teaching is more difficult than single grade teaching” From Mulryan-Kyne (2007).

“Multigrade classes hold instructional potential for some, but they are potentially onerous for most. Indeed, we argue that multigrade classes lead to a negative instructional effect and that they increase teachers’ stress and may jeopardize teachers’ motivation and commitment to teaching.” From Mason and Burns (1996).

“Most teachers, when asked about their feelings toward and organizational and teaching strategies for combination classes, responded negatively and preferred not to teach them.” From Mason and Burns (1995).

“The professional knowledge and skills that are relevant and necessary to teaching effectively in single-grade contexts are also relevant and necessary for effective multigrade teaching… However, many of these skills need heightened emphasis in the context of the preparation of teachers for multigrade teaching.” From Mulryan-Kyne (2007).

“[Teachers] report that these classes require more planning, are more difficult to teach, and diminish instruction and curriculum coverage.” From Mason & Burns (1997).

“We concluded that the difficulties teachers face in multigrade classes are centered around five problem areas: (a) the efficient use of instructional time, (b) the design of effective instruction, (c) classroom management, (d) the organization of independent practice or learning, and (e) the formulation of clear and collectively agreed-upon goals for making the multigrade school work.” From Veenman (1996).

“…multigrade teachers pressed for instructional time and the mastery of basic skills (e.g., reading, writing, and mathematics) might neglect science, social studies, and other subjects, which would lead to negative achievement effects in these areas.” From Mason & Burns (1996).

————————————————-

Academic Achievement Results from Multigrade Classrooms:

In the two most credible recent reviews of the research, the findings have shown no differences in academic achievement between multigrade classrooms and single-grade classrooms (Veenman, 1995; Mason & Burns, 1997). However, because of the likelihood that this research is tainted in having better students and teachers in multi-grade classrooms, it is possible that multigrade classrooms produce “at least small negative effects” (Mason & Burns, 1997).

Separating the results for combination multigrade classrooms and pedagogical multigrade classrooms is difficult because researchers haven’t always noted this difference. In Veenman’s (1995) review, both combination and pedagogic multigrade classrooms produced non-significant results—in other words, they were found to produce the same academic results as single-grade classrooms. However, as Mason and Burns (1996) pointed out, Veenman didn’t take into account some important potential biases. Mason and Burns write: “We conclude that multigrade classes have at least a small negative effect on achievement as well as potentially negative effects on teacher motivation.”

Unfortunately, in Mason and Burns (1997) follow-up research review, they only focused on combination multigrade classrooms—NOT pedagogic multigrade classrooms. Their results on combination multigrade classrooms were consistent with Veenman’s in finding no statistical differences, but they warned that “all things being equal, combination classes have at least small negative effects.” Such a negative outcome was found in a 2008 study of combination multigrade classrooms in California (Sims, 2008).

Conclusions on Academic Achievement:

The research is not definitive because there are factors that have not really been teased out as of yet.

In terms of academic results, the tentative conclusion is that multigrade classes on average are probably no better and no worse than single-grade classes, but if they are different from single-grade classes, they may be slightly worse.

————————————————-

Social/Emotional Results from Multi-age Classrooms:

Veenman’s (1995) research found that a majority of combination multigrade classes performed no better than single-grade classes in terms of attitudes towards school, self-concept, and personal and social adjustment.  Mason and Burns (1997) examination of combination multigrade classes found similar non-significant affective benefits.

For pedagogic multigrade classes, however, Veenman (1996) found a “very small” effect, showing slight benefits for pedagogic multigrade classes in terms of attitudes towards school, self-concept, and personal and social adjustment. Again, because of the potential biasing effects in terms of student and teacher selection, these results could be due to bias instead of any benefits for pedagogic multigrade classes.

Conclusions on Social/Emotional Results:

The research is not definitive because there are factors that have not really been teased out as of yet.

In terms of social/emotional results, the tentative conclusion is that multigrade classes on average are probably no better and no worse than single-grade classes. If there are very small benefits, they might be obtained in well-designed pedagogic multigrade classrooms—as opposed to logistically-driven combination multigrade classrooms.

————————————————-

Do some Students Benefit More than Others from Multigrade Classrooms? 

Although this was beyond the scope of my review—I did come across one recent study that dealt specifically with the question of how different types of students might be affected by a multigrade approach (Ong, Allison, & Haladyna, 2000).

First, it should be noted that in this one study, multigrade classrooms tended to outperform single-grade classrooms for all students, regardless of their background. Given that as background, the study found the following:

  • It found no differences between boys and girls.
  • It found that regular students seemed to benefit more from multigrade classrooms than disadvantaged students (that is, Title 1 students).
  • It found that overall non-Hispanic students seemed to benefit more from multigrade classrooms than Hispanic students.

Caveat: This was just one study and should be evaluated with caution.

————————————————-

Overall Recommendations Regarding Multigrade, Multiage, Combination Classrooms:

The scarcity, frailty, and equivocation in the research make strong recommendations impossible. Instead, I offer these tentative ideas for consideration:

1. It is likely that the quality of the learning methods utilized and the teacher’s performance in the classroom makes more of difference than whether a multigrade or single-grade approach is utilized. Given this, it appears that choosing a multigrade approach would be acceptable, though the following points should also be kept in mind.

2. If a multigrade approach is utilized, then it should be utilized with due diligence—providing teacher support and development, utilizing team teaching, encouraging many diagnostic opportunities (so that learning can be tailored to learner’s current levels*), and so forth.

* Recent research indicates that teachers, tutors, and other learning professionals tend not to be very good at providing instructional explanations at learner’s appropriate levels—but that they can improve on this by specifically being informed of their learners’ level of understanding (Wittwer, Nückles, & Renkl, 2010).

3. Multigrade teaching seems to offer the possibility for alternative methods of learning, including group work targeted to different developmental levels. On the other hand, multigrade teaching by itself is no guarantee of good teaching methodology.

4. It should be recognized that multigrade teaching probably has costs associated with its use. Specifically, teachers may have to invest more effort and care in the process and may have a tendency to tradeoff other desirable educational goals.

5. In some forms of multigrade teaching, especially those that utilize team teaching, teachers have to learn the skill levels (in each discipline) of many more students. Because one of the most important aspects of teaching is providing learners with just the right level of instruction, this may cause students to be underserved at first as the teacher learns their students’ skill levels. One way to ameliorate this problem is for students entering a multigrade cycle to be diagnosed through testing, problem solving, and other performance metrics early on. This “slow-start” issue has a flip side—because multigrade classes stretch into two (or more) years, students in subsequent years will probably experience accelerated learning due to deeper understanding of each student by teachers and hence better instructional scaffolding.

6. Teachers with less experience may be especially unprepared for multigrade teaching. With experience, teachers learn how to automate aspects of their classroom performance so that they can utilize their limited working memory capacity to focus on supporting their learners in learning (for example, differentiating their instruction, etc.). Newer teachers will be unlikely to provide the necessary instructional scaffolding to perform at a high level.

7. Students who have behavioral problems or who are less competent in learning may make the task of multigrade teaching more difficult. This may suggest that extra effort and guidance in the early grades is warranted or that a multigrade approach should be delayed until a time when most students are ready to engage fully in the process.

————————————————-

Research References:

Burns, R. & Mason, D. (2002). Class composition and student achievement in elementary schools. American Educational Research Journal, 39 (1), 207-233.

Burns, R. B., & Mason, D. A. (1998). Class formation and composition in elementary schools. American Educational Research Journal, 35(4), 739-772.

Kinsey, S. (2001). Multiage Grouping and Academic Achievement. ERIC Digest, January 2001.

Mason, D. A., & Burns, R. B. (1995). Teachers’ views of combination classes. Journal of Educational Research, 89(1), 36-45.

Mason, D. A., & Burns, R. B. (1996). “Simply no worse and simply no better” may simply be wrong: A critique of Veenman’s conclusion about multigrade classes. Review of Educational Research, 66(3), 307-322.

Mason, D., & Burns, R. (1997). Reassessing the effects of combination classes. Educational Research and Evaluation, 3(1), 1–53.

Mulryan-Kyne, C. (2007). The preparation of teachers for multigrade teaching. Teaching and Teacher Education, 23(4), 501-514.

Ong, W., Allison, J., & Haladyna, T. M. (2000). Student achievement of 3rd-graders in comparable single-age and multiage classrooms. Journal of Research in Childhood Education, 14(2), 205-215.

Sims, D. (2008). A strategic response to class size reduction: Combination classes and student achievement in California. Journal of Policy Analysis and Management, 27(3), 457-478.

Veenman, S. (1995). Cognitive and noncognitive effects of multigrade and multi-age classes: A best-evidence synthesis. Review of Educational Research, 65, 319-381.

Veenman, S. (1996). Effects of multigrade and multi-age classes reconsidered. Review of Educational Research, 66(3), 323-340.

Wittwer, J., Nückles, M., & Renkl, A. (2010). Using a diagnosis-based approach to individualize instructional explanations in computer-mediated communication. Educational Psychology Review, 22(1), 9-23.

————————————————-

Some of the articles reviewed but not cited:

Song, R.  Spradlin, T. E., & and Plucker, J. A. (2009). The Advantages and Disadvantages of Multiage Classrooms in the Era of NCLB Accountability. Education Policy Brief, 7, 1-7. Published by the Center for Evaluation and Education Policy of Indiana University.

Why not cited:
This article is not from refereed journal so it may not have been fully vetted. Also, there are hints of bias in the article, for example, (a) providing invited proponents of multiage classrooms to describe its value, without providing a similar counterpoint, and (b) talking about the lack of good research but then plowing ahead with a list of the “benefits (perceived and real)” of multiage classrooms—so in a real sense the article ignores the research by plowing ahead to benefits. Still, I wouldn’t discount this article completely. Its research review seems good when it is limited to reviewing the actual research. Also, it makes recommendations that are mostly consistent with the findings contained in this review, so I can’t dismiss their conclusions. The bottom line is that this review does not add anything to a review of the research, and, because it is not vetted in a refereed journal and appears slightly biased, I think it safer not to cite it as separate evidence.

Hoffman, J. (2003). Multiage teachers beliefs and practices. Journal of Research in Childhood Education, 18, 5-17.

Why not cited:
This article looked at only 4 teachers, all who had chosen to be teaching in multiage classrooms. It was really a descriptive research project and did not look at actual learning outcomes. No single-grade teachers were examined so we don’t really know how different results with single-grade teaching might be.

Linley, L. (1999). Multi-Age Classes and High Ability Students. Review of Educational Research, 69, 187-212.

Why not cited:
This article was a review of other reviews, not a review of research studies itself. Also, it focused only on high-ability students.

Gerard, M. (2005). Bridging the gap: Understanding young children’s thinking in multiage groups. Journal of Research in Childhood Education, 19, 243-250.

Why not cited:
This article utilized an incredibly biased research design. It took one classroom at one school that used multiage grouping and compared it on standardized exams with the national average.

Mariano, S., & Kirby N. (2009). Achievement of Students in Multigrade Classrooms Evidence from the Los Angeles Unified School District. From the RAND Education Working Paper Series (WR-685-IES).

Why not cited:
This article is not from refereed journal so it may not have been fully vetted. It also uses somewhat opaque statistical methods to estimate findings, not looking directly at actual comparisons between multigrade and single-grade classrooms. I must admit that I don’t fully understand all the statistical employments utilized (for example, “doubly robust regression,” “non-parametric generalized boosting,” “the Kolmogorov-Smirnov (K-S) statistic”). Thus, I may be missing the full implications of the research employed.

Neon Elephant Award

, , , , ,

 

The Neon Elephant Award

The Neon Elephant Award is awarded to a person, team, or organization exemplifying enlightenment, integrity, and innovation in the field of workplace learning and performance. Announced in December—during the time of year when the northern hemisphere turns away from darkness toward the light and hope of warmer days to come—the Neon Elephant Award honors those who have truly changed the way we think about the practice of learning and performance improvement. Award winners are selected for demonstrated success in pushing the field forward in significant paradigm-altering ways while maintaining the highest standards of ethics and professionalism. 

 

Why “Neon Elephant?”

The elephant represents learning, power, strength, and the importance of nurturing the community. The glow of neon represents enlightenment, illumination, and a spark of something unique and alluring.

 

Selection Methodology

The award is based purely on merit and the criteria detailed above. Proposals are not accepted, nor are any entrance fees solicited or accepted. While advice on the selection is sought from industry thought leaders, Dr. Will Thalheimer of Work-Learning Research is the final arbiter. Awards will only be made in years when exceptional contributions to the workplace learning and performance field are apparent.

 

Winners

The 2025 Neon Elephant Award was awarded to Matt Beane for writing the book, The Skill Code: How to Save Human Ability in the Age of Intelligent Machines—and for his many years researching how humans do work interacting with machines and the people with whom they work.

The 2024 Neon Elephant Award was awarded to Mary C. Murphy and Julie Dirksen—Mary for writing the book, Cultures of Growth: How the New Science of Mindset Can Transform Individuals, Teams, and Organizations; and for Julie for writing the book, Talk to the Elephant: Design Learning for Behavior Change—and for their many contribution to research and research translation.

The 2023 Neon Elephant Award was awarded to Annie Murphy Paul for writing the book The Extended Mind: The Power of Thinking Outside the Brain—and for her lifetime of work bringing research wisdom to light through her books, journalism, writing, and speaking.

The 2022 Neon Elephant Award was awarded to Donald Clark for writing the book Learning Experience Design: How to Create Effective Learning that Works and for the Great Minds On Learning webcast and podcast done in collaboration with John Helmer—and for his tireless work of many years advocating for research-based learning designs and effective innovations in learning technology.

The 2021 Neon Elephant Award was awarded to Clark Quinn and Patti Shank—to Clark Quinn for writing the book, Learning Science for Instructional Designers and Patti Shank for writing the book Write Better Multiple-Choice Questions to Assess Learning—and for their many years translating learning research into practical recommendations.

The 2020 Neon Elephant Award was awarded to Mirjam Neelen and Paul Kirschner for writing the book, Evidence-Informed Learning Design: Use Evidence to Create Training Which Improves Performance—and for their many years publishing their blog 3-Star Learning Experiences.

The 2019 Neon Elephant Award was awarded to David Epstein for writing the book Range: Why Generalists Triumph in a Specialized World, and for his many years as a journalist and science-inspired truth teller.

The 2018 Neon Elephant Award was awarded to Clark Quinn for writing a book debunking the learning myths, Millennials, Goldfish & Other Training Misconceptions: Debunking Learning Myths and Superstitions—and for his many years advocating for research-based practices in the workplace learning field.

The 2017 Neon Elephant Award was awarded to Patti Shank for writing and publishing two research-to-practice books this year, Write and Organize for Deeper Learning and Practice and Feedback for Deeper Learning—and for her many years advocating for research-based practices in the workplace learning field.

The 2016 Neon Elephant Award was awarded to Pedro De Bruyckere, Paul A. Kirschner, and Casper D. Hulshof for their book Urban Myths about Learning and Education—a book that provides a research-based reality check on the myths and misinformation that float around the learning field.

The 2015 Neon Elephant Award was awarded to Julie Dirksen for her book, Design for How People Learn—a book that wonderfully conveys practical, research-based wisdom through the authentic voice of an experienced instructional designer and strategist.

The 2014 Neon Elephant Award was awarded to Peter C. Brown, Henry L. Roediger III, and Mark A. McDaniel for their book, Make it Stick: The Science of Successful Learning—a book that brilliantly conveys scientific principles of learning in prose that is easy to digest, comprehensive and true in its recommendations, highly-credible, and impossible to ignore or forget.

The 2013 Neon Elephant Award was awarded to Gary Klein for his many years doing research and practice in naturalistic decision making, cognitive task analysis, and insight learning–and for reminding us that real-world explorations of human behavior are essential in enabling us to distill key insights.

The 2012 Neon Elephant Award was awarded to K. Anders Ericsson for his many years conducting research on expertise and creating a body of knowledge that has inspired many others to translate his research into recommendations for use by performance-improvement professionals.

The 2011 Neon Elephant Award was awarded to Jeroen van Merriënboer for his many years conducting research on learning and translating that research into practical models for use by learning professionals.

The 2010 Neon Elephant Award was awarded to Richard E. Clark for his many years in leading the workplace learning-and-performance field by bridging the gap between academic research and practical application.

The 2009 Neon Elephant Award was awarded to Ruth Clark for her many years in leading the workplace learning-and-performance field with research-based insights and recommendations, and—by so doing—helping to professionalize our field.

The 2008 Neon Elephant Award was awarded to Robert Brinkerhoff for developing the Success Case evaluation method and for advocating that learning professionals play a more “courageous” role in their organizations.

The 2007 Neon Elephant Award was awarded to Sharon Shrock and Bill Coscarelli for advocating against the use of memorization-level questions in learning measurement and for the use of authentic assessment items, including scenario-based questions, simulations, and real-world skills tests.

The 2006 Neon Elephant Award was awarded to Cal Wick of the Fort Hill Company for his work developing methodologies and software to support learning transfer.

Cognitive Load Theory Coming Under Withering Attacks

First I must explain that there is a difference between empirical research findings and the theoretical formulations that human researchers create to explain their findings. To reiterate, we have:

  1. research findings (data)
  2. theoretical explanations (rationales that researchers invent)

The data can be true, while the theoretical explanations can be wrong.

Academic researchers are paid the big bucks—and gain the highest psychic rewards—for developing theories.

As you know if you’ve followed my work for any length of time, I put much more faith in data than in theories.

So, while I am about to share criticisms of a theory, I think the research findings are still sound.

Here are some recent criticisms of Cognitive Load Theory:

—————–

Ton de Jong says:

What has cognitive load theory brought to the field of
educational design? The three main recommendations that come from cognitive
load theory are: present material that aligns with the prior knowledge of the
learner (intrinsic load), avoid non-essential and confusing information
(extraneous load), and stimulate processes that lead to conceptually rich and
deep knowledge (germane load). These design principles have been around in educational
design for a long time (see e.g., Dick and Carey 1990; Gagne´ et al. 1988; Reigeluth
1983). Work in cognitive load theory often denies the existence of this earlier
research, as illustrated in the following quote by Ayres (2006a, p. 288):
‘‘Whereas strategies to lower extraneous load are well documented…methods to
lower intrinsic load have only more recently been investigated’’ (p. 288). In
his study, Ayres introduces part-tasks as one of the initial approaches to
lower cognitive load. Describing this as a ‘‘recent’’ approach denies much of
the history of instructional design.

de Jong, T. (2010). Cognitive load theory, educational
research, and instructional design: Some food for thought. Instructional
Science, 38(2), 105-134.

—————–

Roxana Moreno says:

Under the light of CLT’s [Cognitive Load Theory’s]
fundamental limitations, I will make the argument that continuing to use the theory
to frame instructional design research is instilling the idea that educational
research cannot aspire to have the same scientific value as that of the hard sciences
(Diamond 1987). The following are some reasons why this might be the case. When
educational researchers are not able to demonstrate that they are making
progress, they give further reasons to believe that the learning sciences are a
lesser form of knowledge (Labaree 1998). Second, although a strength of CL
research is the use of controlled experimental studies—one of the exemplary
methods of scientifically based research (Eisenhart and Towne 2003)—it has
failed to develop adequate methods that permit direct investigation of the
research questions at stake. Science relies on measurements or observational
methods that provide reliable and valid data across studies by the same or
different investigators (National Research Council 2002).

Third, in any science, researchers construct towers of
knowledge on the foundations of the work of others. de Jong raises a valid
concern about the fact that CL research often ignores the existence of earlier
research and theories that may better account for the findings than CLT. The
dangers of this isolated approach to science are clearly stated by Labaree (1998)
‘‘At the end of long and distinguished careers, senior educational researchers
are likely to find that they are still working on the same questions that
confronted them at the beginning. And the new generation of researchers they
have trained will be taking up these questions as well (p. 9).’’

Lastly, although bias may not be completely avoidable,
scientists are expected to be aware of potential bias sources in their work.
One safeguard against bias in any area of study is to be open to reflection and
scrutiny. It is the professional responsibility of educational researchers to
evaluate the state of current knowledge on a regular basis, identify knowledge
gaps, and lay the scientific principles for future investigation. Engaging in
this ‘effortful’ practice is key in fostering a scientific community and
culture.

Moreno, R. (2010). Cognitive load theory: More food for
thought. Instructional Science, 38(2), 135-141.

—————–

Schnotz and Kurschner (2007) say:

Numerous empirical studies have demonstrated that
traditional instruction can and should be re-designed according to principles
of cognitive load theory, and that this re-design results in better learning.
However, there are also numerous conceptual problems related to cognitive load
theory, which sometimes make interpretation of empirical findings difficult.
Although the concept of cognitive load has been frequently described in general
terms and although definitions have been provided for different kinds of
cognitive load, a closer look reveals that the exact nature of these different
kinds of load is not sufficiently clear yet. Further clarification is needed
regarding the relations between different kinds of cognitive load and whether
they can and how they should be manipulated to enhance learning. Other open questions
refer to the role of working memory in the process of learning. Although
working memory is a key concept in cognitive load theory, it is not
sufficiently clear to what extent working memory is in fact required for
learning. Finally, further clarification is needed whether and in which way
different kinds of cognitive load constrain each other, how they relate to the
process of learning and, last not least, how they can be measured.

Schnotz, W., & Kurschner, C. (2007). A reconsideration of
cognitive load theory. Educational
Psychology Review, 19, 469–508.

—————–

So, the theory is shaky, even though it has generated a slew of great research.

The data is still compelling, so, among other things, we can still use worked examples.

  1. Worked examples are useful for novice learners.
  2. Worked examples may hurt more experienced learners. Better to utilize practice problems for more experienced learners.