Tag Archive for: review

Last month I released a new online, self-paced workshop called Presentation Science: How to Help Your Audience to Engage, Learn, Remember, and Act. The workshop is comparable to a two-day workshop and comprises about 12.5 hours of work, including videos, scenario questions, reflection questions, discussions, and a final assessment.

 

People are beginning to “graduate” from the workshop. Here’s what the first two graduates had to say:

Powerful content here! I love this course. It’s the best online course I’ve taken–ever! I only see one problem with the course. You’ve set the price too low based on the actual value of the course! It’s worth much more than what you are charging, considering the quality. I’d set the price at $1,000 minimum personally! In my opinion, it is worth $10,000 in the first 6 months once a person has successfully applied this to build new trainings!

— Gale Stafford, executive coach and learning architect at County of San Mateo

Will Thalheimer’s Presentation Science Workshop provides a TON of strategies, tactics, and tools backed up by learning science that will help you transform your bullet-riddled, mind-numbing PowerPoint presentations into meaningful, memorable, motivating, and (yes!) magnificent learning events.

— Holly H., senior instructional designer at global energy technology company

 

 

As you can imagine, I’m thrilled with this response. At the same time, I feel a responsibility to continue making the workshop better and better. At a later date–when I’ve gathered more data–I will write about how I think the new online-learning technologies are now poised to enable great learning designs. I’ll also talk about how to utilize these tools to follow research-inspired recommendations. For now, I’m just going to brag a bit! SMILE

And encourage you to consider taking this course for yourself, or recommending it to your organization, your subject-matter experts, trainers, teachers, professors, managers, salespeople, executives—anybody who has to give a presentation that has to be maximally effective.

 

The Presentation Science Workshop:
Learn More by Clicking Here!

 

 

The 70-20-10 Framework has been all the rage for the last five or ten years in the workplace learning field. Indeed, I organized a great debate about 70-20-10 through The Debunker Club (you can see the tweet stream here). I have gone on record saying that the numbers don’t have a sound research backing, but that the concept is a good one—particularly the idea that we as learning professionals ought to leverage on-the-job learning where we can.

What is 70-20-10?

The 70-20-10 framework is built on the belief that 10% of workplace learning is, or should be, propelled by formal training; that 20% is, or should be, enabled by learning directly from others; and that 70% of workplace learning is, or should, come from employee’s learning through workplace experiences.

Supported by Research?

Given all the energy around 70-20-10, you might think that lots of rigorous scientific research has been done on the framework. Well, you would be wrong!

In fact, up until today (April 19, 2019), only one study has been published in a scientific journal (my search of PsycINFO only reveals one study). In this post, I will review that one study, published last year:

Johnson, S. J., Blackman, D. A., & Buick, F. (2018). The 70:20:10 framework and the transfer of learning. Human Resource Development Quarterly. Advance online publication.

Caveats

All research has strengths, weaknesses, and limitations—and it’s helpful to acknowledge these so we can think clearly. First, one study cannot be definitive, and this is just one study. Also, this study is qualitative and relies on subjective inputs to draw its conclusions. Ideally, we’d like to have more objective measures utilized. It is also gathering data from a small sample of public sector workers, where ideally we want a wider range of diverse participants.

Methodology

The researchers found a group of organizations who had been bombarded with messages and training to encourage the use of the 70-20-10 model. Specifically, the APSC (The Australian Public Sector Commission), starting in 2011, encouraged the Australian public sector to embrace 70-20-10.

The specific study “draws from the experiences of two groups of Australian public sector managers: senior managers responsible for implementing the 70:20:10 framework within their organization; and middle managers who have undergone management capability development aligned to the 70:20:10 framework. All managers were drawn from the Commonwealth, Victorian, Queensland, and Northern Territory governments.”

A qualitative approach was chosen according to the researchers “given the atheoretical nature of the 70:20:10 framework and the lack of theory or evidence to provide a research framework.”

The qualitative approaches used by the researchers were individual structured interviews and group structured interviews.

The researchers chose people to interview based on their experience using the 70-20-10 framework to develop middle managers. “A purposive sampling technique was adopted, selecting participants who had specific knowledge of, and experience with, middle management capability development in line with the 70:20:10 framework.”

The researchers used a text-processing program (NVivo) to help them organize and make sense of the qualitative data (the words collected in the interviews). According to Wikipedia, “NVivo is intended to help users organize and analyze non-numerical or unstructured data. The software allows users to classify, sort and arrange information; examine relationships in the data; and combine analysis with linking, shaping, searching and modeling.”

Overall Results

The authors conclude the following:

“In terms of implications for practice, the 70:20:10 framework has the potential to better guide the achievement of capability development through improved learning transfer in the public sector. However, this will only occur if future implementation guidelines focus on both the types of learning required and how to integrate them in a meaningful way. Actively addressing the impact that senior managers and peers have in how learning is integrated into the workplace through both social modeling and organizational support… will also need to become a core part of any effective implementation.”

“Using a large qualitative data set that enabled the exploration of participant perspectives and experiences of using the 70:20:10 framework in situ, we found that, despite many Australian public sector organizations implementing the framework, to date it is failing to deliver desired learning transfer results. This failure can be attributed to four misconceptions in the framework’s implementation: (a) an overconfident assumption that unstructured experiential learning will automatically result in capability development; (b) a narrow interpretation of social learning and a failure to recognize the role social learning has in integrating experiential, social and formal learning; (c) the expectation that managerial behavior would automatically change following formal training and development activities without the need to actively support the process; and (d) a lack of recognition of the requirement of a planned and integrated relationship between the elements of the 70:20:10 framework.”

Specific Difficulties

With Experiential Learning

“Senior managers indicated that one reason for adopting the 70:20:10 framework was that the dominant element of 70% development achieved through experiential learning reflected their expectation that employees should learn on the job. However, when talking to the middle managers themselves, it was not clear how such learning was being supported. Participants suggested that one problem was a leadership perception across senior managers that middle managers could automatically transition into middle management roles without a great deal of support or development.”

“The most common concern, however, was that experiential learning efficacy was challenged because managers were acquiring inappropriate behaviors on the job based on what they saw around them every day.”

“We found that experiential learning, as it is currently being implemented, is predominantly unstructured and unmanaged, that is, systems are not put in place in the work environment to support learning. It was anticipated that managers would learn on the job, without adequate preparation, additional support, or resourcing to facilitate effective learning.”

With Social Learning

“Overall, participants welcomed the potential of social learning, which could help them make sense of their con-text, enabling both sense making of new knowledge acquired and reinforcing what was appropriate both in, and for, their organization. However, they made it clear that, despite apparent organizational awareness of the value of social learning, it was predominantly dependent upon the preferences and working styles of individual managers, rather than being supported systematically through organizationally designed learning programs. Consequently, it was apparent that social learning was not being utilized in the way intended in the 70:20:10 framework in that it was not usually integrated with formal or experiential learning.”

Mentoring

“Mentoring was consistently highlighted by middle and senior managers as being important for both supporting a middle manager’s current job and for building future capacity.”

“Despite mentoring being consistently raised as the most favored form of development, it was not always formally supported by the organization, meaning that, in many instances, mentoring was lacking for middle managers.”

“A lack of systemic approaches to mentoring meant it was fragile and often temporary.”

Peer Support

“Peer support and networking encouraged middle managers to adopt a broader perspective and engage in a community of practice to develop ideas regarding implementing new skills.”

“However, despite managers agreeing that networks and peer support would assist them to build capability and transfer learning to the workplace, there appeared to be few organizationally supported peer learning opportunities. It was largely up to individuals to actively seek out and join their own networks.”

With Formal Learning

“Formal learning programs were recognized by middle and senior managers as important forms of capability development. Attendance was often encouraged for new middle managers.”

“However, not all experiences with formal training programs were positive, with both senior and middle managers reflecting on their ineffectiveness.”

“For the most part, participants reported finishing formal development programs with little to no follow up.”

“There was a lack of both social and experiential support for embedding this learning. The lack of social learning support partly revolved around the high workloads of managers and the lack of time devoted to development activities.”

“The lack of experiential support and senior management feedback meant that many middle managers did not have the opportunity to practice and further develop their new skills, despite their initial enthusiasm.”

“A key issue with this was the lack of direct and clear guidance provided by their line managers.”

“A further issue with formal learning was that it was often designed generically for groups of participants…  The need for specificity also related to the lack of explicit, individualized feedback provided by their line manager to reinforce and embed learning.”

What Should We Make of This Preliminary Research?

Again, with only one study—and a qualitative one conducted on a narrow type of participant—we should be very careful in drawing conclusions.

Still, the study can be helpful in helping us develop hypotheses for further testing—both by researchers and by us as learning professionals.

We also ought to be careful in casting doubt on the 70-20-10 framework itself. Indeed, the research seems to suggest that the framework was not always implemented as intended. On the other hand, when it is demonstrated that a model tends to be used poorly in its routine use, then we should become skeptical that it will produce reliable benefits.

Here are a list of reflections generated in me by the research:

  1. Why so much excitement for 70-20-10 with so little research backing?
  2. Formal training was found to have all the problems normally associated with it, especially the lack of follow-through and after-training support—so we still need to work to improve it!
  3. Who will provide continuous support for experiential and social learning? In the research case, the responsibility for implementing on-the-job learning experiences was not clear, and so the implementation was not done or was poorly done.
  4. What does it take in terms of resources, responsibility, and tasking to make experiential and social learning useful? Or, is this just a bridge too far?
  5. The most likely leverage point for on-the-job learning still seems, to me, to be managers. If this is a correct assumption—and really it should be tested—how can we in Learning & Development encourage, support, and resource managers for this role?

Sign Up For Will’s News by Clicking Here

 

 

 

For millennium, scholars and thinkers of all sorts — from scientists to men or women on the street — thought that memories simply faded with time.

Locke said:

"The memory of some men, it is true, is very tenacious, even to a miracle; but yet there seems to be a constant decay of all our ideas, even of those which are struck deepest, and in the minds of the most retentive; so that if they be not sometimes renewed by repeated exercise of the senses, or reflection on those kinds of objects which at first occasioned them, the print wears out, and at last there remains nothing to be seen."  John Locke quoted by William James in Principles of Psychology (p. 445, the 1952 Great Books edition, original 1891).

However, in the mid 1900's research by McGeoch (1932), Underwood (1957) and others found that memories can fade when what is learned interferes with other things learned. Previous things learned can interfere with current learning (proactive interference) and current learning can be interfered with by subsequent learning (retroactive interference).

The debate between decay and interference went on for over a century! Indeed, it paralleled the debate in physics over the property of light. Is it a wave or a particle?

The first ever photograph of light as both a particle and wave

In physics, the debate was so important that Albert Einstein won the Nobel Prize for the solution. Einstein's solution was simple. Light was BOTH a wave and a particle. The picture above is reported by Phys.org to be the first photograph demonstrating light's dual properties.

Now in the psychological research, we have the first experimental evidence that forgetting may be caused by BOTH decay and interference.

In a clever experiment, published just this month, Talya Sadeh, Jason Ozubko, Gordon Winocur, and Morris Moscovitch found evidence for both interference and decay.

Their research appears to be inspired, at least partially, by neuroscience findings. Here's what the authors say:

"Two approaches have guided current thinking regarding the functional distinction between hippocampal and extrahippocampal memories. The first approach maintains that the hippocampus supports a mnemonic process termed recollection, whereas extrahippocampal structures, especially the perirhinal cortex, support a process termed familiarity… Recollection is a mnemonic process that involves reinstatement of memory traces within the context in which they were formed. Familiarity is a mnemonic process that manifests itself in the feeling that a studied item has been experienced, but without reinstating the original context." (p. 2)

To be clear, this was NOT a neuroscience experiment. They did not measure brain activity in any way. They measured behavioral findings only.

In their experiment, they had people engage in a word-recognition task and then gave them either (1) another word-learning task, (2) a short music task, or (3) a long music task. The first group's word-learning task was designed to create the most interference. The longer music task was designed to create the most decay (because it took longer).

The results of the experiment were consistent with the researcher's hypotheses. They claimed to have found evidence for both decay and interference.

Caveats

Every scientific experiment has caveats. Usually these are pointed out by the researchers themselves. Often, it takes an outside set of eyes to provide caveats.

Did the researchers prove, beyond the shadow of a doubt, that forgetting has two causes? Short answer: No! Did they produce some interesting findings? Maybe!

My big worry from a research-design perspective is that their manipulation distinguishing between recollection and familiarity is somewhat dubious, seemingly splitting hairs in the questions they ask the learners. My big worry from a practical learning-design perspective is that they are using words as learning materials. First, most important learning situations utilize more complicated materials. Second, words are associative by their very nature — thus more likely to react to interference than typical learning materials. Third, the final "test" of learning was a recognition-memory task that involved learners determining whether they remembered seeing the words before — again, not very relevant to practical learning situations.

Practical Ramifications for Learning Professionals

Since there are potential experimental-design issues, particularly from a practical standpoint, it would be an extremely dubious enterprise to draw practical ramifications. Let me be dubious then (because it's fun, not because it's wise). If the researchers are correct, that memories that are context-based are less likely to be subject to interference effects; we might want to follow the general recommendation — often made today by research-focused learning experts — to provide learners with realistic practice using stimuli that have contextual relevance. In short, teach "if situation–then action" rather than teaching isolated concepts. Of course, we didn't need this experiment to tell us that. There is a ton of relevant research to back this up. For example, see The Decisive Dozen research review.

Beyond the experimental results, the concepts of delay and interference are intriguing in and of themselves. We know people tend to slide down a forgetting curve. Perhaps from interference, perhaps from decay. Indeed, as the authors say, "it is important to note that interference and decay are inherently confounded."

Research

The experiment:

Sadeh, T., Ozubko, J. D., Winocur, G., & Moscovitch. M. (2016) Forgetting patterns differentiate between two forms of memory representation. Psychological Science OnlineFirst, published on May 6, 2016 as doi:10.1177/0956797616638307.

The research review:

Sadeh, T., Ozubko, J. D., Winocur, G., & Moscovitch, M. (2014). How we forget may depend on how we remember. Trends in Cognitive Sciences, 18, 26–36.

 

 

A recent research review (by Paul L. Morgan, George Farkas, and Steve Maczuga) finds that teacher-directed mathematics instruction in first grade is superior to other methods for students with “math difficulties.” Specifically, routine practice and drill was more effective than the use of manipulatives, calculators, music, or movement for students with math difficulties.

For students without math difficulties, teacher-directed and student-centered approaches performed about the same.

In the words of the researchers:

In sum, teacher-directed activities were associated with greater achievement by both MD and non-MD students, and student-centered activities were associated with greater achievement only by non-MD students. Activities emphasizing manipulatives/calculators or movement/music to learn mathematics had no observed positive association with mathematics achievement.

For students without MD, more frequent use of either teacher-directed or student-centered instructional practices was associated with achievement gains. In contrast, more frequent use of manipulatives/calculator or movement/music activities was not associated with significant gains for any of the groups.

Interestingly, classes with higher proportions of students with math difficulties were actually less likely to be taught with teacher-directed methods — the very methods that would be most helpful!

Will’s Reflection (for both Education and Training)

These findings fit in with a substantial body of research that shows that learners who are novices in a topic area will benefit most from highly-directed instructional activities. They will NOT benefit from discovery learning, problem-based learning, and similar non-directive learning events.

See for example:

  • Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist, 41(2), 75-86.
  • Mayer, R. E. (2004). Should There Be a Three-Strikes Rule Against Pure Discovery Learning? American Psychologist, 59(1), 14-19.

As a research translator, I look for ways to make complicated research findings usable for practitioners. One model that seems to be helpful is to divide learning activities into two phases:

  1. Early in Learning (When learners are new to a topic, or the topic is very complex)
    The goal here is to help the learners UNDERSTAND the content. Here we provide lots of learning support, including repetitions, useful metaphors, worked examples, immediate feedback.
  2. Later in Learning (When learners are experienced with a topic, or when the topic is simple)
    The goal here is to help the learners REMEMBER the content or DEEPEN they’re learning. To support remembering, we provide lots of retrieval practice, preferably set in realistic situations the learners will likely encounter — where they can use what they learned. We provide delayed feedback. We space repetitions over time, varying the background context while keeping the learning nugget the same. To deepen learning, we engage contingencies, we enable learners to explore the topic space on their own, we add additional knowledge.

What Elementary Mathematics Teachers Should Stop Doing

Elementary-school teachers should stop assuming that drill-and-practice is counterproductive. They should create lesson plans that guide their learners in understanding the concepts to be learned. They should limit the use of manipulatives, calculators, music, and movement. Ideas about “arts integration” should be pushed to the back burner. This doesn’t mean that teachers should NEVER use these other methods, but they should be used to create occasional, short, and rare moments of variety. Spending hours using manipulatives, for example, is certainly harmful in comparison with more teacher-directed activities.

 

One of the biggest gaps in the learning-and-performance
field occurs after the training is done. Learners fail to apply what they’ve
learned and their managers fail to support training implementation.
Fortunately, the Fort Hill gang writes again. Where their blockbuster book, The Six Disciplines of Breakthrough Learning,
laid out a comprehensive process for getting training results, their new book (Getting Your Money’s Worth from Training and
Development
) provides a call-to-action for training’s most important
players. Using the brilliantly diabolical approach of dividing the book in
half—one half for learners, the other for managers—Jefferson, Pollock, and Wick
provide an energizing action-plan to help organizations maximize training’s
impact on job performance.

I’m so impressed with the Fort Hill guys. It seems that they
(1) have looked deeply at the training-and-development trade, (2) found an area
where time and time again we fail to do what’s right, and (3) written the
perfect book to ensure that training maximizes business results. Too often in
today’s organizations, training is seen as magic pill that works without alignment
and support. In this double-dose of a book, Jefferson, Pollock, and Wick
explode that myth, helping both learners and their managers bring potency to
the training effort.

The design of the book tells the story itself. Managers read
from one cover while learners read from the other cover. The book’s title stays
the same—Getting Your Money’s Worth from
Training and Development
—but the subtitles change for the two audiences (i.
e., A Guide to Breakthrough Learning for Managers; A Guide to Breakthrough
Learning for Participants.
). Only
in partnership is training truly effective. The symbolism speaks loudly, but so
too does the content, showing how both learners and their managers can work
together to ensure that training transfers to on-the-job performance
improvement.

The book is written in a conversational style. It speaks
directly to the audiences in terms that will resonate. No motherhood and apple
pie in the Fort Hill world. It’s all about results, wiifm’s, and tools. The
example worksheets in the back of each book (remember it’s two books in one)
are worth the cover price.

I recommend this book with the greatest enthusiasm. Companies
ought to buy two copies for every training participant. One for the participant
and one for his/her manager.

You can click the link below to learn more about the book (and go directly to Amazon to decide whether to purchase it).


Training Media Review, a hearteningly unbiased source, just releases its 2009 report on Learning Management Systems.

Click to learn more…