This is NOT a post about Bob Mager. It is something else entirely.
In probably the best video I will ever create, I made the case that learning professionals and learners should NOT receive the same set of learning objectives.
The rationale is this: Because objectives are designed to guide behavior, how could one statement possibly guide the behaviors of two separate audiences? Sometimes maybe! But not always!
Arguments for the Infallibility of an Instructional-Design Hero
Recently, I’ve heard it argued that Bob Mager, in his classic text, “Preparing Instructional Objectives,” urged us to create instructional objectives only for us as learning professionals, that he never intended that instructional objectives be presented to learners. This is a testable assertion, which is great! We can agree that Mager gave us some good advice on how to craft objectives for ourselves as learning professionals. But did Mager also, perhaps, suggest that objectives could be presented to learners?
Here are several word-for-word quotes from Mager’s book:
Page 16: Heading: “Goal Posts for Students”
Page 16: “Clearly defined objectives also can be used to provide students with the means to organize their own time and efforts toward accomplishment of those objectives.”
Page 17: “With clear objectives, it is possible to organize the instruction itself so that instructors and students alike can focus their efforts on bridging the gap…”
Page 19: Chapter Summary. “Objectives are useful for providing: … Tools for guiding student efforts…”
Page 43: “Objectives in the hands of students prevent the students from having to guess at how they might best organize their time and effort.”
So Mager clearly started the confusion! But Mager wrote at a time before research on cognition enabled greater insight.
Forget Mager’s contribution. The big problem is that the most common practice seems to still be efforts to create a set of learning objectives to use for both learners and learning practitioners.
I was even scolded for not knowing the difference between an instructional objective (for learning professionals) and a learning objective (for learners). Of course, these revisionist definitions are not true and are not helpful. They are fake news, concocted perhaps by a person who thinks or was taught that our instructional-design heroes are perfect and their work is sacrosanct. The truth is that these terms have been used interchangeably. For example, in a research study by my mentor and academic advisor, Ernie Rothkopf, he and his research partner used the term instructional objectives to refer to objectives presented to learners.
Rothkopf, E. Z., & Kaplan, R. (1972). An exploration of the effect of density and specificity of instructional objectives on learning from text. Journal of Educational Psychology, 6, 295-302.
My Main Points
- We need at least two types of objectives (although I’ve argued for more)—one to guide the design, development, and evaluation of learning; one to guide learners as they are learning. I’ve called these “focusing objectives,” because the research shows that they guide attention toward objective-relevant content.
- When we make arguments, we ought to at least skim the sources to see if we know what we’re talking about.
- We ought to stop with hero worship. All of us do some good things and some bad things. Even the best of us.
- Hero worship in the learning field is particularly problematic because learning is so complex and we all still have so much to learn. All of us attempting to make recommendations are likely to be wrong some of the time.
- It is ironic that our schools of instructional design teach graduate students to memorize facts and hold up heroes as infallible immortals—when instead they ought to be educating these future citizens how progress gets made over long periods of time by a large collective of people. They also ought to be teaching students to understand at a deeper level, not just a knowledge level. But truly, we can’t blame the schools of instructional design. After all, they started with canonically-correct instructional objectives (focused on low-level knowledge because they are easier to create).
Finally, let me say that in the video I praise Bob Mager’s work on learning objectives for us learning professionals. This post is not about Mager.
I read a brilliantly clear article today by Karen Hao from the MIT Technology Review. It explains what machine learning is and provides a very clear diagram, which I really like.
Now, I am not a machine learning expert, but I have a hypothesis that has a ton of face validity when I look in the mirror. My hypothesis is this:
Machine learning will return meaningful results to the extent that the data it uses is representative of the domain of interest.
A simple thought experiment will demonstrate my point. If a learning machine is given data about professional baseball in the United States from 1890 to 2000, it would learn all kinds of things, including the benefits of pulling the ball as a batter. Pulling the ball occurs when a right-handed batter hits the ball to left field or a left-handed batter hits the ball to right field. In the long history of baseball, many hitters benefited by trying to pull the ball because it produces a more natural swing and one that generates more power. Starting in the 2000s, with the advent of advanced analytics that show where each player is likely to hit the ball, a maneuver called “the shift” has been used more and more, and pulling the ball consistently has become a disadvantage. In the shift, players in the field migrate to positions where the batter is most likely to hit the ball, thus negating the power benefits of pulling the ball. Our learning machine would not know about the decreased benefits of pulling the ball because it would never have seen that data (the data from 2000 to now).
Machine Learning about Learning
I raise this point because of the creeping danger in the world of learning and education. My concern is relevant to all domains where it is difficult to collect data on the most meaningful factors and outcomes, but where it is easy to collect data on less meaningful factors and outcomes. In such cases, our learning machines will only have access to the data that is easy to collect and will not have access to the data that is difficult or impossible to collect. People using machine learning on inadequate data sets will certainly find some interesting relationships in the data, but they will have no way of knowing what they’re missing. The worst part is that they’ll report out some fanciful finding, we’ll all jump up and down in excitement and then make bad decisions based on the bad learning caused by the incomplete data.
In the learning field—where trainers, instructional designers, elearning developers, and teachers reside—we have learned a great deal about research-based methods of improving learning results, but we don’t know everything. And, many of the factors which we know work are not tracked in most big data sets. Do we track the spacing effect, the number of concepts repeated with attention-grabbing variation, the alignment between context cues present in learning materials compared with the cues that will be present in our learners’ future performance situations? Ha! Our large data sets certainly miss many of these causal factors.
Our large data sets also fail to capture the most important outcomes metrics. Indeed, as I have been regularly recounting for years now, typical learning measurements are often biased by measuring immediately at the end of learning (before memories fade), by measuring in the learning context (where contextual cues offer inauthentic hints or subconscious triggering of recall targets), and by measuring with tests of low-level knowledge (compared to more relevant skill-focused decision-making or task performances). We also overwhelmingly rely on learner feedback surveys, both in workplace learning and in higher education. Learner surveys—at least traditional ones—have been found virtually uncorrelated with learning results. To use these meaningless metrics as a primary dependent variable (or just a variable) in a machine-learning data set is complete malpractice.
So if our machine learning data sets have a poor handle on both the inputs and outputs to learning, how can we see machine learning interpretations of learning data as anything but a shiny new alchemy?
Measurement Illuminates Some Things But Leaves Others Hidden
In my learning-evaluation workshops, I often show this image.
The theme expressed in the picture is relevant to all types of evaluation, but it is especially relevant for machine learning.
When we review our smile-sheet data, we should not fool ourselves into thinking that we have learned the truth about the success of our learning. When we see a beautiful data-visualized dashboard, we should not deceive ourselves and our organizations that what we see is all there is to see.
So it is with machine learning, especially in domains where the data is not all the data, where the data flawed, and where the boundaries on the full population of domain data are not known.
With Apologies to Karen Hao
I don’t know Karen, but I do love her diagram. It’s clear and makes some very cogent points—as does her accompanying article.
Here is her diagram, which you can see in the original at this URL.
Like measurement itself, I think the diagram illuminates some aspects of machine learning but fails to illuminate the danger of incomplete or unrepresentative data sets. So, I made a modification in the flow chart.
And yes, that seven-letter provocation is a new machine-learning term that arises from the data as I see it.
Corrective Feedback Welcome
As I said to start this invective, my hypothesis about machine learning and data is just that—a semi-educated hypothesis that deserves a review from people more knowledgeable than me about machine learning. So, what do you think machine learning gurus?
Karen Hao Responds
I’m so delighted! One day after I posted this, Karen Hao responded:
Geese are everywhere these days, crapping all over everything. Where we might have nourishment, we get poop on our shoes.
Big data is everywhere these days…
Even flocking into the learning field.
For big-data practitioners to NOT crap up the learning field, they’ll need to find good sources of data (good luck with that!), use intelligence about learning to know what it means (blind arrogance will prevent this, at least at first), and then find where the data is actually useful in practice (will there be patience and practice or just shiny new objects for sale?).
Beware of the wild goose chase! It’s already here.
Is anyone else getting completely annoyed watching someone’s hand draw and write on videos and elearning?
OMG! It’s beginning to drive me nuts! What the hell is wrong with us?
Here’s the thing. When this was new, it was engaging. Now it’s cliche! Now most people are habituated to it. What we’re doing now is taking one of our tools and completely overusing it.
Let’s be smarter.
Somebody sent me a link to a YouTube video today — a video created to explain to laypeople what instructional design is. Most of it was reasonable, until it gave the following example, narrated as follows:
“… and testing is created to clear up confusion and make sure learners got it right.”
Something is obviously wrong here — something an instructional designer ought to know. What is it?
Scroll down for the answer…
Before you scroll down, come up with your own answer…
The test question is devoid of real-world context. Instead of asking a text-based question, we could provide an image and ask them to point to the access panel.
Better yet, we could have them work on a simulated real-world task and follow steps that would enable them to complete the simulated task only if they used the access panel as part of their task completion.
Better yet, we could have them work on an actual real-world task… et cetera…
Better yet, we might first ask ourselves whether anybody really needs to “LEARN” where the access panel is — or would they just find it on their own without being trained or tested on it?
Better yet, we might first ask ourselves whether we really need a course in the first place. Maybe we’d be better off to create a performance-support tool that would take them through troubleshooting steps — with zero or very little training required.
Better yet, we might first ask ourselves whether we could design our equipment so that technicians don’t need training or performance support.
Or we could ask ourselves existential questions about the meaning and potency of instructional design, about whether a career devoted to helping people learn work skills is worthy to be our life’s work…
Or we could just get back to work and crank out that test…
I'm a bad blogger. I don't analyze my site traffic. I don't drive my readers down a purchase funnel. I don't sell advertising on my blog site. I'm a bad, bad blogger.
Two months ago, I set up Google Analytics to capture my blog's traffic. Holy heck batman! I found out something amazing, and I'm not really sure how to think about it.
Over April and May 2014, my most popular blog post–that is, the one most visited–was a blog post I published in 2006. How popular was this 2006 blog post? It accounted for 50% of all my blog traffic! Fifty freakin' percent! And April and May have been relatively busy blog posting months for me, so it wasn't like I wasn't creating new traffic.
What blog post was the envy of all the others?
It was this one, on one of the biggest myths in the learning field.
I guess this makes sense, (1) it's important, (2) the myth keeps resurfacing, and (3) by now the link has been posted in hundreds of places.
If I die today, at least I have made a bit of a difference, just in this blog post.
I'm a bad, bad blogger. <<<<<WINK>>>>>
My friend, Jonathan Kaye, elearning-guy-extraordinaire, has posted the first parody of the Serious eLearning Manifesto–and I am proud to share it with you. Read his dLearning Manifesto principles (in this blog post) to get a LOL experience.
The Work-Learning Research website is ranked as follows:
- #4 on Google
- #4 on Bing
- #7 on Yahoo
When searching for "learning research."
Interestingly, we hardly ever get paid to do research. Mostly we get paid to use research wisdom to make practical recommendations, for example in the following areas:
- Learning Design
- Learning Evaluation
- Organizational Change
- Leadership Development
- Improving the Learning Department's Results
- Making Fundamental Change in Your Organization's Learning Practices
Research for me is a labor of love, and also, a way to help clients cut through opinions and get to practical truths that will actually make a difference.
But still, we are happy that the world-according-to-search-engines (WOTSE) values the research perspective we offer.
And here's a secret. We don't pay any search-optimizer companies, nor do we do anything intentional to raise our search profile (who has time or money for that?). Must be our dance moves or something…
Check this out (I can’t say more):