FREE Webinosh on Instructional Objectives

This Friday January 9th, 2009, I will talk about Instructional Objectives, and some research, thereof.

Instructional Objectives:

  • Do they produce learning results?
  • Are they all the same?
  • Do we have to use them?
  • Do prequestions work just as well?
  • How specifically do they have to be worded?
  • Can I use the word “Understand”? Answer: In some of them, but not others.
  • Hey Will, do you have a new taxonomy for us?

Click to Sign Up:


I’ve been busy again thinking about the nexus between LEARNING and LEARNING MEASUREMENT.

You can peruse some of my previous thoughts on learning measurement by clicking here.

Here is a brand new article that I wrote for the eLearning Guild on how to evaluate Learning 2.0 stuff. Note: Learning 2.0 is defined (by the eLearning Guild) as: The idea of learning through digital connections and peer collaboration, enhanced by technologies driving Web 2.0. Users/Learners are empowered
to search, create, and collaborate, in order to fulfill intrinsic needs to learn new information.
Evaluating Learning 2.0 differs from evaluating traditional Learning 1.0 training for many reasons, one of which is that Learning 2.0 enables (encourages) learners to create their own content.

Steve Wexler, Director of Research and Emerging Technologies at the eLearning Guild, and I are leading a Webinar on Thursday September 4th on the current state of eLearning Measurement. We’ve got some new data that we’re hot to share.

Finally, Roy Pollock, one of the authors of the classic book, Six Disciplines of Breakthrough Learning, and I are leading a one-day symposium on measuring learning at the eLearning Guild’s DevLearn 2008 conference in November. It’s a great chance to go to one of the best eLearning conferences around while working with Roy and I in a fairly intimate workshop, wrangling with the newest thinking in how to measure learning. Choose Symposium S-4. Note that it may not show Roy’s information there yet–the Guild is still working on the webpage–but let me assure you that Roy and I are equal partners in this one.

Questioning Strategies for
Audience Response Systems:
How to Use Questions to Maximize
Learning, Engagement, and Satisfaction

by Dr. Will Thalheimer


The buzz in the learning industry is focused on e-learning, m-learning, wikis, and blogs; but one of the most powerful learning technologies is being overlooked, probably because it’s an in-the-classroom technology—audience response systems. In this research-to-practice white paper I offer a blueprint for how to use audience response systems to maximize higher-order learning in the classroom and beyond.

What One Reader Wrote to Me

Dr. Thalheimer,

Just wanted to drop you a little note this morning to express my gratitude for your paper "Questioning Strategies for Audience Response Systems: How to Use Questions to Maximize Learning, Engagement, and Satisfaction."

A friend recommended that I read it to prepare for a Higher Order Questioning staff development class that she and I are teaching together (in conjunction with some CPS [audience response] training we’re offering). To tell you the truth, I really wasn’t looking forward to reading it because I expected it to be dry and full of boring I’m-trying-to-sound-snobbily-intellectual writing, but I LOVED it. 🙂

I enjoyed your approachable style and dry sense of humor so much I read all the way through (including the endnotes!) and had many a good laugh along the way. In addition to being a blast to read, the paper challenged and inspired me to find new ways to push my questioning skills to a higher level for the next school year.

Thanks again, for the inspiration and for the great read. I’ll be checking out your website later today and hope to find that equally enjoyable.


Liz Walhof

Spanish Teacher

From the Paper’s Introduction

"Audience response systems have enormous potential for transforming lectures from dry recitals into rich jam sessions of deeply resonant learning. The technology is widely available, but the key to success is not in the technology; it’s in the instruction. To maximize meaningful learning, instructors must become adept in using questioning and discussion techniques. Unfortunately, some of us may come to believe that we can simply sprinkle our lectures with a few multiple-choice questions. This approach is emphatically inadequate, and is simply not worthy of our profession.

This report provides a near-exhaustive list of questioning strategies, and a comprehensive guide on using questions to facilitate classroom learning. No other resource exists that is research-based and comprehensive, while also being practical and useful. It has been designed specifically to provide practical guidance for trainers, teachers, and professors so that their learners—whether they are eight, forty-eight, or eighty years old—can experience deep and meaningful learning."

Special thanks to eInstruction for agreeing to license the paper for distribution to their clients. Such underwriting helps move the audience-response field forward and demonstrates an enlightened commitment to effective learning in classrooms of all types throughout the world. Other underwriting opportunities are available for research on audience-response learning. Contact Dr. Thalheimer with inquiries.

Additional Information

  • Number of Pages: 124
  • Number of Research Citations: 54
  • Publication Date: March 2007
  • Available to you Immediately as downloadable Electronic file (PDF).
  • Purchasing utilizes industry-leading security.
  • 100% Satisfaction Guaranteed.
  • Cost: $40.00 (US)

Click here to purchase…

I’ve just completed a new research-to-practice white paper. As far as I can tell, it is the first work on learning measurement (assessment and evaluation) that actually takes human learning into consideration. I’d like to thank Questionmark for agreeing to support this work.


Words from the paper’s introduction:

In writing this report on using fundamental learning research to inform assessment  design, I am combining two of my passions—learning and the measurement of learning. As an experienced learner and learning designer, I have come to the belief that those of us responsible for designing, developing, and delivering learning interventions are often left in the dark about our own successes and failures. The measurement techniques we use simply do not provide us with valid feedback about our own performances.

The traditional model of assessment utilizes end-of-learning assessments provided to learners in the context in which they learned. This model is seriously flawed, especially in failing to give us an idea of how well our learning interventions are doing in preparing our learners to retrieve information in future situations—the ultimate goal of training and education. By failing to measure our performance in this regard, we are missing opportunities to provide ourselves with valid feedback. We are also likely failing our institutions and our learners because we are not able to create a practice of continuous improvement to maximize our learning outcomes.

This report is designed to help you improve your assessments in this regard. I certainly won’t claim to have all the answers, nor do I think it is easy to create the perfect assessment, but I do believe very strongly that all of us can improve our assessments substantially, and by so doing improve the practice of education and training.

Click here to access the report.


March 2, 2007

Hello. This is Will Thalheimer.

Today I announce the end of Work-Learning Research as my full-time employment. After nine years of engaging my personal quixotic quest, the pragmatist inside me has ripped control from out of the crusty iron hands of the eternal optimist. To put it simply, the financial return is too meager and unreliable to sustain a sensible family.

It’s been a great journey—from the warm euphoric beginning in the summer 1998 to the harsh winter days of 2007—I’ll never regret taking that first step.

The good news is that I am not yet dead, nor are my ambitions diminished. As of today, Work-Learning Research has become a beloved hobby. I’ll work on it in the dark quiet of night, while my daughter naps, while vacuuming the dirt and debris from the floor, while listening from a distance to the voices of the radio elites. For now, I intend to keep my blog ( and the Work-Learning Research website ( up and running, contributing new information when I can.

My Job Search

My first priority today is to find a great job. If you know of my work and you’re willing to advocate on my behalf, I’d like your help. If you know of an organization that could leverage my talents, let me know or let them know of me. Send me an email.

Here’s what I’m looking for: A senior, high-profile position (in New England, New York, or virtually accessible) with the authority, budget, and thoughtful colleagues to think afresh about learning-and-performance challenges. I’d love a place where I can be entrepreneurial, whether in a small or large organization, building value by combining my multiple backgrounds as a learning expert, researcher, instructional designer, business strategist, manager, marketer, speaker, writer, and visionary. I want to build something that works.

In the next few months as I look for a job, I’ll be finishing research that was previously commissioned. I’ll be speaking at conferences, leading workshops, and completing my commitments to my clients. I’ll also be working to expand on a paper I’ve just written on how to maximize learning with audience response systems—and to build it into a book (a book with great potential to improve classroom learning at all levels by helping instructors use questioning strategies to support higher-level learning).

If you know me well, you’ll know that to put Work-Learning Research on the back burner has not been easy. But it’s done now, and I’m ready to engage the next challenge.

You can access my resume by clicking below.

Download Will_Thalheimer_Resume_2007.pdf

I’m going to be a discussant on a panel on m-Learning on December 5th, 2006 in Chicago in an event sponsored by the Chicagoland Learning Leaders and Walgreens.

These events are designed for Learning Executives near Chicago and the midwest, but if that doesn’t exactly fit you, try to crash the party. These are intimate events I hear. Only 20 registrants allowed so that the discussion remains meaningful and focused.

To register for the event, click here.

To read some FAQ’s, click here.

Come hear me rant!!

I’ve been invited to deliver one of six rants on the learning/e-learning field at the MIT Enterprise Forum in Chicago, co-sponsored by The Chicago Learning Council.

When: October 17th, 2006. 5:00PM to 8:30PM

To learn more about the event or to sign up: Click Here.

Title of my rant: The big lie at the heart of our fogbound wannabe profession.

At the eLearningGuild’s upcoming conference in San Francisco, DevLearn06, I’ll be leading a workshop entitled, "Designing Learning Into E-Learning: What the Research Says…"

The workshop focuses on fundamental learning principles and specifically about how to utilize these in e-learning to maximize learning results.

Date of Workshop: October 10th, 2006.

See more about Workshop by clicking here. It’s Workshop P2.

See more about the eLearningGuild conference by clicking here.

Let me tell you about a new program I’m offering. It’s called the Researcher in Residence Program.

It’s a 3-4 day program designed for companies or departments who have a bunch of instructional designers, developers, and/or trainers.

I come to visit. It’s part learning opportunity, part consulting engagement. Although the schedule can be customized, it might typically moves forward as follows:

Day 1: Workshop on the world’s preeminent learning research, led by me. Interactive. Challenging. Thought-provoking.

Day 2: Review of client programs and projects, with feedback from me. With sharing. With learning for me and for client.

Day 3: Special topics. Overall analysis of client strengths and blind spots. Client-led analysis and review. Review of key concepts. Action planning. Setting challenge agenda for client.

Day 4: Client leadership discussion. After-engagement planning.

Companies can choose to have me conduct learning audits to augment the experience. Such learning audits enable me to do in-depth reviews of the client’s learning programs before I arrive. This obviously costs more, but it’s helpful because it enables the discussions to go deeper.

I recently finished two Researcher in Residence Programs at two wonderful custom e-learning companies, Allen Interactions and Type A Learning Agency. If all e-learning was developed by folks like these, effectiveness would double or triple.

Although these were the first run-throughs for the Researcher in Residence Programs, they seemed to go very well. The Type A folks were so thrilled they posted a press release about the program and Michael Allen (Allen’s CEO and industry-leading e-learning guru) called the program "a very beneficial exercise" in his monthly ezine column.

If you’d like to explore the possibility of bringing the Researcher-in-Residence Program to your organization, email me or call me directly at 617-718-0067.

January 2006 was a very busy month for me. I spent it on three major initiatives. First, I got down to the business of writing a book on learning and instructional design. Second, I got up and running, a Work-Learning Research service providing benchmarking and assessment services. Third, I spent half the month at client sites, involved with two of the most innovative and effective e-learning custom-development houses in the world. I learned a ton.

Celebrity News. Start.

One of my clients told me this month that they hired a firm to help them garner insights about the training/learning/e-learning field, AND that the firm that they hired—obviously a world-class firm—analyzed the evidence and discovered through exhaustive data-analytic techniques that I, cuddly curmudgeon Will Thalheimer, was one of the top 5 analysts in the learning field.

Upon hearing the news, I buzzed with delight. "I made the top 5 mom!" But then I realized that I had no idea what an analyst does. I still have no idea how metrics could be made valid to rate the top analysts. But I guess being in the top 5 is pretty good. Right?

But why not #1? And what if this is like the Oscars and there were five nominees in each category? Maybe I’m just one of the four also-rans. And damn, now I’m going to have to rent a Tux.

And the real kicker is this. I keep trying to tell everyone that I don’t have time to be an industry analyst. Other people care a lot more about who gobbles up whom, who’s going where to work, and who’s sleeping with whom. I have too much real work to do already, providing my clients with insights and keeping up to date with the research on learning. I don’t want to be an analyst! Take me off the list, damn it!!

But here is a little secret I learned about the industry this month (just so I don’t lose my top-five ranking). Some of the best e-learning developers are becoming a bit frustrated with their clients because their clients won’t let them build the most effective learning designs.

So, if you’re out there buying e-learning, I have two pieces of advice. First, hire Work-Learning Research, Inc. (and the top-five guy) to help you find the best provider for your particular needs (shameless plug), and second, once you’ve hired one of these excellent providers, get the hell out of their way!!

I know it’s hard to trust anyone these days living in a society driven by a million little lies, but there are e-learning vendors out there who really know what they’re doing.

Different topic. Start.

Although I am a great proponent of research-based learning design, I have believed for quite some time that research has to be interpreted intelligently to be useful. From my work as a consultant, I have also learned that  practitioners regular create innovative and effective learning, sometimes based on research insights, but often not. That’s okay with me. If it’s effective, it ought to be celebrated.

The book I’m writing will provide a research-based perspective, but I want it to be more. I want it to provide examples of the best learning designs out there, whether they are intentionally research-based or not. If you’d like to nominate folks who are doing world-class work, please comment here or send me a private email. I’ve got some pretty good ideas about who is doing the best work, but I’m sure I’ll miss somebody if you don’t enlighten me.

In your estimation, which companies are building the most effective and innovative learning interventions? Who should I talk with? Who can I learn from?

Final Topic. Groundhog Day. February 2nd. Today.

Raised in Pennsylvania, I used to be partial to Punxsutawney Phil, the Groundhog reputedly able to predict whether winter will be long or short. If he sees his shadow, we get six more weeks of winter. According to Stormfax Weather Almanac, Phil has seen his shadow 96 times since 1887. He’s seen NO shadow 14 times. And 9 times no report has been forthcoming. The bad news is that the groundhog has only been correct 39% of the time.

This reminds me of the training industry. We see sunny weather for almost all our training interventions, but the reality is that we actually fail more than we will admit. We use crude and inappropriate methods for predicting on-the-job training transfer, just like those deluded souls in Pennsylvania who believe in their large sleepy rodent. We do things because of the commercial benefits, not because of effectiveness, accuracy, or appropriateness. Clymer H. Freas, city editor of the Punxsutawney Spirit newspaper, invented Groundhog Day in 1887 and the town of Punxsutawney has kept it going as a revenue-generating tourist attraction ever since. The predominant messages in our industry are created by or for our vendor elites.

Weird coincidence. Oprah Winfrey hosted the groundhog on her television show in 1995. This should have been an omen for Oprah watchers—the fraud of the groundhog perpetrated by Clymer Freas is very similar to the fraud perpetrated by the author of the Oprah bookclub selection, "A Million Little Pieces." The author was James Frey, who lied repeated in his book, but sold it as a non-fiction memoir. Did you notice the synchronicity? Freas and Frey. Probably pronounced the same too. Oprah originally defended Frey, saying the truth didn’t matter if the outcome was good. Later she pulled a stunning live-TV reversal, ripping into the author and his publisher, and apologizing for not previously upholding the value of truth. "Truth matters," Oprah said simply.

Is there a lesson here for us?

Do you smell a rat in the training industry? Are we destined to continue repeating ourselves like Bill Murray in the movie? Or does truth matter in our work?

If we really want to learn from our work, we need to measure our training outcomes.

Happy Groundhog Day!