I’ve just completed a new research-to-practice white paper. As far as I can tell, it is the first work on learning measurement (assessment and evaluation) that actually takes human learning into consideration. I’d like to thank Questionmark for agreeing to support this work.


Words from the paper’s introduction:

In writing this report on using fundamental learning research to inform assessment  design, I am combining two of my passions—learning and the measurement of learning. As an experienced learner and learning designer, I have come to the belief that those of us responsible for designing, developing, and delivering learning interventions are often left in the dark about our own successes and failures. The measurement techniques we use simply do not provide us with valid feedback about our own performances.

The traditional model of assessment utilizes end-of-learning assessments provided to learners in the context in which they learned. This model is seriously flawed, especially in failing to give us an idea of how well our learning interventions are doing in preparing our learners to retrieve information in future situations—the ultimate goal of training and education. By failing to measure our performance in this regard, we are missing opportunities to provide ourselves with valid feedback. We are also likely failing our institutions and our learners because we are not able to create a practice of continuous improvement to maximize our learning outcomes.

This report is designed to help you improve your assessments in this regard. I certainly won’t claim to have all the answers, nor do I think it is easy to create the perfect assessment, but I do believe very strongly that all of us can improve our assessments substantially, and by so doing improve the practice of education and training.

Click here to access the report.


March 2, 2007

Hello. This is Will Thalheimer.

Today I announce the end of Work-Learning Research as my full-time employment. After nine years of engaging my personal quixotic quest, the pragmatist inside me has ripped control from out of the crusty iron hands of the eternal optimist. To put it simply, the financial return is too meager and unreliable to sustain a sensible family.

It’s been a great journey—from the warm euphoric beginning in the summer 1998 to the harsh winter days of 2007—I’ll never regret taking that first step.

The good news is that I am not yet dead, nor are my ambitions diminished. As of today, Work-Learning Research has become a beloved hobby. I’ll work on it in the dark quiet of night, while my daughter naps, while vacuuming the dirt and debris from the floor, while listening from a distance to the voices of the radio elites. For now, I intend to keep my blog (www.willatworklearning.com) and the Work-Learning Research website (www.work-learning.com) up and running, contributing new information when I can.

My Job Search

My first priority today is to find a great job. If you know of my work and you’re willing to advocate on my behalf, I’d like your help. If you know of an organization that could leverage my talents, let me know or let them know of me. Send me an email.

Here’s what I’m looking for: A senior, high-profile position (in New England, New York, or virtually accessible) with the authority, budget, and thoughtful colleagues to think afresh about learning-and-performance challenges. I’d love a place where I can be entrepreneurial, whether in a small or large organization, building value by combining my multiple backgrounds as a learning expert, researcher, instructional designer, business strategist, manager, marketer, speaker, writer, and visionary. I want to build something that works.

In the next few months as I look for a job, I’ll be finishing research that was previously commissioned. I’ll be speaking at conferences, leading workshops, and completing my commitments to my clients. I’ll also be working to expand on a paper I’ve just written on how to maximize learning with audience response systems—and to build it into a book (a book with great potential to improve classroom learning at all levels by helping instructors use questioning strategies to support higher-level learning).

If you know me well, you’ll know that to put Work-Learning Research on the back burner has not been easy. But it’s done now, and I’m ready to engage the next challenge.

You can access my resume by clicking below.

Download Will_Thalheimer_Resume_2007.pdf

I’m going to be a discussant on a panel on m-Learning on December 5th, 2006 in Chicago in an event sponsored by the Chicagoland Learning Leaders and Walgreens.

These events are designed for Learning Executives near Chicago and the midwest, but if that doesn’t exactly fit you, try to crash the party. These are intimate events I hear. Only 20 registrants allowed so that the discussion remains meaningful and focused.

To register for the event, click here.

To read some FAQ’s, click here.

Come hear me rant!!

I’ve been invited to deliver one of six rants on the learning/e-learning field at the MIT Enterprise Forum in Chicago, co-sponsored by The Chicago Learning Council.

When: October 17th, 2006. 5:00PM to 8:30PM

To learn more about the event or to sign up: Click Here.

Title of my rant: The big lie at the heart of our fogbound wannabe profession.

At the eLearningGuild’s upcoming conference in San Francisco, DevLearn06, I’ll be leading a workshop entitled, "Designing Learning Into E-Learning: What the Research Says…"

The workshop focuses on fundamental learning principles and specifically about how to utilize these in e-learning to maximize learning results.

Date of Workshop: October 10th, 2006.

See more about Workshop by clicking here. It’s Workshop P2.

See more about the eLearningGuild conference by clicking here.

Let me tell you about a new program I’m offering. It’s called the Researcher in Residence Program.

It’s a 3-4 day program designed for companies or departments who have a bunch of instructional designers, developers, and/or trainers.

I come to visit. It’s part learning opportunity, part consulting engagement. Although the schedule can be customized, it might typically moves forward as follows:

Day 1: Workshop on the world’s preeminent learning research, led by me. Interactive. Challenging. Thought-provoking.

Day 2: Review of client programs and projects, with feedback from me. With sharing. With learning for me and for client.

Day 3: Special topics. Overall analysis of client strengths and blind spots. Client-led analysis and review. Review of key concepts. Action planning. Setting challenge agenda for client.

Day 4: Client leadership discussion. After-engagement planning.

Companies can choose to have me conduct learning audits to augment the experience. Such learning audits enable me to do in-depth reviews of the client’s learning programs before I arrive. This obviously costs more, but it’s helpful because it enables the discussions to go deeper.

I recently finished two Researcher in Residence Programs at two wonderful custom e-learning companies, Allen Interactions and Type A Learning Agency. If all e-learning was developed by folks like these, effectiveness would double or triple.

Although these were the first run-throughs for the Researcher in Residence Programs, they seemed to go very well. The Type A folks were so thrilled they posted a press release about the program and Michael Allen (Allen’s CEO and industry-leading e-learning guru) called the program "a very beneficial exercise" in his monthly ezine column.

If you’d like to explore the possibility of bringing the Researcher-in-Residence Program to your organization, email me or call me directly at 617-718-0067.

January 2006 was a very busy month for me. I spent it on three major initiatives. First, I got down to the business of writing a book on learning and instructional design. Second, I got LearningAudit.com up and running, a Work-Learning Research service providing benchmarking and assessment services. Third, I spent half the month at client sites, involved with two of the most innovative and effective e-learning custom-development houses in the world. I learned a ton.

Celebrity News. Start.

One of my clients told me this month that they hired a firm to help them garner insights about the training/learning/e-learning field, AND that the firm that they hired—obviously a world-class firm—analyzed the evidence and discovered through exhaustive data-analytic techniques that I, cuddly curmudgeon Will Thalheimer, was one of the top 5 analysts in the learning field.

Upon hearing the news, I buzzed with delight. "I made the top 5 mom!" But then I realized that I had no idea what an analyst does. I still have no idea how metrics could be made valid to rate the top analysts. But I guess being in the top 5 is pretty good. Right?

But why not #1? And what if this is like the Oscars and there were five nominees in each category? Maybe I’m just one of the four also-rans. And damn, now I’m going to have to rent a Tux.

And the real kicker is this. I keep trying to tell everyone that I don’t have time to be an industry analyst. Other people care a lot more about who gobbles up whom, who’s going where to work, and who’s sleeping with whom. I have too much real work to do already, providing my clients with insights and keeping up to date with the research on learning. I don’t want to be an analyst! Take me off the list, damn it!!

But here is a little secret I learned about the industry this month (just so I don’t lose my top-five ranking). Some of the best e-learning developers are becoming a bit frustrated with their clients because their clients won’t let them build the most effective learning designs.

So, if you’re out there buying e-learning, I have two pieces of advice. First, hire Work-Learning Research, Inc. (and the top-five guy) to help you find the best provider for your particular needs (shameless plug), and second, once you’ve hired one of these excellent providers, get the hell out of their way!!

I know it’s hard to trust anyone these days living in a society driven by a million little lies, but there are e-learning vendors out there who really know what they’re doing.

Different topic. Start.

Although I am a great proponent of research-based learning design, I have believed for quite some time that research has to be interpreted intelligently to be useful. From my work as a consultant, I have also learned that  practitioners regular create innovative and effective learning, sometimes based on research insights, but often not. That’s okay with me. If it’s effective, it ought to be celebrated.

The book I’m writing will provide a research-based perspective, but I want it to be more. I want it to provide examples of the best learning designs out there, whether they are intentionally research-based or not. If you’d like to nominate folks who are doing world-class work, please comment here or send me a private email. I’ve got some pretty good ideas about who is doing the best work, but I’m sure I’ll miss somebody if you don’t enlighten me.

In your estimation, which companies are building the most effective and innovative learning interventions? Who should I talk with? Who can I learn from?

Final Topic. Groundhog Day. February 2nd. Today.

Raised in Pennsylvania, I used to be partial to Punxsutawney Phil, the Groundhog reputedly able to predict whether winter will be long or short. If he sees his shadow, we get six more weeks of winter. According to Stormfax Weather Almanac, Phil has seen his shadow 96 times since 1887. He’s seen NO shadow 14 times. And 9 times no report has been forthcoming. The bad news is that the groundhog has only been correct 39% of the time.

This reminds me of the training industry. We see sunny weather for almost all our training interventions, but the reality is that we actually fail more than we will admit. We use crude and inappropriate methods for predicting on-the-job training transfer, just like those deluded souls in Pennsylvania who believe in their large sleepy rodent. We do things because of the commercial benefits, not because of effectiveness, accuracy, or appropriateness. Clymer H. Freas, city editor of the Punxsutawney Spirit newspaper, invented Groundhog Day in 1887 and the town of Punxsutawney has kept it going as a revenue-generating tourist attraction ever since. The predominant messages in our industry are created by or for our vendor elites.

Weird coincidence. Oprah Winfrey hosted the groundhog on her television show in 1995. This should have been an omen for Oprah watchers—the fraud of the groundhog perpetrated by Clymer Freas is very similar to the fraud perpetrated by the author of the Oprah bookclub selection, "A Million Little Pieces." The author was James Frey, who lied repeated in his book, but sold it as a non-fiction memoir. Did you notice the synchronicity? Freas and Frey. Probably pronounced the same too. Oprah originally defended Frey, saying the truth didn’t matter if the outcome was good. Later she pulled a stunning live-TV reversal, ripping into the author and his publisher, and apologizing for not previously upholding the value of truth. "Truth matters," Oprah said simply.

Is there a lesson here for us?

Do you smell a rat in the training industry? Are we destined to continue repeating ourselves like Bill Murray in the movie? Or does truth matter in our work?

If we really want to learn from our work, we need to measure our training outcomes.

Happy Groundhog Day!

Work-Learning Research in 2006 will undertake some ambitious initiatives.

  1. I’m going to come out with a book on Instructional Design. My goal is to make it the most important book on instructional design since Gagne. Tall order, I know. But if I’m going to go to all the trouble to write a book, why not make it a damn good one.
  2. LearningAudit.com has been launched, Work-Learning Research’s research benchmarking and evaluation service to help organizations and instructional designers learn from their efforts. If we—who tout ourselves as learning experts—can’t learn and improve our own practices, we’re not only being hypocritical, but we’re not creating the most effective learning designs.
  3. I’ve also launched a new program to help organizations build their knowledge and competency around research-based instructional design. The idea behind this program is simple. You invite me into your world, and we learn together over several days. The intimate setting helps you and your team build a deep understanding of human learning—based on the world’s best learning research, of course. Call me to find out more.
  4. Work-Learning Research will continue to release more research reports and white papers on research-based practice.

The theme for 2006 is to focus on the major leverage points that are available to improve learning design and development. When I asked myself, "How can Work-Learning Research help to facilitate more effective learning, my answer was this:

  • Continue to bridge research and practice, especially by helping to translate research into practical recommendations.
  • Provide the field with a way to learn from their efforts. Because the feedback loop in our field is so impoverished, we learn and improve much too slowly. LearningAudit.com was a response to this problem.
  • Highlight the good work of instructional designers who are using research-based practices (whether they may know it or not). By shining a spotlight on good design, we help everyone learn.

Good luck to all in 2006. May we learn. May we help others learn.

It’s time for a change.

The Work-Learning Research Newsletter is becoming the Work-Learning Journal. In the short term, the content will consist of the same kind of pithy research-based commentary I’ve always aimed to deliver. In the long term, you’ll probably see some additions as well, but those plans will have to remain a secret for now.

The big change for the moment is the delivery mechanism. I’ll still send you an email reminder, but the content will now be contained in a blog-like format. This has several benefits:

  1. You can post your comments, so we can learn from each other.
  2. The content will remain where you can access it—online all the time.
  3. Others can link to it and share our intelligence far and wide.
  4. You can subscribe using your aggregator. If you don’t know what one is, you will within the next year or so.
  5. We can all learn about Web 2.0 and what it means for learning.

In addition to publishing a monthly newsletter, I’m going to write a blog as well. You can check out the beginnings of that effort at www.willatworklearning.com.

The Newsletter is dead. Long live the Work-Learning Journal.

By the way, if you’re not yet on one or more of our email-reminder mailing lists, you can subscribe at this location.