I had the great pleasure of being invited to provide a keynote in Leuven, Belgium at the VOV Congress. There were two highlights in my day, my keynote and being interviewed on HRD TV by Sandra De Milliano.
Audience members asked me questions and I did my best to answer them. We had questions about elearning, microlearning, PowerPoint, learning styles, rewards for learners, the spacing effect, remembering and application, and more.
Take a look…
Is elearning effective? As effective as classroom instruction — more or less effective? What about blended learning — when elearning and classroom learning are combined?
These critical questions have now been answered and are available in the research report, Does eLearning Work? What the Scientific Research Says!
In this research review, I looked at meta-analyses and individual research studies, and was able to derive clear conclusions. The report is available for free, it includes an executive summary, and research jargon is kept to a minimum.
Click here to download the report…
Note that the August 10, 2017 version of this report incorrectly cited the Rowland (2014) study in a footnote and omitted it from the list of research citations. These issues were fixed on August 11, 2017. Special thanks to Elizabeth Dalton who notified me of the issues.
Is my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, award worthy?
I think so, buy I'm hugely biased! SMILE.
Here's what I wrote today on an award-submission application:
Performance-Focused Smile Sheets: A Radical Rethinking of Dangerous Art Form is a book, published in February 2016, written by Will Thalheimer, PhD, President of Work-Learning Research, Inc.
The book reviews research on smile sheets (learner feedback forms), demonstrates the limitations of traditional smile sheets, and provides a completely new formulation on how to design and deploy smile sheets.
The ideas in the book — and the example questions provided — help learning professionals focus on "learning effectiveness" in supporting post-learning performance. Where traditional smile sheets focus on learner satisfaction and the credibility of training, Performance-Focused Smile Sheets can also focus on science-of-learning factors that matter. Smile sheets can be transformed by focusing on learner comprehension, factors that influence long-term remembering, learner motivation to apply what they've learned, and after-learning supports for learning transfer and application of learning to real-world job tasks.
Smile sheets can also be transformed by looking beyond Likert-like responses and numerical averages that dumb-down our metrics and lead to bias and paralysis. We can go beyond meaningless averages ("My course is a 4.1!") and provide substantive information to ourselves and our stakeholders.
The book reviews research that shows that so-called "learner-centric" formulations are filled with dangers — as research shows that learners don't always know how they learn best. Smile-sheet questions must support learners in making smile-sheet decisions, not introduce biases that warp the data.
For decades our industry has been mired in the dishonest and disempowering practice of traditional smile sheets. Thankfully, a new approach is available to us.
Sure! I'd love to see my work honored. More importantly, I'd love to see the ideas from my book applied wisely, improved, and adopted for training evaluation, student evaluations, conference evaluations, etc.
You can help by sharing, by piloting, by persuading, by critiquing and improving! That will be my greatest award!
OMG! The best deal ever for a full-day workshop on how to radically improve your smile-sheet designs! Sponsored by the Hampton Roads Chapter of ISPI. Free book and subscription-learning thread too!
Friday, June 10, 2016
Reed Integration
7007 Harbour View Blvd #117
Suffolk, VA
Performance Objectives:
By completing this workshop and the after-course subscription-learning thread, you will know how to:
Avoid the three most troublesome biases in measuring learning.
Persuade your stakeholders to improve your organization’s smile sheets.
Create more effective smile sheet questions.
Create evaluation standards for each question to avoid bias.
Envision learning measurement as a bulwark for improved learning design.
Recommended Audience:
The content of this workshop will be suitable to those who have at least some background and experience in the training field. It will be especially valuable to those who are responsible for learning evaluation or who manage the learning function.
Format:
This is a full-day workshop. Participants are encouraged to bring laptops if they prefer to use a computer to write their questions.
Bonus Take-Away:
Each Participant will receive a copy of Dr. Thalheimer’s Book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form.
Send me your tired, your salaried, your harried Chief Learning Officers (and other Talent Development Executives) so that I can collect data on the state of our industry — focusing on a practical science-of-learning perspective.
Areas Covered in this Research Effort:
Organizations who contribute will be entitled to the full report. An executive summary will be available for free to others.
Send me names and contact information to info@work-learning.com.
Now that my book publishing responsibilities are out of the way, I'm ready to begin a research effort designed to find out more ways to improve the effectiveness of our smile sheets.
If you're organization would like to participate in this effort — and be among the first to see the results of the research — check out the full announcement on my book's website: http://smilesheets.com/2016/05/18/3705/.
Next week, I'm headed to Denver, Colorado for ATD's Annual Conference for 2016. The largest conference in the workplace learning and development field, it brings together all kinds of folks for a wondrous bacchanal of learning.
I'll be talking about smile sheets (learner response forms) on Tuesday May 24th, 4:30 – 5:30 pm
TU420 – Utilizing Radically Improved Smile Sheets to Improve Learning Results at Room: 708/710.
I'll also be joining a "Science of Learning" panel on Monday May 23rd, 1:00 – 2:00 pm
M1CE – Community Express: Science of Learning Fast Track
along with Sebastian Bailey, Justin Brusino, Paul Zak, Patti Shank at Room: Mile High 1c.
If you're there at ATD's ICE — and you want to meet to discuss your organization's needs for a practical research-based approach to learning or evaluation design — send me a note at info@work-learning.com.
Wow!!
I almost can't believe it. Finally, after 17 years of research and writing, I'm finally a published author.
Today is the day!
It's kind of funny really.
When I began this journey back in 1997 I had a well-paying job running a leadership-development product line, building multimedia simulations, and managing and working with a bunch of great folks.
As I looked around the training-and-development field — that's what we called it back then — I saw that we jumped from one fad to another and held on sanctimoniously to learning methods that didn't work that well. I concluded that what was needed was someone to play a role in bridging the gap between the research side and the practice side.
I had a very naive idea about how I might help. I thought the field needed a book that would specify the fundamental learning factors that should be baked into every learning design. I thought I could write such a book in two or three years, that I'd get it published, that consulting gigs would roll in, that I'd make good money, that I'd make a difference.
Hah! The blind optimism of youth and entrepreneurship!
I've now written over 700 pages on THAT book…without an end in sight.
How The Smile-Sheet Book Got its Start
Back in 2007, as I was mucking around in the learning research, I began to see biases in how we were measuring learning. I noticed, for instance, that we always measured at the top of the learning curve, before the forgetting curve had even begun. We measured with trivial multiple-choice questions on definitions and terminology — when these clearly had very little relevance for on-the-job performance. I wrote a research-to-practice report on these learning measurement biases and suddenly I was getting invited to give keynotes…
In my BIG book, I wrote hundreds of paragraphs on learning measurement. I talked about our learning-measurement blind spots to clients, at conferences, and on my blog.
Where feedback is the lifeblood of improvement, we as learning professionals were getting very little good feedback. We were practicing in the dark.
I'd also come to ruminate on the meta-analytic research findings that showed that traditional smile sheets were virtually uncorrelated with learning results. If smile sheets were feeding us bad information, maybe we should just stop using them.
It was about three or four years ago that I saw a big client get terrible advice about their smile sheets from a well-known learning-measurement vendor. And, of course, because the vendor had an industry-wide reputation, the client almost couldn't help buying into their poor smile-sheet designs.
I concluded that smile-sheets were NOT going away. They were too entrenched and there were some good reasons to use them.
I also concluded that smile sheets could be designed to be more effective, more aligned with the research on learning, and designed to better support learners in making smile-sheet decisions.
I decided to write a shorter book than the aforementioned BIG book. That was about 2.5 years ago.
I wrote a draft of the book and I knew I had something. I got feedback from learning-measurement luminaries like Rob Brinkerhoff, Jack Phillips, and Bill Coscarelli. I got feedback from learning gurus Julie Dirksen, Clark Quinn, and Adam Neaman. I made major improvement based on the feedback from these wonderful folks. The book then went through several rounds of top-tier editing, making it a much better read.
As the publication process unfolded, I realized that I didn't have enough money on hand to fund the printing of the book. Kickstarter and 227 people raised their hands to help, reserving over 300 books in return for their generous Kickstarter contributions. I will be forever indepted to them.
Others reached out to help as well, from people on my newsletter list, to my beloved clients, to folks in trade organizations and publications, to people I've met through the years, to people I haven't met, to followers on Twitter, to the industry luminaries who agreed to write testimonials after getting advanced drafts of the book, to family members, to friends.
Today, all the hard work, all the research, all the client work, all the love and support comes together for me in gratitude.
Thank you!
= Will Thalheimer
P.S. To learn more about the book, or buy it: SmileSheets.com
I've been running Work-Learning Research for 17 years. Over that time, my websites have morphed many times, getting easier to use, getting more powerful, getting better looking.
I've always built my own websites, even going so far as to program some eCommerce functionality way back in 2000 or 2001. This is bootstrapping. Sure, I wish I could afford a brilliant web designer, but alas…I muddle through, often slowly.
Now, motivated by my impending book release, I decided to take my designs to another level, at least for my book website.
Here is the result:
I'm kind of psyched about the new design, but I may have some blind spots. Please send your feedback to me at info@work-learning.com or leave a comment here.
Thanks!
= Will Thalheimer
I'm delighted to be attending the eLearningGuild's DevLearn conference in Las Vegas coming up in late September and early October.
The eLearning Guild always puts on a great conference and I'm excited to learn the latest and greatest on elearning and mobile learning. This year, I'm going to be keeping my eyes out for examples of micro learning and subscription learning — as I see more an more interest in smaller learning nuggets.
Also, I'll be speaking on "Measuring eLearning to Create Cycles of Improvement." In my session, I'll share research-based findings and their implications for elearning measurement designs.
Come join me 10:45 AM – 11:45 AM Thursday, October 1.
You must be logged in to post a comment.