Net Promoter Score — Maybe Fine for Marketing, Stupid for Training

More and more training departments are considering the use of the Net Promoter Score as a question–or the central question–on their smile sheets.

This is one of the stupidest ideas yet for smile sheets, but I understand the impetus–traditional smile sheets provide poor information. In this blog post I am going to try and put a finely-honed dagger through the heart of this idea.

Note that I have written a replacement question for the Net Promoter Score (for getting learner responses).

What is the Net Promoter Score?

Here’s what the folks who wrote the book on the Net Promoter Score say it is:

The Net Promoter Score, or NPS®, is based on the fundamental perspective that every company’s customers can be divided into three categories: Promoters, Passives, and Detractors.

By asking one simple question — How likely is it that you would recommend [your company] to a friend or colleague? — you can track these groups and get a clear measure of your company’s performance through your customers’ eyes. Customers respond on a 0-to-10 point rating scale and are categorized as follows:

  • Promoters (score 9-10) are loyal enthusiasts who will keep buying and refer others, fueling growth.
  • Passives (score 7-8) are satisfied but unenthusiastic customers who are vulnerable to competitive offerings.
  • Detractors (score 0-6) are unhappy customers who can damage your brand and impede growth through negative word-of-mouth.

To calculate your company’s NPS, take the percentage of customers who are Promoters and subtract the percentage who are Detractors.

So, the NPS is about Customer Perceptions, Right?

Yes, its intended purpose is to measure customer loyalty. It was designed as a marketing tool. It was specifically NOT designed to measure training outcomes. Therefore, we might want to be skeptical before using it.

It kind of makes sense for marketing right? Marketing is all about customer perceptions of a given product, brand, or company? Also, there is evidence–yes, actual evidence–that customers are influenced by others in their purchasing decisions. So again, asking about whether someone might recommend a company or product to another person seems like a reasonable thing to ask.

Of course, just because something seems reasonable, doesn’t mean it is. Even for its intended purpose, the Net Promoter Score has a substantial number of critics. See wikipedia for details.

But Why Not for Training?

To measure training with a Net-Promoter approach, we would ask a question like, “How likely is it that you would recommend this training course to a friend or colleague?” 

Some reasonable arguments for why the NPS is stupid as a training metric:

  1. First we should ask, what is the causal pathway that would explain how the Net Promoter Score is a good measure of training effectiveness? We shouldn’t willy-nilly take a construct from another field and apply it to our field without having some “theory-of-causality” that supports its likely effectiveness.Specifically we should ask whether it is reasonable to assume that a learner’s recommendation about a training program tells us SOMETHING important about the effectiveness of that training program? And, for those using the NPS as the central measure of training effectiveness–which sends shivers down my spine–the query than becomes, is it reasonable to assume that a learner’s recommendation about a training program tells us EVERYTHING important about the effectiveness of that training program?Those who would use the Net Promoter Score for training must have one of the following beliefs:
    • Learners know whether or not training has been effective.
    • Learners know whether their friends/colleagues are likely to have the same beliefs about the effectiveness of training as they themselves have.

    The second belief is not worth much, but it is probably what really happens. It is the first belief that is critical, so we should examine that belief in more depth. Are learners likely to be good judges of training effectiveness?

  2. Scientific evidence demonstrates that learners are not very good at judging their own learning. They have been shown to have many difficulties adequately judging how much they know and how much they’ll be able to remember. For example, learners fail to utilize retrieval practice to support long-term remembering, even though we know this is one of the most powerful learning methods (e.g., Karpicke, Butler, & Roediger, 2009). Learners don’t always overcome their incorrect prior knowledge when reading (Kendeou & van den Broek, 2005). Learners often fail to utilize examples in ways that would foster deeper learning (Renkl, 1997). These are just a few examples of many.
  3. Similarly, two meta-analyses on the potency of traditional smile sheets, which tend to measure the same kind of beliefs as NPS measures, have shown almost no correlation between learner responses and actual learning results (Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008).
  4. Similarly, when we assess learning in the training context at the end of learning, several cognitive biases creep in to make learners perform much better than they would perform if they were in a more realistic situation back on the job at a later time (Thalheimer, 2007).
  5. Even if we did somehow prove that NPS was a good measure for training, is there evidence that it is the best measure? Obviously not!
  6. Should it be used as the most important measure. No! As stated in the Science of Training review article from last year: “The researchers [in talking about learning measurement] noted that researchers, authors, and practitioners are increasingly cognizant of the need to adopt a multidimensional perspective on learning [when designing learning measurement approaches].”
    Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012).
  7. Finally, we might ask are there better types of questions to ask on our smile sheets? The answer to that is an emphatic YES! Performance-Focused Smile Sheets provide a whole new approach to smile sheet questions. You can learn more by attending my workshop on how to create and deploy these more powerful questions.

The Bottom Line

The Net Promoter Score was designed to measure customer loyalty and is not relevant for training. Indeed, it is likely to give us dangerously misguided information.

When we design courses solely so that learners like the courses, we create learning that doesn’t stick, that fails to create long-term remembering, that fails to push for on-the-job application, etc.
Seriously, this is one of the stupidest ideas to come along for learning measurement in a long time. Buyers beware!! Please!

My First Workout-Workplace Workshop — Performance-Focused Smile Sheets

Enrollments are open for my first Workout-Workplace Workshop on the World Wide Web (w6).

Fittingly, it will cover the most significant improvement in smile-sheet design in a generation–The Performance-Focused Smile Sheet.

Click for details…

Likert-Like Scales Create Poor Data on Smile Sheets!

I know I'm going completely against most training-industry practice in saying this, but it's the truth. Likert-like scales create poor data on smile sheets.

If you're using questions on your smile sheets with answer choices such as:

  • Strongly Agree
  • Agree
  • Neither Agree Nor Disagree
  • Disagree
  • Strongly Disagree

You're getting data that isn't that useful. Such questions will create
data that your stakeholders–and you too–won't be able to decipher very
well. What does it mean if we average a 4.2 rating? It may sound good,
but it doesn't give your learners, your stakeholders, or your team much
information to decide what to do.

Moreover, let's remember that our learners are making decisions with every smile-sheet question they answer. It's a lot tougher to decide between "Strongly Agree" and "Agree" than between two more-concrete answer choices. 

Sharon Shrock
and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion-Referenced Test Development,
offer the following wisdom: On using Likert-type
Descriptive Scales (of the kind that use response words such as “Agree,”
“Strongly Agree,” etc.):

“…the
resulting scale is deficient in that the [response words] are open to many
interpretations.” (p. 188)

So why do so many surveys use Likert-like scales? Answer: It's easy, it's tradition, and surveys have psychometric advantages often because they are repeating the same concepts in multiple items and they are looking to compare one category to another category of response.

Smile sheets are different. On our smile sheets, we want the learners to be able to make good decisions, and we want to send clear messages about what they have decided. Anything that fuzzes that up, hurts the validity of the smile-sheet data.

Insights for Improving the Workplace Learning-and-Performance Field

,

We, the members of the workplace learning-and-performance field, think about our field from time to time. Of course we do.

Here’s what I wonder. How much have our mental models changed in the last 50 years?

That’s too big a question for me to answer right now, but it does raise an interesting question. Today, as I was reading an scientific article (citation below) on how people have insights, the authors reported that, in their study of naturalistic insights, most sudden insights occured when people looked at new data–after having spent a great deal of time before the new data thinking about the issue.

Here’s the thing: We, in our field, haven’t really created new data sets or methodologies too often. Yes, we have Jack Phillips ROI methodology and Robert Brinkerhoff’s Success Case Method–both of which got many of us to rethink what we’re doing–but these methods have been at the results end of the causal chain from learning to performance to results. Important stuff–there is no doubt–but not enough.

When it comes to getting new data about learning engagement, remembering, and on-the-job application; we haven’t seen much innovation in our field.

If the research on insight is right, then without new data (or new methods to gather data in the case of an industry-wide perspective) we will not have breakthrough insights about how to improve our training and other learning interventions.

We need to continue working toward better data-gathering methods.

I think the Performance-Focused Smile Sheet offers a glimmer of hope, but the recent infatuation with benchmarking does NOT.

What have you done recently to help your company get better data about your learning-and-performance initiatives?

 

Citation for article that triggered this insight:

Klein, G., & Jarosz, A. (2011). A naturalistic study of insight. Journal of
Cognitive Engineering and Decision Making, 5(4), 335-351.

A Better Youth Soccer Evaluation Form

,

As a youth soccer coach for many years I have struggled to evaluate my own players and have seen how my soccer league evaluates players to place them on teams. As a professional learning-and-performance consultant who has focused extensively on measurement and evaluation, I think we can all do better, me included. To this end, I have spent the last two years creating a series of evaluation tools for use by coaches and youth soccer leagues. I’m sure these forms are not perfect, but I’m absolutely positive that they will be a huge improvement over the typical forms utilized by most youth soccer organizations. I have developed the forms so that they can be modified and they are made available for free to anyone who coaches youth soccer.

At the bottom of this post, I’ll include a list of the most common mistakes that are made in youth-soccer evaluation. For my regular blog readers–those who come to me for research-based recommendations on workplace learning-and-performance–you’ll see relevance to your own work in this list of evaluation mistakes.

I have developed four separate forms for evaluation. That may seem like a lot until you see how they will help you as a coach (and as a soccer league) meet varied goals you have. I will provide each form as a PDF (so you can see what the form is supposed to look like regardless of your computer configuration) and as a Word Document (so you can make changes if you like).

I’ve also provided a short set of instructions.

 

Note from Will in November 2017:

Although my work is in the workplace learning field, this blog post–from 2012–is one of the most popular posts on my blog, often receiving over 2000 unique visitors per year.

 

The Forms

1.   Player Ranking Form:  This form evaluates players on 26 soccer competencies and 4 player-comparison items, giving each player a numerical score based on these items AND an overall rating. This form is intended to provide leagues with ranking information so that they can better place players on teams for the upcoming season.


2. Player Development Form:  This form evaluates players on the 26 soccer competencies. This form is intended for use by coaches to help support their players in development. Indeed, this form can be shared with players and parents to help players focus on their development needs.


3. Team Evaluation Form:  This form helps coaches use practices and games to evaluate their players on the 26 key competencies. Specifically, it enables them to use one two-page form to evaluate every player on their team.

4. Field Evaluation Form:  This form enables skilled evaluators to judge the performance of players during small-group scrimmages. Like the Player Ranking Form, it provides player-comparison information to leagues (or to soccer clubs).

The Most Common Mistakes in Youth-Soccer Evaluation

  1. When skills evaluated are not clear to evaluators. So for example, having players rated on their “agility” will not provide good data because “agility” will likely mean different things to different people.
  2. When skills are evaluated along too many dimensions. So for example, evaluating a player on their “ball-handling skills, speed, and stamina” covers too many dimensions at once—a player could have excellent ball-handling skills but have terrible stamina.
  3. When the rating scales that evaluators are asked to use make it hard to select between different levels of competence. So for example, while “ball-handling” might reasonably be evaluated, it may be hard for an evaluator to determine whether a player is excellent, very good, average, fair, or poor in ball-handling. Generally, it is better to have clear criteria and ask whether or not a player meets those criteria. Four or Five-Point scales are not recommended.
  4. When evaluators can’t assess skills because of the speed of action, the large number of players involved, or the difficulty of noticing the skills targeted. For example, evaluations of scrimmages that involve more than four players on a side make it extremely difficult for the evaluators to notice the contributions of each player.
  5. When bias affects evaluators’ judgments. Because the human mind is always working subconsciously, biases can be easily introduced. So for example, it is bad practice to give evaluators the coaches’ ratings of players before those players take part in a scrimmage-based evaluation.
  6. When bias leads to a generalized positive or negative evaluation. Because evaluation is difficult and is largely a subconscious process, a first impression can skew an evaluation away from what is valid. For example, when a player is seen as getting outplayed in the first few minutes of a scrimmage, his/her later excellent play may be ignored or downplayed. Similarly, when a player is intimidated early in the season, a coach may not fully notice his/her gritty determination later in the year.
  7. When bias comes from too few observations. Because evaluation is an inexact process, evaluation results are likely to be more valid if the evaluation utilizes (a) more observations (b) by more evaluators (c) focusing on more varied soccer situations. Coaches who see their players over time and in many soccer situations are less likely to suffer from bias, although they too have to watch out that their first impressions don’t cloud their judgments. And of course, it is helpful to get assessments beyond one or two coaches.
  8. When players are either paired with, or are playing against, players who are unrepresentative of realistic competition. For example, players who are paired against really weak players may look strong in comparison. Players who are paired as teammates with really good players may look strong because of their teammates’ strong play. Finally, players who only have experience playing weaker players may not play well when being evaluated against stronger players even though they might be expected to improve by moving up and gaining experience with those same players.
  9. When the wrong things are evaluated. Obviously, it’s critical to evaluate the right soccer skills. So for example, evaluating a player on how well he/she can pass to a stationary player is not as valid as seeing whether good passes are made in realistic game-like situations when players are moving around. The more game-like the situations, the better the evaluation.
  10. When evaluations are done by remembering, not observing. Many coaches fill out their evaluation forms back home late at night instead of evaluating their players while observing them. The problem with this memory-based approach is that introduces huge biases into the process. First, memory is not perfect, so evaluators may not remember correctly. Second, memory is selective. We remember some things and forget others. Players must be evaluated primarily through observation, not memory.
  11. Encouraging players to compare themselves to others. As coaches, one of our main goals is to help our players learn to develop their skills as players, as teammates, as people, and as thinkers. Unfortunately, when players focus on how well they are doing in comparison to others, they are less likely to focus on their own skill development. It is generally a mistake to use evaluations to encourage players to compare themselves to others. While players may be inclined to compare themselves to others, coaches can limit the negative effects of this by having each player focus on their own key competencies to improve.
  12. Encouraging players to focus on how good they are overall, instead of having them focus on what they are good at and what they still have to work on. For our players to get better, they have to put effort into getting better. If they believe their skills are fixed and not easily changed, they will have no motivation to put any effort into their own improvement. Evaluations should be designed NOT to put kids in categories (except when absolutely necessary for team assignments and the like), but rather to show them what they need to work on to get better. As coaches, we should teach the importance of giving effort to deliberate practice, encouraging our players to refine and speed their best skills and improve on their weakest skills.
  13. Encouraging players to focus on too many improvements at once. To help our players (a) avoid frustration, (b) avoid thinking of themselves as poor players, and (c) avoid overwhelming their ability to focus, we ought to have them only focus on a few major self-improvement goals at one time.

 

Is Your Training Course Likely to Boost Performance? Free Course Review Template.

, ,

Despite decades of advocacy by our best trade associations, our wisest gurus, and our most practical researchers, most organizations today still rely on training courses that have little impact in promoting on-the-job performance.

As I mentioned in a recent article, we as learning professionals continue to fail in five major ways. You can access that article by clicking here.

I used to think that this was just a failure of knowledge, but in most of the organizations in which I’ve consulted, there are at least a few learning-and-performance professionals who understand that training alone is not enough. Part of the problem is the dead weight of tradition–the “old normal” continues to blind us to new possibilities. The enlightened few have a hard time pushing back against the gravitational pull of this mass hypnosis.

I recently had a new insight–a way of looking at this problem that I think might enable organizations to break out of their bad habits. The solution is that we have to gain control of the leverage points we have to push for change. We have to change the levers that warp and control our thinking. The big lever is learning measurement. I’ve been pushing this for years as our most important leverage point. If we measured better, we’d get better feedback, which would push us to create better learning interventions.

But learning measurement isn’t our only lever and changing your learning measurement practices is not always easy politically. Beside learning measurement, I’ve compiled a whole list of other leverage points that  really matter. In fact, it was only recently that I had this incredible insight (one I maybe should have had 10 years ago), that we ought to figure out all the levers we have at our disposal and change them to help push our organizations toward a performance orientation. I’d like to reveal one of those levers today.

One of the things we do in our organizations is review our training courses from time to time–either intentionally or by osmosis and feeling. Well, instead of using the wrong metrics, why not use methods that we know–based on our understanding of learning-and-performance–are likely to be good indicators of whether our training course will support actual on-the-job performance.

The Course Review Template is something that can be used on any training course–classroom training or e-learning.It includes a set of questions that are indicators of how performance-based your training course is. Each rubric in this tool is inspired by research or proven practices which I’ve learned in my 25+ years in the workplace learning field.

I should give you a warning. You’re unlikely to be happy with what you find. If I bet each of you one dollar for each training course of yours that doesn’t support performance, I’d be a millionaire overnight.

But to be fair, I’m going to let you try out the tool yourself. It’s free. Use it. And, let me know how your training courses rate. Are they likely to improve on-the-job performance or not?

Click to Download the Course Review Template

After you review a course, post your results at the following link, and when we get enough responses, we’ll let you compare your results to others.

Click to Post Your Course Review Results — SORRY, we’re done collecting data in a survey format.

Maybe I’m having a momentary bout of delusional cognition, but I’m thinking right now that this simple Course Review Template might just revolutionize our ability to simply review our courses to see how performance focused they are.

Such a grandiose statement will provoke eye rolls in some, so let me stipulate a few things. First, this is a first draft, so the Course Review Template is going to be imminently improvable. Second, the Course Review Template is NOT a precision instrument. It is not psychometrically derived, the numbers it assigns to each rubric are best guesses, and there was no super-committee here–just me. Third, the rubrics themselves are subject to interpretation. Instead of over-complicating the form and making it unusable, I decided to keep it simple and make it less precise. Finally, course reviews are just one of the levers you’ll need to completely transition from a course-focus to a performance-focus.

The bottom line is that we have to try some innovate new things to push our organizations to a performance focus. The old ways have not worked. The Course Review Template–or something like it–is worth a try. And seriously, I think it could revolutionize the way your organization views its training courses.

NOTE 2017: While this is the original blog post, it now includes the latest version of the Course Review Template. A later post that introduced the improvements is available here.

 

2011 Social-Media Measurement Survey (for Workplace Learning Field)

,

Social Media is hot, but it is not clear how well we are measuring social media.

A couple of years ago I wrote an article for the eLearning Guild about measuring social media. But it's not clear that we've got this nailed yet.

With this worry in mind, I've created a research survey to begin a process to see how best social-media (of the kind we might use to bolster workplace learning-and-performance) can be measured.

Here's the survey link. Please take the survey yourself. You don't have to be an expert to take it.

Here's my thinking so far on this. Please send wisdom if I've missed something.

  1. We can think about measuring social media the same way we measure any learning intervention.
  2. We can also create a list of all the proposed benefits for social media, and the proposed costs, and all the proposed harms, and we can see how people are measuring these now. The survey will help us with this second approach.

Note: Survey results will be made available for free. If you take the survey, you'll get early releases of the survey results and recommendations.

Also, this is not the kind of survey that needs good representative sampling, so feel free to share this far and wide.

Here is the direct link to the survey:   http://tinyurl.com/4tlslol

Here is the direct link to this blog post:   http://tinyurl.com/465ekpa

Benchmarking

,

Research Benchmarking

Research Benchmarking is the process by which your learning interventions are benchmarked against research-based best practices.

While random-assignment between-group research is likely to be too expensive and time-consuming for most of us, and benchmarking our work against other industry players is likely to push us toward mediocrity, Research Benchmarking offers a potent alternative.

Learning programs are examined to determine how well they (a) create understanding, (b) support long-term remembering (minimize forgetting), and (c) motivate on-the-job performance. They will be research-benchmarked against the 12 most decisive factors in learning design.

If you’d like to discuss research benchmarking further, contact me, Dr. Will Thalheimer, at 1-888-579-9814 or email me.

 

Assessment Guru Hints at Trends Relevant to Learning Professionals

,

Learning professionals (like me) can often gain insights about our industry from people in the field who have different vantage points than our own. I recently talked with Eric Shepherd, CEO of Questionmark, to get a sense of our industry and how it has been affected by the bad economy. Eric has been a good friend and long-time supporter of my research over the years and I’ve come to value his counsel.

Questionmark is the leading provider of assessment software according to a recent eLearning Guild study. I thought from his perch overseeing all-things-assessment, Eric might be able to give us some unique insight into the learning-and-performance field in general.

Check out my interview with him at the recent Guild conference. I divided it into two parts to make viewing easier.

Part 1: What trends do you see that we may be missing? 

Part 2: How is the bad economy affecting the learning assessment marketplace? 

Another Smile Sheet Improvement

,

Last year I wrote at length about my efforts to improve my own smile sheets. It turns out that this is an evolving effort as I continue to learn from my learners and my experience.

Check out my new 2009 version.

You may remember that one of the major improvements in my smile sheet was to ask learners about the value and newness of EACH CONCEPT TAUGHT (or at least each MAJOR concept). This is beneficial because people respond more accurately to specifics than to generalities, they respond better to concrete learning points than to the vague semblance of a full learning experience.

What I forgot in my previous version was the importance of getting specific feedback on how well I taught each concept. Doh!

My latest version adds a column for how well each concept is taught. There is absolutely no more room to add any columns (I didn't think I could fit this latest one in), so I suppose this may allow diminishing returns on any more improvements.

Check it out and let me know what you think.