Some reflections on "Measuring training from a new science perspective," by Jim Rough
[Journal for Quality and Participation, October/November 1994]

1) Change happens through manageable forces; 2) Knowledge is an objective thing, transferable from one person to another; 3) Setting learning goals and measuring progress toward their achievement makes sense.

To quote Deming, "Simple…obvious…and wrong!" People have insights and breakthroughs in "fits and starts"…not linearly. Considering desired outcomes of quality training such as "integrity," "listening skills," "creativity," or "level of caring and quality," how would one measure them?

Conceptual understanding and practice aren't good enough. Participants must discover these for themselves, within themselves to new levels of understanding. It is not a mere simple replication process.

Rating attitudes will not do it.

When transformation is involved, people either "get it" ("Everything was perfect!)," want to "get it," but don't (very sincerely pointing out limitations in the facilitator's style, methodology, or materials), or...

…don't want to "get it." This last group is easily recognizable -- they look like they'd rather be in a dental chair -- and wants easy answers. They are looking for any excuse not to change, usually taking something the facilitator says out of context, feign "offense," and use that as an excuse to question the ability/integrity of the facilitator, negating their need to learn -- never face-to-face, of course, but anonymously and passive-aggressively through the written "evaluation."

• It has been my experience that people are never angry for the reason they think.  Learn to say "God bless 'em" and "swallow your ego" when you read such evaluations! 

The good news is…

…an initial negative reaction can be a sign that the people are finally waking up -- struggle and frustration are natural and healthy. Motivated participants will grope, lurch, break through, and evolve. 

If conference organizers or in-house facilitators are not careful, literal consideration of ratings may even drive future sessions toward superficiality. Transformational sessions will never compare well with entertainment sessions or informational sessions.

• Educators have an obligation to be conscious of "How's it going?" and continually solicit feedback through questions such as "How many of you are struggling to understand this right now?"

Testing conceptual understanding won't do it either

It's not what people know, but, rather, their new way of being. All a test tells you is they "know it" -- today -- or, in many of my experiences, they know how to tell the instructor what he/she wants to hear!

Measuring the use of skills may not do it either

Even with successful seminars, people may not change their outward activities in a measurable way! It may take a crisis or critical mass of people who've "gotten" it before skills may be called into use. 

And, shockingly, measuring the ultimate results won't necessarily do it

"Did costs go down?," "Did overall quality improve?" -- seems foolproof. Sometimes, the definition of "results" can change in the transformed environment -- current cost accounting methods may no longer be relevant!

What if Van Gogh changed his style of painting to be more popular, please the critics, or the make more money? To quote the author: 

Often, at the beginning of seminars, I have people write down their goals; I encourage them to aim high. At the end of the seminar, they often see the original goals as unimportant or even irrelevant…[However], acknowledging self-reflection and reference to the past as the ultimate yardstick does not deny the importance of listening to the customer or seeking extrinsically objective measures…We must be sensitive to the barriers that managed change can create. Training processes that are fully measurable won't give us the results we seek. The training we need is transformational.