The Trainer’s Handbook

Chapter 5 - Evaluating Your Effectiveness

Level Three: Bottom-Line Evaluation

In order to move from a reactive to a proactive status, training must become an integral part of strategic planning in the organization. Indeed, as we shall see in Chapter 19, U.S. industry is moving in the direction of increasing the importance of training in its operations. However, in order to become incorporated in strategic planning, training must be able to demonstrate its impact on organizational operations in a tangible way. This means training must be evaluated in terms of the bottom line. If training is to create a positive change in work-related behavior, then that change must be examined and demonstrated. It must be evaluated as a return on investment.

We looked at the basic procedures for creating a relationship between costs of training and the results of that training in Chapter 4. But it must be emphasized here that the key element in the process is creating a dollar value unit that can be related back to improved skill performance upon completion of training. Obviously, if you have justified the costs of providing the training against promised improved performance, you are obligated to monitor the posttraining performance in order to assess the bottom-line impact your training has had.

Two issues need to be addressed. The first one is what an executive in one of my courses called “Jell-O on [her] desk.” If you are going to make cost comparisons, they must be as substantial as possible, not just shimmering, insubstantial numbers that look nice but have no solidity to them. Jell-O has no place in strategic planning and so cannot contribute to either cost justification or evaluation. Make your cost estimates and justifications as solid and tight as possible. To the extent that you succeed, you will find them easy to evaluate. To the extent that your cost/benefit relationships are ephemeral, you will find it difficult to evaluate the bottom line.

The second issue arises in those situations where there is no dollar-oriented return on investment. This is a particularly common situation in the civil service and in nonprofit organizations. It is still vital, though, to define a relationship between the costs of operations and the results obtained. In such organizations the bottom line is usually the volume of work to be done, the distribution of the budget, or a combination of both.

The volume of work is a function of staffing and hours; therefore, relationship between training and the number of necessary staff or a reduction in the time needed to complete a task will both constitute the bottom line. For example, many companies in the U.S. have undergone reengineering and downsizing. This invariably means maintaining a staff of fewer workers to perform the same amount of work. Training these workers to become more efficient, better organized, or more skilled at their tasks will impact on how well or how poorly the organization maintains its vital services. Thus, dollars spent on training can be directly related to specific improvements in performance. This improved performance can be measured and evaluated as a bottom-line improvement, a better distribution of limited budget moneys.

Finally, in order to establish a non?Jell-O, factual relationship between improved performance and training, whether by dollars or efficiency, it is often necessary to set up a pilot program in order to verify and document that money spent on training does affect the bottom line. In the face of limited budgets and management unfamiliar with the benefits of training, it is advisable to propose such a pilot study be done first. This is, after all, how contractors sell large programs to the government. Pilot first to prove that training works, then roll out a full program using the data collected from the pilot study as parameters for evaluation.

Level three evaluation is a measure of the impact of training on the organization at large. It can relate to a mission statement, a budget, departmental goals, or training objectives, but its purpose is to assess how well training has succeeded in creating changed performance in the organization.

Summary

Evaluation is the feedback that keeps training programs on track. It is how you can measure the changes you are bringing about. Evaluations are also vital for trainees to see how they are progressing. Evaluation engages the law of effect by forming the base against which performance can be measured.

There are three levels of evaluation: short-term during the training; long-term evaluations on the job; and bottom-line, return-on-investment, corporate-impact evaluations. There are also two areas in which to evaluate: cognitive learning and affective learning. Of these, the short-term impact of cognitive training is the easiest to evaluate, using traditional tools such as exams, quizzes, questions, or projects. Affective learning is harder to evaluate because it is more subjective. However, by observing mood and effort, you often have enough information for a short-term evaluation.

By far the most difficult are long-term evaluations of affective learning. Measuring long-term cognitive skills is only marginally less troublesome. For both, the essential tasks are (1) to monitor key variables, (2) to remain in touch with those you’ve trained, and (3) to convert all long-term data to percentages so as to counter the problem of a shrinking population.

Third level, bottom-line evaluation fits the training function into the overall activity of the organization and provides training with perspective on the impact it has throughout the organization.

Finally, remember that the purpose of evaluation is not just to see how well you’ve done, though that is important, but to diagnose those areas in which you can do better. We don’t evaluate just to prove we did something, but rather to improve how we do it in the future.

Applications

1. Select a course of study either from existing training in your organization or from an off-the-shelf package or a college curriculum. List the tools for evaluation of learning that are apparent from this course. Create three more means for evaluating short-term learning.

2. Select a lesson. Create a short test to assess learning. Be sure to include at least three different types of questions to assess three different levels of learning. Try the test out on trainees and record the data in a distribution curve.

3. Take a critical incident from your workplace and write up a detailed case history that could be used in class to teach principles and skills.

4. Think of the worst training you have ever received. Write down what made it poor. Now create new methods and evaluation tools to change it into dynamic and effective training.

Excerpts from Chapter 5, The Trainer's Handbook

1. Evaluating Effectiveness
2. Short-term Evaluation
3. Project Sessions
4. Case Histories and Practice Sessions
5. Examinations
6. Types of Exam Questions
7. Assessment Sessions
8. Self-evaluation
9. On-the-Job Evaluation
10. Long-term Evaluations
11. Bottom-line Evaluation

More learning articles:

• The miracle of learning
• Orienting new employees
• More about the role of the trainer
• Learning Organizations
• Knowledge Management
• The value of lectures