Use of ROI approach in measuring learningMeasuring learning should be about generating business change and results. What is a business result and how do you recognize it? The business result is basically the outcome of the behavior and attitude changes. When measuring learning you generally determine the business result through discussions with your stakeholders about their project objectives and likely outcomes, or based on noticeable changes in behavior which impact the business (e.g., longer hours worked, better teamwork, etc.) ROI is basically a numeric comparison of the total business result to the learning program (intervention) costs. It is a numerical representation of the value added by a program, and ties the program’s success to a financial result. The ROI methodology was developed by Dr Jack Phillips and provides a step by step approach for detemining the business results of a program and measuring learning. Calculating the ROI of a learning initiative is a very positive thing to do but shuld not be undertaken lightly.
However many of the principles and approaches can be scaled down and use to aid in measuring learning. This page will give you a high level overview and a starting point but get yourself knowledgeable and skilled before having a go (or telling your CEO.) You can find more about the ROI methodology and Dr Jack Phillips here
Adapted from Phillips, Jack (1997) Return on Investment in Training and Performance Improvement Programs 1. Collect Pre-Program Data Example: Suppose the objective for a company is to reduce customer complaints numbers and costs. The company decides to implement a training program that focuses on educating participants about successfully avoiding complaints and how to handle those complaints that happen as effectively as possible. In order to be able to track the change in number and cost of complaints, you will probably need to collect the following pre-program data for the previous numbers and cost of complaints over the last 6 to 9 months. You need data over a time period to identity any possible existing trends Having the essential pre-program data will allow you to compare post-program costs and benefits with the previous year in order to determine the ROI of doing the training program and complte the measuring of learning. Click here for more information on measuring learning2. Collect Post-Program Data The six major categories of data sources include:
Keep in mind that the best methods used to collect program data are those that are non-obtrusive. The more human opinion that enters into the measurement, the greater chance for bias and inaccuracies. Isolating the Effects of the Learning Programme
The biggest barrier and key challenge to measuring learning and building a credible ROI is to accurately depict a cause and effect relationship between training programs and business results.
The challenge is to develop one or more specific strategies to isolate the effects of training early in the process and communicate these strategies with key stakeholders. As a first step, all of the key factors that might have contributed to the performance improvement should be identified. This will communicate to your client and interested parties that other factors may have influenced the results, underscoring that the training program may not be the sole source of improvement. Identifying these other factors illustrates that you are sharing credit for the improvement with several possible variables, which is likely to gain the respect of management. Several potential sources can help identify these other factors:
4. Convert Data to Monetary Benefits
After isolating the effect of the program, the next major step in the ROI process is converting the data collected to monetary values and comparing it to the program costs. This requires a value to be placed on each data item connected with the program. Values can be placed on both hard and soft data. Hard data is objective, easy to measure and easy to convert to monetary values. Soft data is usually subjective, behaviorally oriented, sometimes difficult to measure, and almost always difficult to convert to monetary values. The following general steps should be used to convert both hard and soft data to monetary values:
5. Tabulate Programme Costs
After all the program data has been collected, flushed out, and converted to monetary values, the costs of the training program can be calculated. Tabulating program costs is an essential step in the ROI calculation process. Although costs are often more easily captured than benefits, it is just as important to pay attention to costs as it is to benefits. Tabulating the costs involves monitoring all of the related costs of the program targeted for the ROI calculation. Among the cost components that should be included are: Ø The cost to design and develop the program, possibly prorated over the expected life of the program Ø The cost of all program materials provided to participants Ø The cost for the instructors, including preparation and delivery time Ø The cost of facilities for the training program Ø Travel, lodging, and meal costs for participants Ø Salaries plus employee benefits of participants who attend training Ø Administrative and overhead costs of the training function, allocated in some convenient way In addition, specific costs related to the needs assessment and evaluation should be included. Because of the scrutiny involved in ROI calculations, the conservative approach is to include all of these costs so that the total is fully loaded. In our customer complaint example the total costs were established to be $15,200 5. Caculate the Return
After the learning program benefits are collected and converted to monetary values and the program costs are developed in a fully loaded profile, the ROI calculation becomes very easy. It is just a matter of plugging the values into the appropriate formula. The basic approach for calculating the return: (1) The ROI formula. The return on investment calculation uses net benefits divided by program costs. The net benefits are the program benefits minus the costs. In formulaic form, the ROI becomes: ROI (%) = Net Program Benefits (Benefits – Costs) / Program Costs x 100 For our complaints example Net benefits are =$24,615 - $15,200/ $15,200 x 100
This provides a ROI of: 61.9% While there are no generally accepted standards when measuring learning return some organizations establish a minimum requirement for an ROI on training, say 25%. This target value is usually above the percentage required for other types of investments Because of the complexity and sensitivity of the ROI process, caution is needed when developing, calculating, and communicating the return on investment. A few precautions should be noted: Ø The ROI process should be developed for programs where a needs assessment has been conducted. Ø The ROI analysis should always include one or more strategies for isolating the effects of training. Ø The most reliable and credible sources should be used when making estimates. Ø Take a conservative approach for calculating both benefits and costs. Ø Use caution when comparing the ROI in training and development with other financial returns. Ø Involve management in developing the return. Ø Approach sensitive and controversial issues with caution (i.e., what is measurable vs. not) Ø Teach others the methods for calculating the return. Ø Do not boast about a high return. Ø Do not try to use ROI on every program. 6. Identify Intangible Measures
In addition to tangible, monetary benefits, most training and learning programs will have intangible, non-monetary benefits. When measuring learning identifying the intangible benefits associated with the training program is the final step in the implementation phase. Intangible benefits are those benefits that it is impossible or too costly to quantify. For some programs, intangible, non-monetary benefits are extremely valuable, often carrying as much influence as the hard data items. Intangible benefits can be identified in four different ways: Ø During the needs assessment Ø Through discussions with the client or sponsor Ø During the attempt to convert the data to monetary values Ø During a follow-up evaluation Typical intangible variables typically linked when measuring learning include items such as:
Intangible measures should be identified, explored, examined, and monitored for changes when they are linked to the program. When measuring learning collectively, they add a unique dimension to the overall program results since most, if not all, programs have intangible measures associated with them. More on measuring learning here
|
