Use of ROI approach in measuring learning

Measuring learning should be about generating business change and results.

What is a business result and how do you recognize it? The business result is basically the outcome of the behavior and attitude changes.

When measuring learning you generally determine the business result through discussions with your stakeholders about their project objectives and likely outcomes, or based on noticeable changes in behavior which impact the business (e.g., longer hours worked, better teamwork, etc.)

ROI is basically a numeric comparison of the total business result to the learning program (intervention) costs. It is a numerical representation of the value added by a program, and ties the program’s success to a financial result.

The ROI methodology was developed by Dr Jack Phillips and provides a step by step approach for detemining the business results of a program and measuring learning.

Calculating the ROI of a learning initiative is a very positive thing to do but shuld not be undertaken lightly.



Note that Jack Philips, an American ROI 'guru', recently commented about training ROI: "Organisations should be considering measuring learning and implementing ROI impact studies very selectively on only 5 to 10 per cent of their training programme, otherwise it becomes incredibly expensive and resource intensive."

However many of the principles and approaches can be scaled down and use to aid in measuring learning.

This page will give you a high level overview and a starting point but get yourself knowledgeable and skilled before having a go (or telling your CEO.)

You can find more about the ROI methodology and Dr Jack Phillips here

Adapted from Phillips, Jack (1997) Return on Investment in Training and Performance Improvement Programs

1. Collect Pre-Program Data

The first step in the measuring learning implementation phase is to collect pre-program data. If you are going to implement a measurement strategy to compare metrics before and after an intervention has been completed, then you will need to collect pre-program data so that you have baseline measures for comparison.

Example: Suppose the objective for a company is to reduce customer complaints numbers and costs. The company decides to implement a training program that focuses on educating participants about successfully avoiding complaints and how to handle those complaints that happen as effectively as possible. In order to be able to track the change in number and cost of complaints, you will probably need to collect the following pre-program data for the previous numbers and cost of complaints over the last 6 to 9 months. You need data over a time period to identity any possible existing trends

Having the essential pre-program data will allow you to compare post-program costs and benefits with the previous year in order to determine the ROI of doing the training program and complte the measuring of learning.

Click here for more information on measuring learning

2. Collect Post-Program Data

Collecting data after the training program is the next step of the ROI process. There are a wide variety of data sources and data collection methods available to capture post-program data for measuring learning.

The six major categories of data sources include:

  • Organizational Performance Records
  • Participants
  • Supervisors to Participants
  • Subordinates of Participants
  • Team/ Peer Group
  • Internal/External Groups


    There are two common methods used to obtain data from these data sources

  • Extracting from existing organization performance records
  • Conducting follow-up questionnaires

    Keep in mind that the best methods used to collect program data are those that are non-obtrusive. The more human opinion that enters into the measurement, the greater chance for bias and inaccuracies.

    Isolating the Effects of the Learning Programme

     

    The biggest barrier and key challenge to measuring learning and building a credible ROI is to accurately depict a cause and effect relationship between training programs and business results.

     

    The challenge is to develop one or more specific strategies to isolate the effects of training early in the process and communicate these strategies with key stakeholders. 

     

    As a first step, all of the key factors that might have contributed to the performance improvement should be identified.  This will communicate to your client and interested parties that other factors may have influenced the results, underscoring that the training program may not be the sole source of improvement.  Identifying these other factors illustrates that you are sharing credit for the improvement with several possible variables, which is likely to gain the respect of management.

     

     

                Several potential sources can help identify these other factors:

     

    •      Client Leadership
    •       Program Participants
    •       Program Analysts and Developers
    •       Participants’ Supervisors

     After these resources have been identified, there are various techniques you can use to approximate the impact of the training program:

     

    •      Use of control groups
    •       Trend line analysis
    •       Forecasting methods
    •       Participant estimate of training’s impact
    •       Supervisor’s estimate of training’s impact
    •       Customer’s estimate of training’s impact
    •       Expert’s estimate of training’

     

     

    4. Convert Data to Monetary Benefits

     

    After isolating the effect of the program, the next major step in the ROI process is converting the data collected to monetary values and comparing it to the program costs.  This requires a value to be placed on each data item connected with the program.  Values can be placed on both hard and soft data.  Hard data is objective, easy to measure and easy to convert to monetary values.  Soft data is usually subjective, behaviorally oriented, sometimes difficult to measure, and almost always difficult to convert to monetary values.

     

    The following general steps should be used to convert both hard and soft data to monetary values:

     

    ·         Focus on a unit of measure

     

    Reduction in number of complaints

    ·         Determine a value of each unit

     

    Take an average cost per complaint

    Include direct and indirect costs

    In this case $547

    ·         Calculate the change in performance data

     

    6 months after the program complaints

    dropped by 50

    30 of those reductions were  directly attributed by supervisors to be as a result of techniques taught on the training programme

    ·         Determine an annual amount for the change

     

    Looking at the trend it was decided an annual reduction in 45 complaints was both conservative and realistic

    ·         Calculate the total value of the improvement

     

    Total value of improvement attributable to training is 45 x 447 =$24,615

     

     

    5. Tabulate Programme Costs

     

    After all the program data has been collected, flushed out, and converted to monetary values, the costs of the training program can be calculated.  Tabulating program costs is an essential step in the ROI calculation process.  Although costs are often more easily captured than benefits, it is just as important to pay attention to costs as it is to benefits.  Tabulating the costs involves monitoring all of the related costs of the program targeted for the ROI calculation.  Among the cost components that should be included are:

     

    Ø      The cost to design and develop the program, possibly prorated over the expected life of the program

    Ø      The cost of all program materials provided to participants

    Ø      The cost for the instructors, including preparation and delivery time

    Ø      The cost of facilities for the training program

    Ø      Travel, lodging, and meal costs for participants

    Ø      Salaries plus employee benefits of participants who attend training

    Ø      Administrative and overhead costs of the training function, allocated in some convenient way

     

    In addition, specific costs related to the needs assessment and evaluation should be included.  Because of the scrutiny involved in ROI calculations, the conservative approach is to include all of these costs so that the total is fully loaded.

     

    In our customer complaint example the total costs were established to be $15,200

     

    5. Caculate the Return

     

    After the learning program benefits are collected and converted to monetary values and the program costs are developed in a fully loaded profile, the ROI calculation becomes very easy.  It is just a matter of plugging the values into the appropriate formula.  The basic approach for calculating the return: (1)  The ROI formula.

     

    The return on investment calculation uses net benefits divided by program costs.  The net benefits are the program benefits minus the costs.  In formulaic form, the ROI becomes:

     

    ROI (%) = Net Program Benefits (Benefits – Costs) / Program Costs x 100

     

    For our complaints example  Net benefits are =$24,615 - $15,200/ $15,200 x 100

     

    This provides a ROI of: 61.9%

     

     

    While there are no generally accepted standards when measuring learning return  some organizations establish a minimum requirement for an ROI on training, say 25%.  This target value is usually above the percentage required for other types of investments

     

    Because of the complexity and sensitivity of the ROI process, caution is needed when developing, calculating, and communicating the return on investment.  A few precautions should be noted:

     

    Ø      The ROI process should be developed for programs where a needs assessment has been conducted.

    Ø      The ROI analysis should always include one or more strategies for isolating the effects of training.

    Ø      The most reliable and credible sources should be used when making estimates.

    Ø      Take a conservative approach for calculating both benefits and costs.

    Ø      Use caution when comparing the ROI in training and development with other financial returns.

    Ø      Involve management in developing the return.

    Ø      Approach sensitive and controversial issues with caution (i.e., what is measurable vs. not)

    Ø      Teach others the methods for calculating the return.

    Ø      Do not boast about a high return.

    Ø      Do not try to use ROI on every program.

     

    6. Identify Intangible Measures

     

    In addition to tangible, monetary benefits, most training and learning programs will have intangible, non-monetary benefits.  When measuring learning identifying the intangible benefits associated with the training program is the final step in the implementation phase. 

     

    Intangible benefits are those benefits that it is impossible or too costly to quantify.

     

     For some programs, intangible, non-monetary benefits are extremely valuable, often carrying as much influence as the hard data items.

     

    Intangible benefits can be identified in four different ways:

     

    Ø      During the needs assessment

    Ø      Through discussions with the client or sponsor

    Ø      During the attempt to convert the data to monetary values

    Ø      During a follow-up evaluation

     

    Typical intangible variables typically linked when measuring learning include items such as:

     

    EMPLOYEE SATISFACTION

    Ø        Attitude Survey Data

    Ø        Organizational Commitment

    Ø        Climate Survey Data

    Ø        Employee Complaints

    Ø        Grievances

    Ø        Discrimination Complaints

    Ø        Stress Reduction

     

    EMPLOYEE WITHDRAWAL

    Ø        Employee Turnover

    Ø        Employee Absenteeism

    Ø        Employee Tardiness

    Ø        Employee Transfers

    CUSTOMER SERVICE

    Ø        Customer Satisfaction Survey Data

    Ø        Customer Complaints

    Ø        Customer Response Time

     

    TEAM EFFECTIVENESS

    Ø        Teamwork

    Ø        Cooperation

    Ø        Conflict

    Ø        Decisiveness

    Ø        Communication

     

     

    Intangible measures should be identified, explored, examined, and monitored for changes when they are linked to the program.  When measuring learning collectively, they add a unique dimension to the overall program results since most, if not all, programs have intangible measures associated with them. 

    More on measuring learning here

    Back to learning evaluation