Advanced Concepts: Evaluating Training Programs
One of the questions that I get frequently is “How will we know that a training effort is working?”
It’s a good question – and one that deserves a thorough answer. Whether you are an HR Director, Training Manager, Department Manager, or even a classroom participant it can be helpful to know the process of evaluation.
The following article is a little longer than usual, however I think you’ll find it interesting and thought provoking. I look forward to hearing your feedback – and discussing how these practices can increase the effectiveness of training in yourorganization!
Section One: Conducting a Needs Analysis
Even when training needs appear to be obvious, it is still necessary to take steps to ensure that your programs meet a specific business need and provide your organization with proof that your efforts add to the bottom line rather than subtract.
In order to analyze specific knowledge and skill requirements you first must choose your methods of determination for the sake of accuracy and prioritization.
Without the benefit of analysis, it is easy to see how time and finances can be invested in the wrong areas. The consequences can be grave if you aren’t making careful considerations. You need to avoid having your efforts be perceived as an expense, and something that does not add value to the organization.
When determining whether or not training is the answer to a particular issue in your organization there are three main areas of analysis to consider;
1) Skills Analysis – Is there a gap in employee knowledge or skill?
2) Situational Analysis – Have you identified process and/or behavioral issues?
3) Organizational Analysis – Is the issue one of organizational development?
If your issue falls clearly into one of these areas, you can concentrate your efforts on examining information that enables you to operate effectively in that one area alone. However if you are like most of the clients we work with, you’ll find that these analysis areas tend to change over time.
Assessing Needs – In practical terms, assessment is a process used by individuals or organizations to determine the value or worth of something. Assessments often relate to the determination of monetary values, but for us the focus is on the assessment of performance.
Value or worth is then derived from the relationship of current performance versus desired performance. This process is commonly referred to as “needs assessment”.
Typically, needs can and should be looked at through several “lenses”. The three recommended lenses in the graphic below give a global viewpoint of potential training needs.
1) Mega: The societal or industrial level stretching beyond your own walls
2) Macro: The organizational or departmental level
3) Micro: Individual or team needs level
Even when applied to a single issue within an organization, assessments at all three levels help us to determine or approximate the value of various discrepancies identified between current and desired results across vast spectrums. These results provides an essential ingredient in the identification and prioritization of the desired training initiatives.
Don’t rely on the assumptions that desired results are known, agreed upon, and similarly evaluated throughout the organization. This can be dangerous. However, when you collect data on both current and desired results, your needs assessment won’t reflect an assumption that desired performance has already been well defined and measured. You will instead be able to identify gaps using a consistent scale of measurement and be able to see real and/or perceived differences between current individual or organizational performance and the desired or required performance.
After you identify and collect data regarding desired and current performance at each level, calculate the discrepancies between current and desired results. Then classify these discrepancies as either needs or strengths. When current results are not meeting the necessary requirements of the desired results, you have identified a need. Similarly, when current results meet or exceed desired expectations, you have identified a strength on which the organization can capitalize.
Collecting this kind of data allows you to prioritize discrepancies by examining the location of the gaps along a uniform scale
Combine both “hard” and “soft” data, as well as qualitative and quantitative data, to define gaps between the current and desired accomplishments. This will broaden your perspective and define multiple measures for each need. Hard data refers to the results of those measures that are externally verifiable; soft data are the results of those measures that cannot be externally verified.
Section Two: Strategic Planning and Alignment of Learning
The improvement of performance and the accomplishment of useful results begin with a review of the strategic goals and objectives of everyone involved in the organization’s success. That means employees, clients and customers, the suppliers who provide resources to your organization, as well as contributing or impacted community groups. Each of these partners is critical to the success of the others – and of the organization in whole.
The strategic goals and objectives of your organization, your customers, and your partners are not always the same. Given your unique needs and desires, each will likely have some distinctive goals and ambitions. Nevertheless, there will also be many goals and objectives that are shared among the organizational partners. The combination of unique and shared ambitions provides important information for defining what results are expected from any performance improvement effort.
Classroom time is not the time to identify which organizational goals you would like to achieve or how those accomplishments will be measured. Regularly revisit the needs assessments and apply those measures towards evaluation of success based on the chosen organizational goals you are trying to meet.
During the planning stage use organizational performance objectives sequentially from each level to align and define your desired results. Correspondingly, when evaluating results, you should reverse the sequence. First evaluate at the Input level, and then the process, micro, macro, and mega levels in that order.
As you move sequentially from industry goals to individual goals during the planning stages and from individual to industry goals during the implementation and evaluation stages, you will achieve the strategic goals and objectives of the each of the groups that determine your organization’s overall success.
When you align your training initiatives to your organizational goals and subsequently identify performance solutions that aren’t accomplishing desired results, review their selection, design, development, and implementation for opportunities to improve. You may find that the selected training delivery method was not the right choice, or that you made errors during implementation. It is important, however, that you evaluate each component using only the performance measures agreed upon at the beginning of the design process. If those measures have changed, then the entire process should be reviewed and possibly revised.
Solutions for Alignment – When you find discrepancies between current and desired results first see if there are any conflicts or redundancies leading to these less than desirable accomplishments. A review of performance technologies and their associated performance objectives will help you to identify any misalignments that have developed during your development and implementation processes.
Begin by reviewing your performance objectives, performance analysis, and selected training delivery method. You should especially examine the relationships between implemented performance technologies and their associated objectives. Also look for contributing factors that were not addressed during the implementation. Sometimes a breakdown in the alignment will be obvious.
Effectiveness and Efficiency – You should constantly review your training initiatives to re-affirm alignment. Programs need to change over time to reflect any changes to the organizations goals. Failure to do so may result in programs that are not producing the anticipated or desired results.
Review the plan for each training initiative for both effectiveness and efficiency. Based on your information are you aimed in the right direction? Will it achieve the results that were necessary for its success? The answers to these questions will often lead you to opportunities for improvement even before an initial investment is made.
Making Improvements – Improvements to alignment requires action. From misalignments between solutions and objectives to implementations that meet unanticipated resistance, opportunities to improve have to be acted on if useful results are going to be achieved. Revise each performance solution whenever there is evidence that you can better accomplish desired results. Use a combination of hard and soft data to verify opportunities to improve in specific areas. Prioritize improvements based on associated benefits and costs in time, resources, and people.
Create a detailed implementation plan for each training initiative and get stakeholder sign off. The scope of a plan varies; your plan should make the necessary revisions without creating $1,000 solutions to $5 problems.
As with most implementation efforts, follow your plan and monitor results regularly to make sure the changes are useful and that your training starts, and stays, aligned. Don’t assume that you will immediately see your desired results. Performance improvement is usually incremental. Continue to check your alignment and make improvements, even if you do not initially accomplish your objectives, eventually you will achieve your desired results.
Stakeholder input: Perhaps the most important step in the planning process is involving your stakeholders. Without their input, and sign-off, you will not have a verified and agreed upon methodology to proving a return through your efforts.
Interviewing your stakeholders is a simple process, and shows that you have your mind on business results. These interview meetings should take place before the development of a program begins, and should happen at an appropriate time once a training initiative ends in order to collect data on how the training was attended, the level of learning transfer, on the job behavior change, or other measurable information that the stakeholder either has interest in, or is a supplier of.
Section Three: Measuring Training Outcomes
Organizations can only improve what they can measure, and the initial focus of a training offering should be a determination of where the individual or organization stands now, versus where the organization either wants or needs to be. Establishing measures helps you fully understand where you are, provides direction in terms of where you want to go, and tells you when you have arrived.
There are a number of important reasons to establish a measurement system as you develop goals, objectives, and outcomes and choose the activities that will allow you to implement your performance improvement plan and make sound decisions about how that plan is unfolding.
Performance Feedback. By collecting data on the performance and progress of individuals, work groups, and the organization you can determine if you have made the right choices, whether or not your plans are proceeding on schedule, and when you have gotten to where you need to be.
Clarification of Direction. As you know, a mission statement provides direction but effective measures help clarify direction by establishing operational guidelines. Effective measures help employees determine if their day-to-day activities are in line with overall company direction. They help answer the question: “Is what I’m doing today contributing to our improvement effort?
Usable data. We’ve all had the experience of being overwhelmed by data. How many hours have you spent trudging through 500 page reports in order to locate data “gems” that are really helpful in your decision making? A good system provides only data that you can really use.
Measuring the Right Thing. It’s important that you are measuring the right thing. An example that leaps to mind are the training activity reports of the past, where the requirement was to report on the number of types of sessions held, the number of participants attending, the locations reached, and so on. This data allowed our manager to track our activities, but it provided no data on what, how much, and how well people actually learned in our sessions. Was the program cost-effective? Did it meet the learning needs of participant? Did it help meet the learning needs of the organization?
Including Positive Measures. Some systems tend to focus upon negative measures—return rates, reworks, complaints per month, and so on. A good system takes into account the human need for positive feedback based upon successful performance (as well as negative performance)— the number of new sales appointment set, the number of positive customer comments, and so on.
Measures That Can be Understood by Everyone. Avoid the use of technical jargon and industry acronyms. It is very frustrating for a non-technician or a non-engineer to try and understand some technical performance reports. Go for the lowest common denominator.
Measures That are Worth Collecting. When you are establishing performance measures you should take care to determine that the costs of collection and distribution are within reasonable bounds. That is, is the cost-benefit ratio favorable? There may be situations where the data must be collected but it’s wise to analyze the situation carefully and determine whether or not the resource expenditures are really justified.
Not all training will contribute useful results, at least not at first. Improvements will routinely have to be made to most training initiatives in order for desired results at all levels to be accomplished. Regularly assess and monitor the results being accomplished, and then verify that those results are aligned with desired performance objectives, as mentioned in the previous section. Measure current accomplishments against the desired results defined in each performance objective to identify new opportunities for improvement.
Use the same performance assessments you used to judge potential performance solutions. Since all decisions regarding the initial design and development of each initiative were based on the achievement of these results, it is only reasonable to use them as standards for performance during implementation.
An organization’s strategic direction can also shift in the middle of an improvement effort. If it does, performance objectives that were drivers for early decisions may have to be adjusted. Review and revise the related performance objectives for each initiative. Based on your review, decide if the selected performance technologies can accomplish the newly identified results, if current interventions can be revised, or if an alternative set of solutions should be considered.
One of the most daunting challenges in proving a return on investment is attempting to stratify results, and make general cost “input” determinations on courses versus their financial or organizational “outputs”. These are typically reserved for courses that are going to be subject to “Kirkpatrick Level Four” evaluations, though all courses should have a set of measurable data.
Determining your costs will allow you to:
- Identify the costs of existing programs
- Improve the efficiency of all programs
- More effectively project cost of future programs
- Compare the cost of using different delivery methods for the same trainings
In determining as accurate a picture as possible when attempting to deliver return on investment information there are several factors and costs that have to be given careful consideration. Basic ROI studies may simply be a matter of determining what a given output was before training versus what it is after, but that doesn’t look deep enough at the costs for most CFO’s. In order to satisfy a financial stakeholder you may also need to “peel the onion” a bit in looking at your actual outputs. Some standard training costs that should be part of your evaluation and measurement are:
- Lost productivity time of employee being trained
- Actual salary of an employee not on the job
- Instructor costs
- Classroom or electronic materials
- Facility and or technology costs
- Any associated travel
Not all training programs are subject to such deep analysis, however all training initiatives should have their own measures of cost and success. Return on investment can be as simple as determining that a course is worth investing in if there is an established alignment with a base measure of how many people attend, or how many pass a post-course assessment.
Conversely Return on Investment can be as complex as determining whether or not a sales training program has given the organization a return when also needing to consider that the improvement may have been partially due to a product improvement, new phone system, territory shift, or other affect. Involving your stakeholders and other contributors early and getting sign-off is key when attempting to make such determinations.
Section Four: Reporting to Stakeholders
When all of the hard work is done you’ll be left with one last critical step – proving the effectiveness of your training to your stakeholders. In order to keep yourself accountable and also make it as simple as you can for those who will be viewing the reporting data we recommend creating a “metric dashboard” of results to your stakeholders that can be accessed instantaneously. These dashboards contain vital information regarding your work specifically designed to show the results that your training programs have promised.
A metrics dashboard is a term used to refer to an illustration of the Key Performance Indicators, those measures you have identified from the earlier steps, in a single glance—a one-page document or set of documents that helps determine the effective return of your programs. The specifically aligned goals and the numerical measures of how they are tied to executables for the organization determine what should be on the “dashboard.” Each metric on the dashboard should be supported by other metrics.
Here is how it works: Let’s say your company needs reporting on how many classes are offered, and how many employees are attending, as part of an agreed upon ROI methodology. Obviously, in this case the key performance metrics are going to be found in measures that speak to those organizational and/or department goals. This should be on the dashboards of the CLO, COO, CFO, CEO, or any other stakeholder that you have worked with or needs to see the results for their own purposes.
There may be other times when you are asked to prepare other, deeper methods of reporting results of your training efforts. An evaluation report can be a more effective way to display and share results, especially considering that you will have more time – and more space – to share anecdotal and other summary information than you can on a dashboard.
Standard evaluation reports may include information such as:
The methods you used to collect information
The rationale for the dollar amounts you are citing
The limitations of your reporting mechanisms
The beginning metrics being used, and any available post-training measure
These reports should include charts and graphs when possible, and be as inclusive as you can make them. You should also include summaries, recommendations, planned changes, and any other information that will support the data being provided.
Always check your report several time for accuracy, and get another set of eyes to assist you with the basics of grammar, spelling, etc. We also recommend that you leave any large surprises out of the report – these should never occur, as your stakeholders are kept abreast of your progress and any issues as training is being developed and implemented. Head-turning surprises are better discussed in person by calling a meeting than they are simply listed out on an evaluation form.
When training has worked, it has contributed something of value to the organization. Training begins as a response to a need or opportunity in an organization. Aligning your learning and establishing the value of training brings your efforts “full circle” and ensure that your efforts are indeed making a contribution, and also brings it into the light of being recognized as more than an expense.
When each training programs begins with a specific end in mind, rather than simply being a reaction to whim, you have an incredible opportunity to showcase your training department as a critical piece of your organization’s success.
By taking the steps to proving ROI, not only does your organization view the initiative differently, but a positive side effect is that it will nearly always improve the training initiatives themselves.
Have a question about one of your programs? Ready to begin the process of building a new program? Click on the button below to view our capabilities, and begin the process of improving your efforts today!