Most advice about course evaluation is advice about tools. Which survey platform. Which analytics dashboard. Which LMS report to run. The tooling is not the problem. The problem is where evaluation stops, and there is data on exactly where that happens.
ATD Research surveyed 779 talent development professionals about their evaluation practice. Roughly eight in ten organizations evaluate at Level 1, reaction, and Level 2, learning. Just over half reach Level 3 and ask whether anyone applied it. Fewer than four in ten reach Level 4 and connect it to a business measure. Sixteen percent reach Level 5 and put a financial figure on it.
That survey ran in 2019, and ATD noted the numbers had barely moved since 2015. Treat it as the most recent published breakdown rather than a current-year reading. The shape of it has not been seriously challenged since.
This post is about that shape. Why evaluation stops where it does, what it costs you, and what changes if you plan the measurement before you build the program rather than after.
Where Evaluation Actually Stops
The four-level model is over sixty years old. Donald Kirkpatrick first published it in 1959, and it survives because its central structure is correct: there are different grades of evidence, and they are not interchangeable. Reaction is the cheapest to collect and the weakest thing to know. Behavior change and business results are expensive and they are what anyone outside L&D actually asked about.
The ATD numbers show the field clustered at the cheap end. More telling than the ladder itself is what practitioners said about the value they got. Only half said their evaluation work helped them meet their organization’s learning goals. Only 40 percent said it helped meet business goals. So most evaluation activity fails at the purpose it was funded for, whichever level it reaches.
Why the Drop Happens Where It Does
The same survey asked what gets in the way. Three barriers dominate, and each one fails at a different point.
The data belongs to someone else
Thirty-nine percent said their team does not have access to the data needed for higher-level evaluation. This is the barrier people underestimate, because it is not a skills problem or a budget problem. Performance data sits with sales operations, quality, compliance or HR analytics. Getting it takes a relationship and a reason, not a tool.
Level 3 and Level 4 evaluation are therefore partly a stakeholder problem. If you cannot see the number, you cannot move it, and nobody hands over a dashboard to a team that has not explained what it plans to do with it.
Isolating the program’s effect is genuinely hard
Forty-one percent, the largest group, said they struggle to isolate the training’s influence on results. That difficulty is real and it deserves more respect than it usually gets in L&D writing.
A sales number moves because of training, coaching, a comp change, a product launch, a competitor stumble, or the weather in a key territory. Claiming the training did it is an overclaim, and experienced business leaders recognize an overclaim immediately.
The honest response is not to abandon Level 4. It is to separate the measures you control from the measures you contribute to, and to say which is which before anyone asks. Jack and Patricia Phillips built an entire methodology around isolating programme effects, and their work on forecasting addresses the same question before the money is spent rather than after.
Cost, and what it actually buys
A third said higher-level evaluation costs too much. Sometimes that is accurate. Full Level 4 work on every program would bankrupt the function and produce very little.
The useful question is not whether to evaluate everything at Level 4. It is which two or three programs carry enough organizational visibility to justify it, and what the minimum defensible claim looks like for everything else.
Why Level 3 Is the One That Matters
Level 4 gets the attention because it speaks the business’s language. Level 3 is where programs actually fail, and it is the level the ATD data shows nearly half of teams skipping.
Blume, Ford, Baldwin and Huang meta-analysed 89 studies of training transfer and confirmed what most practitioners suspect: transfer depends heavily on the work environment, on supervisor support and on whether learners get an early opportunity to apply what they learned. They also found that the longer the gap between training and application, the weaker the transfer.
None of that is visible at Level 2. A learner can pass your assessment and never use the skill, and completion and score data will look identical either way.
There is a caution worth carrying into any Level 3 work. Will Thalheimer, reviewing the same literature, estimates that well over 80 percent of transfer studies measure learners’ perceptions of transfer rather than transfer itself. Blume and colleagues noted that because of this, many transfer findings have likely been overstated. So when you design Level 3 measurement, a follow-up survey asking people whether they applied the training is the weakest available instrument, and it is the one most teams reach for first. Manager observation, workflow data and before-and-after measures are harder and worth the difference.
The Fix Is Sequencing, Not Tooling
Every barrier above gets worse when evaluation is bolted on after delivery, and most of them soften when it is designed in from the start.
That is the practical contribution of the New World Kirkpatrick Model, published by James and Wendy Kirkpatrick in 2016. It inverts the planning order. Instead of running a program and then working up the ladder from reaction, you begin with the Level 4 result the business wants, identify the leading indicators that would show movement toward it, then design backward through behavior, learning and reaction.
The difference is not academic. Planning forward produces the data the LMS happens to generate. Planning backward produces the data someone asked for.
In practice this is one conversation, held before design starts. Thirty to sixty minutes with the stakeholder and the designer, answering a single question: what evidence would tell us this worked? The output is two or three observable outcomes. Most programs will never reach a full Level 4 evaluation. Every program can carry one observable outcome beyond completion.
What the Regulated Case Already Gives You
If you work in financial services or insurance, part of this is already done for you, and almost nobody reports it.
In most states, earning insurance continuing education credit through self-study requires a passing examination score, typically 70 percent. Read that as an evaluator rather than a compliance officer. A mandated passing score is Level 2 evidence. The regulator is not asking whether the agent attended. It is asking whether the agent knows the material, and it set a threshold. That data is already being collected, stored and audited, and it almost never appears in an L&D report.
FINRA works differently and is worth separating rather than lumping together. Rule 1240(b)(2)(A) requires each member firm to maintain a continuing and current education programme, to evaluate and prioritise its training needs at least annually, and to develop a written training plan that accounts for regulatory developments. That is a needs analysis obligation written into a rule. Rule 3110 Supplementary Material .04 goes further on evidence, specifying that an on-demand annual compliance meeting must track time spent, capture confirmation as the learner goes, and end in an attestation of completion.
Notice what the regulator specifies and what it does not. It is exhaustive about how completion gets evidenced. It says nothing about whether the meeting changed anyone’s behavior. That gap is the whole argument of this post, written by someone other than me.
Choosing Tools, Briefly
Tool selection matters far less than sequence, so it belongs at the end rather than the beginning. Your LMS produces completion and assessment data without effort, which is why most reporting stops there. Survey platforms handle reaction and self-reported application. Learning analytics platforms connect activity to patterns across cohorts. Performance data for Level 4 almost never lives in any of them, which returns you to the access problem above.
Pick the tool after you know the question. A team with a clear evaluation question and a spreadsheet will produce better evidence than a team with an analytics licence and no question.
The One Change Worth Making
If you take a single thing from this post, take the sequencing. Have the evidence conversation before design begins, not after delivery ends. Ask what would tell us this worked. Write down two or three observable outcomes. Name which of them you control and which you only contribute to. Then build the program so those outcomes are collectable rather than hoping the LMS produces something usable at the end.
That will not get every program to Level 4. It will stop you reporting Level 1 data as though it answered a Level 4 question, which is the failure that costs L&D credibility.
The instrument that produces most of this evidence is the assessment itself, and not every assessment is capable of producing it. I have written separately about which assessment types actually support which claims, including the one widely cited benefit that the research does not support.
If you are working through what a defensible measurement approach looks like for your programmes, the Work With Me page is where that conversation starts. The AI Readiness Checklist includes a measurement readiness component if you would rather begin with a self-assessment.
Editor’s Note: This post was originally published in October 2023 and has been completely revamped and updated for accuracy and comprehensiveness. in September 2026.
