How Do You Know Cybersecurity Training Changes Job Performance?

Cybersecurity training has credible evidence of workplace value when people perform relevant tasks better, retain that capability beyond the course, and can use it under their normal working conditions. To claim that training caused the improvement, you also need a defensible comparison that addresses changes in the people, tasks, tools and environment. Completion records establish participation. They don't establish that causal chain.

I would begin with the decision the evidence must support. Are you deciding whether to renew a course, redesign a reporting process, give analysts more practice or demonstrate that a critical team can work safely? Those decisions require different observations. A single awareness score can conceal the distinction between a person who needs instruction and a person who understands the task but cannot perform it with the available permissions.

This paper develops a practical evaluation approach for CISOs, learning leaders and executives. It brings together original research, methodological guidance and my proposed decision framework. No customer study or training experiment was conducted for this article. The worked comparison is fictional. The framework is an analytical proposal, not a validated instrument or a claim that every organization needs a formal research trial.

Define the task before selecting the training metric

Start with a task whose successful execution matters. For a service-desk employee, that might be verifying a sensitive account-recovery request through the approved process. For a developer, it might be diagnosing why a change exposes a secret and repairing the cause. For an executive, it might be distinguishing confirmed incident facts from assumptions before authorizing a disruptive action. These are proposed examples, not findings about any organization.

The NICE Framework separates tasks from the knowledge and skills that support them (Petersen et al., 2020, §§2.1–2.3). I would use that distinction to ask what must be observed. Knowing the policy, explaining the reason for a safeguard and applying the safeguard under conflicting demands are related but different outcomes. An assessment should not silently substitute the easiest one to collect for the one the business needs.

Write the expected performance in ordinary operational terms: given a request with an ambiguous identity signal, use the approved independent verification route, preserve the relevant record and escalate unresolved uncertainty. Define what would count as an unacceptable outcome, including a correct final answer reached through an unsafe process. Otherwise, a lucky guess can receive the same score as a repeatable method.

Keep this task definition separate from job titles and broad competency labels. My workforce task analysis addresses the staffing question. Here, the narrower question is whether a particular learning intervention changes performance on selected tasks. Select a manageable set of consequential tasks before purchasing a larger catalogue of courses.

Separate learning, transfer and organizational outcomes

NIST's learning-program guidance distinguishes reactions, learning, workplace behavior and organizational results, and recommends assessments aligned with learning objectives (Merritt et al., 2024, §5.2.3.3). That vocabulary helps organize evidence. It does not make the categories a causal staircase: a positive course survey does not prove learning, and a higher knowledge score does not prove that a later reduction in incidents came from the course.

I would retain three separate questions in the evaluation brief. Did participants acquire the targeted capability? Did they apply it when relevant work occurred? Did that application improve the operational result? A person may learn a verification method yet have no occasion to use it during the observation period. Another may use the method correctly while a downstream queue delays the response. Neither situation is adequately described by a single pass rate.

The observation window must fit the claim. An immediate practical assessment can show what participants could do at the end of instruction. A delayed assessment investigates maintenance. A sample of genuine work products investigates application in the workplace, subject to confidentiality and selection limits. Each has value without impersonating the others.

Salas et al. (2012, pp. 90–92) argue for evaluation tied to its purpose and precise learning outcomes, while explaining shortcomings of treating evaluation levels as a checklist. Their review also emphasizes support after training. My interpretation is that evaluation should identify a decision you can change: the instruction, practice opportunities, job aids, supervisory support or the work process. Counting more categories is not enough.

Design an assessment that cannot be passed by memorizing the example

Suppose a course teaches staff to verify an unusual payment-change request. Repeating the same names, wording and suspicious attachment in the final test may assess recognition of the lesson. A more useful assessment changes surface details while preserving the underlying decision. Include legitimate requests as well as malicious or ambiguous ones, so that rejecting everything cannot masquerade as good judgment.

For technical roles, retain the reasoning and the work product. A participant who repairs one deliberately planted configuration issue should also explain the relevant trust boundary and check the effect of the change. The assessment can then distinguish diagnosis from copying a command. It should allow legitimate alternative solutions and the reference materials normally available on the job.

I would specify critical errors separately from a total score. A high aggregate result should not conceal a dangerous step, such as bypassing identity verification to finish faster. At the same time, a rubric should distinguish a harmless variation from a material failure. Have knowledgeable reviewers score a small common sample and resolve disagreements before interpreting differences between groups.

This is a design proposal, not a claim that an elaborate simulation is always necessary. A carefully chosen work sample can be enough for a narrow task. The important constraint is that the task, scoring rule and allowable conclusion stay aligned. An easy-to-administer quiz is useful for knowledge; it becomes misleading when its score is relabeled as demonstrated operational competence.

Keep difficulty and opportunity in the denominator

A declining phishing click rate is ambiguous if the later message is easier to recognize. NIST's Phish Scale considers observable cues and how closely the email premise fits the recipient's context (Dawkins & Jacobs, 2023, §2). A message about a process someone actually uses can present a different detection challenge from an irrelevant offer. Difficulty belongs in the interpretation of the result.

For repeated assessments, document the task version, relevant context and intended challenge. Reusing an identical item invites familiarity; changing it without recording difficulty invites a different confound. I would use varied tasks designed around the same capability, review their comparability and avoid presenting them as psychometrically equivalent merely because their descriptions sound similar.

The denominator also matters. A reporting rate among delivered messages differs from a rate among people who actually encountered the message. If exposure cannot be measured reliably, name the available denominator and its limit. Similarly, an employee with no relevant work opportunities should not be recorded as failing to apply training. Record the absence of opportunity separately from an observed unsuccessful attempt.

Balance protective actions against unnecessary disruption. Report both unsafe acceptance and inappropriate rejection of legitimate work. A process that makes every request wait indefinitely could reduce one error while creating another. For an operational decision, I would examine severity, reversibility and workload alongside counts, rather than reward a single metric that can improve by shifting costs elsewhere.

Test retention without pretending that testing is neutral

Roediger and Karpicke's (2006) experiments with students learning prose passages found that repeated study could outperform testing on immediate recall, while testing favored delayed retention. The experiments were not cybersecurity workplace studies. Their narrower relevance is methodological: the timing and form of assessment can change what appears effective, and a test can itself be a learning activity.

That creates a practical choice. If you want to evaluate a whole program that deliberately includes retrieval practice and reminders, include those components in the intervention definition. If you want to isolate the effect of a course, repeatedly rehearsing the assessment in one group introduces another explanation. Neither design is inherently wrong; the reported claim must identify what was actually compared.

I would plan an immediate assessment and a later task sample, with timing justified by how often the capability is needed and the consequences of decay. A rare emergency task may need scheduled practice because ordinary work provides little reinforcement. A frequently performed task may be better assessed through appropriately governed work samples. There is no universal reminder interval established by these examples.

Reinheimer et al. (2020, §§6–7) found improved phishing identification directly after instruction and at four months, but no statistically significant improvement over the pretraining group at six months. Their task used email screenshots, with security as the primary goal, in one German public-sector organization. The authors explicitly limit inference to ordinary inbox behavior. That evidence supports examining retention locally; it does not establish that every organization should retrain every six months.

Make the causal question explicit

“Performance improved after training” and “training improved performance” are different claims. The first describes a sequence. The second excludes plausible alternative explanations. During the same period, a team may gain experience, adopt a safer interface, receive clearer procedures or handle simpler cases. Selecting only weak initial performers also creates a risk that unusually poor starting results will look better on repetition without a course effect.

The What Works Clearinghouse handbook addresses comparison-group formation, confounding, attrition and baseline equivalence in educational effectiveness studies (What Works Clearinghouse [WWC], 2022, Chapters II–III). Its standards are not a cybersecurity compliance requirement. I borrow the questions, not its formal ratings: what differed between the groups, what changed after assignment, and could those differences explain the result?

Where feasible and appropriate, compare alternative training approaches through a planned rollout with a common assessment. Random assignment can strengthen the design, but it does not rescue unclear outcomes, substantial missing data or a simultaneous change delivered only to one group. A comparison between two departments may confound the course with their supervisors, workloads and existing expertise.

Where randomization is impractical, retain baseline measures, relevant contextual differences and repeated observations, and state the remaining uncertainty. A matched comparison can be more informative than no comparison without becoming equivalent to a randomized study. Urgent protective instruction should not be withheld merely to improve a measurement design. You can deliver the necessary protection and make a narrower claim about the evidence available.

Read a before-and-after result without overselling it

Consider this fictional illustration. Two groups each complete 100 scored task opportunities before and after a training rollout. The trained group improves from 60 acceptable performances to 80. A comparison group improves from 65 to 75. Assume for the arithmetic that all outcomes are observed and the tasks are comparable; those assumptions would need investigation in a real evaluation.

Group	Before	After	Change
Training group	60/100 acceptable	80/100 acceptable	+20 percentage points
Comparison group	65/100 acceptable	75/100 acceptable	+10 percentage points
Difference in changes	—	—	+10 percentage points

The trained group's twenty-point increase is not, by itself, the estimated training contribution. Subtracting the comparison group's ten-point change gives a ten-point difference in changes. This calculation illustrates a comparison, not a demonstrated causal effect. It depends on whether the comparison group's change represents what would have happened to the trained group without the intervention.

One baseline observation cannot establish that their underlying trends were comparable. Perhaps the trained group also received a new verification screen, or its supervisor changed the work allocation. Perhaps the comparison group shared the training materials. Those possibilities change the interpretation even though the arithmetic is correct. Multiple prior observations and a record of concurrent changes can help investigate them without proving every assumption.

The 100 opportunities are also not necessarily 100 independent people. Repeated attempts by the same employees and shared team conditions can make a simple independence-based uncertainty estimate inappropriate. A real analysis should respect those groupings and report uncertainty using a justified method. This fictional table supplies no confidence interval, significance claim or promised effect size.

Follow the people who disappear from the result

A delayed assessment can look stronger because the least confident participants did not return. The WWC handbook treats missing outcome data and changes in group composition as threats to interpretation, including after randomized assignment (WWC, 2022, Chapter III). Missingness is not repaired by displaying the completion percentage beside a polished chart.

Retain a participant-flow record appropriate to the evaluation: eligible population, assignment or invitation, participation, assessment completion and available follow-up. Explain exclusions made before analysis. If the course was offered broadly but the report describes only enthusiastic completers, label that scope. An analysis of the offer and an analysis of actual participation answer different questions.

For missing follow-up, investigate reasons where this can be done fairly and proportionately. Staff turnover, leave, workload and technical access failures have different implications. Show whether missingness differs across groups and whether the conclusion depends heavily on assumptions about the missing outcomes. Do not silently code every missing result as success, failure or unchanged performance.

Small teams need particular restraint. Individual cases can help diagnose process problems, but a handful of observed successes cannot establish a stable organization-wide effect. I would retain the qualitative evidence, report the limited coverage and use it to choose the next observation. The honest result may be that the task design improved and promising performance was observed, while the training effect remains uncertain.

Diagnose the workplace conditions that permit transfer

Blume et al.'s (2010) meta-analysis associated training transfer with several individual and work-environment factors. It also found inflated relationships when predictors and transfer were measured by the same source in the same context; some findings rested on small subsets of studies, and repeated field measurements were limited (pp. 1090–1097). That is evidence against relying on one enthusiastic self-report as the whole evaluation, not a causal estimate for your team.

I would pair reported confidence with a relevant work sample and a review of the conditions needed to use the skill. Does the employee have the necessary access? Is the approved verification channel available during their shift? Does the manager allow time for the process? Is the policy clear enough to resolve an exception? The answers identify different interventions.

Consider an analyst who correctly explains how to preserve evidence but cannot access the approved storage location. More content may raise their test score without fixing the operational failure. Conversely, a functioning tool does not show that the analyst understands when preservation is required. Assessing the person and the environment separately helps avoid spending the training budget on a permissions problem.

Self-report still has a role. It can reveal confusion, perceived obstacles and unobserved workarounds that a task score misses. I would treat those reports as leads to investigate and as evidence of the participant's experience. They should not be dismissed, nor should they be converted automatically into proof of competent execution or causal improvement.

Distinguish instruction from a redesigned interface

Lain et al. (2025) evaluated active URL-inspection tasks in online studies involving 2,673 participants. The interventions improved phishing detection relative to their comparison conditions, but the authors identify limits including the single-session setting, selected URLs and population characteristics (§8). This is evidence about those interventions under those conditions, not proof of durable employee learning or universal protection against malicious links.

The distinction matters when a company introduces training and a new workflow together. If the interface now requires an independent verification step, improved outcomes may result from the constraint, from learning, or from their combination. If the business objective is safer work, the combined improvement may be valuable. It simply should not be sold internally as the isolated return from the course.

I would document the intervention as people actually received it: instruction, practice, feedback, job aids, interface prompts, managerial reinforcement and process changes. If the procurement question concerns one component, design a comparison capable of separating it where feasible. If the decision concerns the complete program, evaluate the package and identify which mechanisms remain uncertain.

Do not deliberately remove a necessary safeguard from production to see whether staff can compensate. Assess unaided understanding safely where that matters, and assess supported performance under the conditions in which the work should happen. The useful question is whether the combination produces dependable decisions, with clear limits on what would happen if an element became unavailable.

Treat contradictory phishing findings as a design problem

Lain et al. (2022, §VI.B) found that their voluntary contextual training, delivered after participants fell for simulated phishing, did not improve subsequent detection and was associated with worse outcomes. The authors explicitly state that they did not test mandatory training. Their long-running workplace design offers a different kind of evidence from a short, focused identification assessment.

I would not reconcile that result with the Reinheimer study by declaring one correct and ignoring the other. The interventions, attention conditions, measurements and settings differ. A task performed with security as the primary goal can show an acquired discrimination skill while leaving unanswered whether it will be used during routine work. A voluntary page displayed after a mistake may be encountered without meaningful engagement.

For a local program, define the proposed mechanism before selecting the evidence. If the mechanism is better recognition of suspicious features, assess discrimination on unfamiliar examples. If it is faster reporting, examine whether the report reaches the right team with useful context. If it is safer account recovery, examine verification decisions. A click-only dashboard cannot establish all three.

The NCSC describes phishing defense as multiple layers and warns against expecting training to enable everyone to identify every phishing email (National Cyber Security Centre [NCSC], 2024). My implication is to evaluate learning alongside preventive and response controls. A training success should not become a reason to remove those controls, and a training failure should not become an excuse to blame employees for the entire defense.

Include harms, workload and fairness in the evaluation

Schöps et al. (2024) found higher reported stress and lower phishing self-efficacy among respondents who clicked than among those who reported a simulated email. Their study had no before-and-after measurement, a limited response sample and one organizational setting (§3.5). Those limitations prevent treating the group difference as a clean causal estimate of campaign harm. They still make participant experience a legitimate evaluation concern.

I would measure whether the program creates avoidable interruption, excessive reporting load or reluctance to ask for help. A protective behavior that overwhelms the receiving team can fail operationally despite a higher reporting count. Equally, declining reports might reflect fewer suspicious messages, reporting friction or silence after a punitive experience. The count alone cannot decide among those explanations.

Keep evaluation proportionate and explain its purpose. Collect the minimum information needed, restrict detailed access, set retention expectations and separate learning support from disciplinary decisions. Use synthetic material for assessments where possible. Avoid embedding sensitive personal events in deceptive exercises merely to increase the challenge. These are proposed governance choices, not a legal opinion about a particular workplace.

Check whether the assessment unintentionally measures language fluency, inaccessible interaction design or familiarity with a device rather than the intended security capability. Accommodations should preserve the relevant task while removing irrelevant barriers. If a result may affect an individual's role, provide a review route and sufficient contextual evidence. A single unexplained score is a poor basis for a consequential judgment.

Decide whether to retain, redesign or replace the intervention

My proposed decision framework uses four questions in sequence. Is the assessment relevant and interpretable? Was capability acquired and maintained? Was there a fair opportunity to apply it? Is there credible evidence that the intervention contributed to the desired outcome? A weak answer early in the sequence limits what later numbers can support. This is a reasoning aid, not a composite maturity score.

If assessment validity is weak, repair the measurement before declaring the course effective or useless. If learning is weak, examine the instruction and practice. If learning is demonstrated but application fails, examine the work environment. If performance improves while causal attribution remains uncertain, retain the bounded finding and choose a better comparison for the next decision. Do not erase useful progress simply because it falls short of causal proof.

An executive report should state the task, population, intervention, observation period, comparison, result and remaining uncertainty. Include the operational cost and the decision requested. Avoid translating a test-score change into avoided breach losses without a defensible model and relevant inputs. A credible report may justify a targeted redesign even when it cannot produce a monetary return estimate.

The evidence base has limits. Much of the cybersecurity research examined here concerns phishing; it cannot validate every conclusion about administrator, developer or executive training. Cross-domain learning research helps frame mechanisms and measurement, but local transfer still requires investigation. Rare incident outcomes, changing work systems and small cohorts further constrain attribution. Those limits should guide the size of the claim and the next observation.

If your training report shows attendance and quiz scores but cannot answer what changed at work, I can help scope an assessment around your most consequential tasks. The useful deliverables are a task-specific evidence plan, interpretable assessments, a record of workplace barriers and a decision brief on what to retain or redesign. Start a conversation about evaluating your security-learning program before committing to another cycle of activity that you cannot meaningfully assess.