How Do You Know a Cyber Tabletop Found a Real Response Gap?

A tabletop has identified a credible response gap when you can connect an observable problem to an agreed objective, show the conditions under which it occurred, and distinguish an organizational weakness from a flaw in the exercise. Closing that gap requires evidence that the relevant capability changed. Attendance, a lively discussion and a completed action list don't establish either conclusion.

I would start with the claim you need to make afterward. Can the team identify who may approve disruptive containment? Can it produce a useful executive decision brief from incomplete facts? Can it perform a technical action? Those are different capabilities. A discussion can expose uncertainty about the first two and produce documents you can examine. It cannot, on its own, demonstrate the third.

This paper develops an evaluation method for security leaders, exercise planners and executives who need defensible findings. It combines exercise guidance, original research and analytical examples. No exercise or customer assessment was conducted for this article. The proposed evidence categories and closure method are my synthesis, not a validated readiness score. The aim is to make the limits of an exercise useful rather than hide them behind a successful-event label.

Decide what kind of evidence the exercise can produce

NIST SP 800-84 distinguishes discussion-based tabletop exercises from functional exercises in which participants perform duties in a simulated operational environment. Its tabletop definition does not involve deploying equipment or other resources (Grance et al., 2006). That older distinction remains useful here: discussing a task and executing it generate different evidence. A mixed event can include both, but its report should identify which observation came from which activity.

Suppose a participant says, “I would revoke the affected sessions.” That supports a limited observation about the proposed response. It doesn't establish that the participant has the permission, that the identity system is reachable, that the selected control affects the relevant sessions, or that the result can be verified. Calling the statement “successful containment” would enlarge the conclusion without adding evidence.

Some things can be demonstrated within a tabletop. Participants can interpret the information supplied, locate an approved procedure, identify a decision authority, challenge an unsupported assumption or prepare a briefing. If producing that artifact is an objective, evaluators can inspect it. The limitation is not that discussion has no value. It is that the claim must match the activity and the conditions in which it occurred.

I would therefore classify observations by what actually happened: a participant described an action; the group produced a usable artifact; an authorized action was executed in a defined environment; or an effect was independently observed. These are evidence descriptions, not a universal maturity ladder. A carefully assessed decision brief may be exactly what an executive exercise needs, while an unrelated technical demonstration may add nothing to its objective.

NIST SP 800-34 similarly separates training people, exercising plan content and testing system components for operability in the specified environment (Swanson et al., 2010, §3.5). My implication is to reserve technical capability claims for evidence that actually reaches the mechanism. If you need to establish that recovery works, use the separate recovery acceptance analysis to define the technical observations beyond the tabletop.

Define the decision before writing the attack story

An objective such as “improve ransomware readiness” doesn't tell an evaluator what to observe. I would narrow it to a consequential task and its conditions: given uncertain evidence of an identity compromise and an unavailable primary approver, identify the delegated authority and produce an actionable containment request. The exercise can then expose a missing delegation, an unusable procedure or a failure to communicate the required scope.

ENISA’s 2026 exercise methodology connects objectives to indicators, metrics and data-collection methods, with evaluation strategy prepared before execution (Zacharis et al., 2026, §3.2). I use that structure to avoid deciding what success meant after hearing the participants' answers. A criterion can be qualitative and still be explicit. “The brief distinguishes confirmed facts from assumptions and names the decision required” is more assessable than “the briefing was good.”

For each objective, define the starting information, permitted resources, expected observable output and material exclusions. If participants may consult their existing plans, say so. If the supplier is simulated, say what the simulation can answer. If the exercise won't test technical access, state that boundary before anyone claims that the response path worked. These choices determine what the result can legitimately support.

The criterion should admit more than one defensible response when uncertainty permits alternatives. A team might isolate one service immediately or temporarily restrict access while collecting additional evidence. I would assess whether the decision was authorized, proportionate to the supplied facts and accompanied by a verification plan. An answer key that rewards only the facilitator's preferred option can confuse strategic disagreement with incompetence.

The UK National Cyber Security Centre advises agreeing exercise metrics and warns that poorly focused measures can create bias and false lessons. It also distinguishes evaluation of participants from feedback about the exercise itself (National Cyber Security Centre [NCSC], 2020, Steps 5 and 9). That separation is essential: a confusing inject is a design problem until you have evidence that it also exposed a genuine operational dependency.

Build a scenario that can challenge the assumption fairly

Realism is not the number of technical acronyms in the scenario. I would make the important constraints resemble the organization’s actual work: incomplete visibility, distributed authority, a supplier handoff or a business process that cannot simply stop. The scenario needs enough accuracy to make the target decision meaningful. Decorative detail that doesn't affect that decision consumes attention without strengthening the inference.

Information should arrive through a defined sequence. Record which participant received which inject, when it was available and whether it was acknowledged. If a team is evaluated on escalating a threat it was never told about, the result describes an exercise-delivery failure. If information reached the designated channel but nobody monitored that channel, the observation may expose a different problem. Those explanations require different corrective actions.

I would avoid quietly giving the most capable participant the missing fact and then judging the rest of the group against that person's knowledge. Unequal information can be deliberate when testing handoffs, but it must be part of the design. Evaluators need to know whether the objective concerns individual reasoning, team information sharing or the procedure that connects teams. Otherwise they may attribute a system-level failure to the wrong person.

CISA’s exercise planner handbook separates planner material from what players should receive and describes distinct supporting materials for facilitation, evaluation and feedback (Cybersecurity and Infrastructure Security Agency [CISA], n.d.). The useful principle is role separation. Planners need the scenario's hidden structure; participants need their authorized exercise information. Advance access to the answer can make a response look stronger while removing the uncertainty the exercise was meant to examine.

An exercise should also have a safe boundary. Label simulated communications, define stop conditions and prevent scenario prompts from becoming accidental real-world instructions. If the event includes technical actions, obtain specific authorization for their systems, scope and effects. NIST’s technical-assessment guidance provides a rules-of-engagement structure covering scope, limitations and risks (Scarfone et al., 2008, Appendix B). A tabletop invitation is not authorization to disable production identities or contact customers as though an incident were real.

Observe the chain between information and action

The core evidence is a trace, not a facilitator's overall impression. For a consequential decision, I would retain the relevant inject, the participant's interpretation, the requested or approved action, the resulting artifact and the assistance provided. This need not mean recording every conversation. It means preserving enough context to distinguish what the team inferred from what the exercise supplied.

My proposed evidence table keeps the observation and the allowable conclusion together. It is an analytical aid, not an official standard or a scoring system.

Observation retained	What it can support	What remains unestablished
Participant describes contacting a provider	The proposed escalation route was identified	Contact information, entitlement and provider response actually work
Team retrieves the current delegation and drafts a bounded request	Decision authority and request content were demonstrated under exercise conditions	Technical permission and execution in the service
Approved isolated test action produces a recorded control result	The specified action executed in that test environment	Wider production coverage or the absence of alternate access paths
Separate observation confirms the intended effect	The tested effect occurred within the stated scope	Persistence of the result after configuration or dependency changes
Team completes a corrected scenario with recorded assistance	Performance with that assistance	Unassisted performance or transfer to a different situation

A blank row should remain blank. If the evaluator did not observe the provider handoff, the report should not infer success from the team's later statement that it would have been easy. Conversely, failure to observe an action isn't automatically proof that the action failed. “Not observed,” “not attempted” and “attempted without the expected result” carry different evidentiary meanings.

For cross-organizational actions, record both sides of the boundary. A customer can formulate a valid request while the provider's response remains simulated. That is useful evidence about the request, with an explicit limit. The cloud incident responsibility analysis explains the underlying authority problem; this paper asks what an exercise actually established about that arrangement.

Protect the record as operationally sensitive material. Use synthetic identifiers and sanitized artifacts where possible, restrict access to detailed findings and avoid collecting credentials or unnecessary personal data. The report needs enough specificity to reproduce the reasoning, but it doesn't need a public list of weaknesses or individual participants' private information. Evidence quality and privacy are compatible when collection is designed around the claim.

Treat facilitator help as part of the result

Facilitation is not inherently contamination. A learning exercise may appropriately pause, explain an unfamiliar process and let the group try again. The problem arises when the resulting assisted performance is reported as what the team could already do unaided. I would distinguish explanation of the exercise rules from a hint that supplies an operational answer.

Consider a fictional decision exercise. At minute zero, the group receives evidence of an abused privileged identity. At minute eight, it has identified a possible containment action but cannot locate delegated approval. At minute ten, the facilitator names the person holding emergency authority. At minute sixteen, the group submits a coherent request. The observation is a sixteen-minute exercise response that included a decisive prompt. It is not a measured sixteen-minute production containment capability.

That sequence produces two useful findings. The group could formulate the request after authority was supplied. It could not demonstrate locating that authority beforehand under the exercise conditions. The facilitator should preserve the unresolved issue while continuing the learning activity. There is no need to humiliate participants or stop every discussion at its first mistake to retain an honest result.

The exact meaning of a prompt depends on the objective. If the objective is to evaluate request quality after receiving an authorization, supplying that authorization may be normal scenario input. If the objective is to find the authorized decision-maker, the same information answers the test. Evaluators need this distinction in advance; otherwise identical assistance may be scored differently across teams.

After teaching the missing process, use a later, appropriately varied scenario to assess whether the capability is available without the same help. Repeating the exact question immediately can show that the group understood the correction, but it also benefits from memory and practice. I would record that result as a learning check rather than evidence of durable transfer to an unfamiliar incident.

Separate elapsed time from the capability being measured

A stopwatch can make weak evaluation look precise. Time-to-decision requires a defined start: delivery of an inject, acknowledgment by the intended role or availability of sufficient information under the exercise rules. Time-to-action requires a defined end. A spoken intention, an approved request and a verified technical effect are different endpoints. I would not combine them under one response-time metric.

Keep exercise time and wall-clock time separate when the scenario compresses hours into minutes. If the facilitator advances the story overnight, that doesn't measure the duration of an overnight operational process. Pauses for teaching, breaks and technical problems should be recorded with the stated treatment used in any reported interval. Otherwise comparisons between runs can reflect clock conventions rather than capability.

Unfinished tasks belong in the result. Reporting the average of only completed escalations can hide the one critical escalation that never happened. With a small exercise, I would usually show the individual consequential task outcomes and their conditions instead of a headline average. A result such as “authority remained unresolved when the exercise ended” is often more decision-relevant than a polished number.

Quality needs its own assessment. A rapid request to isolate an entire customer environment may be less useful than a slightly slower request that defines the affected scope, business impact and verification step. Speed matters only in relation to the task and consequences. There is no universal number of minutes in the cited guidance that makes a private organization’s containment decision acceptable.

Recovery illustrates the distinction. NIST SP 800-184 discusses dependencies, personnel, alternate communications and service arrangements as recovery-planning concerns (Bartock et al., 2016, §2). A tabletop can uncover an unowned dependency in that chain. It cannot measure restoration throughput merely by advancing the scenario to “systems restored.” The next test should target the newly discovered uncertainty rather than reuse the fictional elapsed time as a recovery benchmark.

Diagnose the gap without inventing its cause

An observation and its explanation should be separate entries. “No approved delegation was located during the decision” is an observation. “The organization lacks a delegation” is a hypothesis until the relevant policy and access path have been checked. The policy may exist but be inaccessible, outdated, ambiguous or unknown to the role expected to use it. Each explanation points to a different intervention.

FEMA's Homeland Security Exercise and Evaluation Program connects evaluation to capability targets, critical tasks and analysis of why expected actions did not occur. Its improvement-planning guidance also calls for tracking corrective actions and validating implemented changes (Federal Emergency Management Agency [FEMA], 2020, Chapters 5–6). I would retain that causal discipline without pretending a single exercise establishes a unique root cause.

A useful investigation asks what information was available, which procedure applied, whether authority existed, whether the necessary resource was accessible and whether the observer captured the relevant behavior. Check the exercise delivery too. A defective link to the plan could be an intentional simulation of unavailable documentation, a real access problem or a planner's mistake. Its provenance changes what the organization should conclude.

Triangulate consequential findings with another form of evidence. Compare the observed request with the approved procedure. Confirm the delegation with the accountable role. Inspect a sanitized access record or arrange a separately authorized technical test where necessary. These checks can substantiate or weaken the original hypothesis. They should not be used to rewrite the exercise history as though the participant had evidence that arrived only afterward.

I would also preserve strengths at the same level of specificity. If the team explicitly challenged an unsupported declaration of containment and requested the missing observation, that is a useful behavior to retain. “Great teamwork” is too vague to reproduce. A well-supported positive finding helps future exercises preserve the working part of the process while changing the failed dependency.

Check whether the evaluator's judgment is dependable

Two observers can watch the same exchange and disagree about whether a decision was authorized or merely proposed. I would calibrate evaluators using a short sample artifact before the event. Ask them to apply the criteria independently, compare their reasoning and clarify ambiguous terms. For high-consequence findings, keep an independent review of the evidence and record unresolved disagreement rather than averaging incompatible judgments.

This is a measurement problem as much as an exercise problem. If the criterion is “effective leadership,” a confident speaker may receive credit without making the required decision. If the criterion specifies acknowledgment of uncertainty, delegated authority and a bounded action, an evaluator has something more concrete to inspect. The rubric should reward the work needed by the incident, not the participant's speaking style.

Vykopal and colleagues’ review of tabletop-exercise research identified limited systematic assessment in the papers they selected. Their review screened 140 candidates and examined 14 in detail, under criteria oriented toward education and training in IT and operational technology contexts (Vykopal et al., 2024). This is evidence about that literature, not proof that every corporate exercise is poorly evaluated. It reinforces the need to explain how a claimed improvement was measured.

Participant confidence is another useful but different outcome. ENISA’s Cyber Europe 2024 after-action report describes pre- and post-exercise self-assessment alongside observations, interviews and feedback collection (European Union Agency for Cybersecurity [ENISA], 2024, “Methodology”). I would keep those evidence types distinct. Feeling more prepared may support a learning or engagement conclusion; it does not independently establish that a technical control works or that an organizational dependency has been repaired.

Use the debrief to test explanations, not manufacture agreement

A hotwash can capture observations while the context is fresh. It should not require immediate consensus on every cause. I would first ask what participants understood at the time, which options they considered and what information they lacked. Then compare that account with the retained artifacts. The distinction matters because a sensible explanation after the event may rely on facts unavailable during the decision.

Tannenbaum and Cerasoli’s meta-analysis found better performance associated with debriefing across the studies they reviewed, while cautioning that many designs were quasi-experimental and that causal interpretation required care (Tannenbaum & Cerasoli, 2013). The research spans domains and is not a guarantee of a particular cybersecurity outcome. I would use it to support structured reflection, not promise a percentage improvement from buying a tabletop.

Give participants a way to challenge a finding with evidence. An observer may have missed a message, or the exercise may have provided a contradictory instruction. Correcting that record improves the result. Equally, an executive's discomfort with a finding is not evidence that it is wrong. A factual review period should distinguish amendments to observations, competing interpretations and decisions to accept the remaining risk.

Avoid turning the debrief into a blame exercise. The goal is to find the relationship between information, authority, resources and behavior. If a failure depends on a poorly maintained escalation directory, telling one participant to “be more proactive” leaves the dependency intact. An individual training need can be real, but it should be supported by the evidence rather than assumed because an individual was visible when the process failed.

Close the finding with evidence of the changed capability

An action marked complete is not the same as a gap closed. Updating the delegation document may resolve a policy defect while leaving discovery and access untested. Training attendance may establish participation without showing that the intended role can use the procedure. I would write the closure condition when assigning the corrective action, so the owner knows what result is needed.

For the fictional authority gap, closure might require an approved delegation, an accessible procedure and a new exercise observation in which the designated role finds and applies it without the decisive prompt. If technical execution is part of the claim, add a separately authorized test of that mechanism and its effect. The closure record should retain the tested scope, assistance, remaining exclusions and the person accepting the evidence.

NIST SP 800-61 Revision 3 explicitly includes identifying improvements from security tests and exercises, including work with suppliers and relevant third parties (Nelson et al., 2025, ID.IM-02). That broader framing prevents an internal action list from overlooking a provider dependency. If only the customer side was exercised, closure of the customer's task should not be reported as validation of the complete supplier response path.

Retesting should answer the original uncertainty while changing enough irrelevant surface detail to reduce simple rehearsal effects. Use a different affected service or an unavailable alternate contact when those variations remain within the intended capability. Don't change every condition simultaneously and then pretend the comparison isolates one corrective action. A stronger outcome after training, a new procedure and a different team is useful, but attribution among those changes remains uncertain.

Maintain an explicit reason to reopen the finding. A changed provider contract, identity platform, delegation or recovery dependency can invalidate the tested arrangement. A scheduled review can complement these triggers, but a date alone cannot establish continuing validity. The evidence supports a capability under named conditions; it should travel with those conditions rather than become a permanent readiness badge.

Report what the exercise earned

The executive report should identify the objectives evaluated, the consequential observations, the interpretation of each gap, the corrective actions and the boundaries of the conclusion. I would lead with decisions: which capability needs investment, which authority must change, which dependency requires a real test and which risk remains accepted. Put the detailed evidence behind that summary so a reviewer can trace the argument.

The strongest practical objection is cost. Small teams cannot instrument every exchange or fund a large evaluation staff. Start with a few high-consequence objectives, clear outputs and a modest evidence record. A focused tabletop can expose an authority conflict cheaply. Requiring technical proof for every discussion point would waste resources and undermine the value of choosing an appropriate exercise format.

Another objection is that unexpected scenarios are more realistic than carefully bounded evaluation. Surprise can reveal assumptions, but uncontrolled surprise also makes explanations harder to separate. I would retain bounded objectives while allowing multiple response paths and recording unanticipated findings as hypotheses for follow-up. Realism should challenge the capability without making every result impossible to interpret.

This method has limits. A small number of scenarios cannot represent every incident, participants may behave differently under actual consequences, and an observed improvement can reflect practice or personnel changes. The guidance sources prescribe approaches rather than validate this combined method. The research papers address particular literatures and settings. No universal readiness score, causal effect or guarantee of future response performance follows from them.

If you need an exercise that produces findings you can act on and closure evidence you can defend, request an incident-response exercise briefing. Bring the decisions you need to test and sanitized descriptions of the response dependencies. I can help scope the objectives, design the observations and distinguish what the tabletop can establish from the technical work that must follow.