When Should Security Evidence Block a Green Software Release?

I would block a green release when a mandatory security claim is contradicted, when the evidence needed to evaluate it is missing or invalid, or when the proposed deployment exceeds the scope of approval. I would permit release when the required claims have credible evidence for the actual candidate and environment, with any allowed residual risk explicitly accepted by the right authority. The pipeline’s color is an input to that decision. It isn't the decision itself.

That distinction matters most when the dashboard looks reassuring. A successful scan, a high coverage percentage and an approved pull request answer different questions. Combining them into one badge can conceal the question nobody answered: what makes this particular change acceptable for these users, data and privileges?

This paper develops a release-evidence method for engineering leaders, application-security teams and accountable product owners. It synthesizes standards, original industrial experience and platform documentation; it reports no experiment or customer assessment. My proposed decision rules are analytical recommendations, not a validated scoring model. The aim is a reproducible decision with visible limits, rather than a declaration that software contains no vulnerabilities.

Start with the claim the release could invalidate

A release review needs a defined object. “The application passed security” is too broad to investigate and too vague to disprove. I would identify the candidate artifact, relevant configuration, intended environment, changed behavior and consequential security obligations. A dependency update, an authorization change and a reporting-screen adjustment don't necessarily threaten the same properties, even when they share one pipeline.

NIST’s Secure Software Development Framework version 1.1 calls for defined security-check criteria and mechanisms to collect and protect the information supporting them. Its examples include recording approvals, rejections and exception requests, as well as periodically reviewing automated decisions (Souppaya et al., 2022, PO.4). That supports a critical separation: deciding what counts as acceptable comes before choosing which tool produces a green indicator. I use the named 1.1 publication here, without treating a later draft as an adopted requirement.

Consider a fictional export feature. Its important claim might be that a user can export only the records allowed by the product’s permission model, including after a permission change. Another claim concerns whether the export worker can write outside its designated storage location. A third concerns whether the package being released is the candidate that was reviewed. These claims have different failure mechanisms and need different evidence.

The change description should explain which claims might be invalidated and why. A change in an identity library can affect several services without changing their business code. Conversely, a change to display copy may leave a security mechanism untouched. I would record the argument for reusing prior evidence, including the assumptions that make reuse defensible. “Only five lines changed” is a description of patch size, not an argument about security consequences.

OWASP ASVS supplies technical security requirements and recommends version-qualified requirement identifiers because numbering can change. Its project page identifies 5.0.0 as the release published on May 30, 2025 (OWASP Foundation, n.d.-a). I would use selected, applicable requirements to make obligations precise, then connect them to the product’s own data and permission rules. Copying a standard into a spreadsheet doesn't establish that its requirements were tested or that the product-specific obligations are complete.

Verify that the checks actually happened

Before interpreting a test result, establish whether the intended check ran. GitHub’s status-check documentation explicitly says that a skipped job reports success and does not prevent merging even when it is a required check (GitHub, n.d.-b). That is a specific platform behavior, not a claim that every workflow skip or every CI service behaves identically. It is enough to show why a success label alone cannot prove execution.

I would make evidence completeness a separately evaluated condition. For each mandatory check, record the expected job, actual run identifier, candidate identity, conclusion, relevant tool configuration and report location. A job excluded by a path filter should have a reason tied to the release scope. A missing report, an upload failure and a completed scan with no findings need different states. Otherwise the pipeline can accidentally reward the disappearance of evidence.

Analyzer execution also deserves its own check. SARIF 2.1.0 defines an executionSuccessful Boolean for an invocation, distinguishing the engineering system’s knowledge of tool success from failure. Its explanation warns that exit-code conventions are not uniform across tools (OASIS Open, 2020, §3.20.14). A consumer must interpret the actual producer’s contract; merely finding a report file or an empty results list is insufficient. This is a narrow use of the named SARIF version, not a claim about every report producer’s conformance.

For a scanner, I would retain the analyzed scope, enabled rule set, relevant exclusions and whether analysis completed. For dynamic tests, I would retain the tested service version and enough environment identity to distinguish the candidate from an older staging deployment. These are proposed evidence requirements. They don't imply that every scanner exposes identical fields or that collecting more metadata automatically makes the analysis correct.

The practical acceptance test is deliberately adversarial: could a required check disappear, analyze zero relevant files, or point at the wrong candidate while the release still appears ordinary? Evaluate those failure cases in an authorized test environment. The evidence evaluator should reject or explicitly classify them as incomplete. Its behavior should be tested independently of the application tests it summarizes, because an evaluator that cannot notice absent evidence defeats the purpose of the gate.

Match each technique to what it can establish

A completed analysis still has boundaries. Bessey and colleagues’ original industrial account describes a bug finder designed to discover errors rather than verify their absence, with engineering tradeoffs involving difficult constructs, false positives and analysis complexity (Bessey et al., 2010). The paper is historical experience from one tool-development effort. It doesn't establish the accuracy of today's products. It does explain why “no reported defect” and “proved defect-free” are fundamentally different claims.

NIST IR 8397 recommends complementary techniques, including threat modeling, automated tests, static analysis, historical tests and fuzzing. It describes threat assessment as a way to focus testing on consequential areas and notes that verification depends on references such as specifications or security policies (Black et al., 2021). I would therefore ask what reference made each expected result correct, not simply whether the tool ran successfully.

A static analyzer can supply evidence about the classes of problems it analyzes under its configured model. A dependency tool can supply evidence about identified components and its vulnerability information. A behavioral test can supply evidence about an observed property under particular inputs and state. None should silently stand in for a different obligation. An authenticated scanner reaching a page doesn't, by itself, establish that the authorization policy is the right one.

OWASP’s Web Security Testing Guide advocates combining reviews and technical testing throughout development rather than relying on a single technique. Its discussion distinguishes source-level inspection from black-box observations and explains why context matters (OWASP Foundation, n.d.-b). My release-level implication is to choose complementary evidence where a consequential claim has several plausible failure mechanisms. Running two tools that share the same blind spot is not equivalent to examining the claim from two independent directions.

For the fictional export feature, I would expect a reviewed permission rule and observations that permitted work succeeds while prohibited outcomes do not occur. I would also examine the worker’s privilege boundary if the change affects it. The detailed API authorization testing method belongs in its own paper. Here, the decision is whether the resulting evidence covers the claim affected by this release, and whether any untested dependency materially changes that conclusion.

Read coverage as a map of uncertainty

Code coverage helps locate execution gaps, but it doesn't tell you whether a test made a meaningful assertion. Google’s engineering guidance makes that distinction and rejects a universal ideal coverage percentage; it also discusses mutation testing as a way to examine whether tests detect changes in exercised code (Arguelles et al., 2020). Those are engineering recommendations, not a product-specific guarantee or a reason to abandon coverage measurement.

I would review the changed security-sensitive paths before the aggregate number. Suppose the export tests execute the denial branch but assert only that a response exists. Execution is real; evidence that protected records remained inaccessible is weak. The missing observation may concern a response body, a written file or a deferred action. The decision depends on the intended property, not on whether a line was colored as covered.

Mutation testing can be useful when a small, authorized change to a security condition should cause a test to fail. If removing a permission check leaves the targeted test green, that suggests the test isn't detecting that change under the chosen setup. It doesn't prove the entire test suite is useless. Nor does detecting selected mutations prove that every relevant fault would be caught. I would keep the inference local to the mutation, assertion and execution context.

Coverage also has a denominator problem. Excluding generated code, unsupported languages or a separate worker may be legitimate, but the summary must reveal whether an excluded component participates in a mandatory claim. A product with fewer measured lines can display a higher percentage without gaining evidence about the behavior that matters. The release review should reconcile the measurement scope with the claim scope before comparing percentages across builds.

My preferred report would name the consequential obligations, the observations supporting them and the remaining exclusions. Aggregate coverage can remain a useful diagnostic beside that report. It shouldn't be allowed to offset a missing assertion for a critical boundary. Ten thousand passing formatting or rendering tests cannot establish that an untested administrative operation enforces its authorization requirement.

Establish candidate identity and evidence validity

The right test against the wrong candidate is not release evidence for the candidate being shipped. SLSA’s version 1.2 verification guidance checks the relationship between an artifact and its provenance, trust in the evidence, and expected build properties. It includes matching the artifact digest and evaluating expected source and builder information (SLSA, n.d.). Those checks support identity and build-origin claims. They don't establish that the application’s business behavior is secure.

I would treat artifact verification as an input to the release argument and use the separate build-attestation verification paper for its implementation details. A verified origin cannot compensate for a demonstrated authorization defect. Equally, strong behavioral testing cannot establish that an unrelated artifact is the one that underwent that testing. The two evidence types are necessary for different reasons where policy requires both.

Evidence validity should be tied to relevant changes, not just its age. A result from yesterday may be stale after a policy change; a carefully scoped review from last week may remain relevant to an unchanged mechanism. I would define invalidation triggers such as a changed identity provider, test fixture, compiler option, dependency resolution, authorization configuration or deployment topology. The record should explain why each trigger matters to the supported claim.

Not every changed input requires every test to run again. The engineering task is to trace the change to affected claims, rerun the necessary observations and document the boundary of reuse. If the mapping is uncertain for a consequential property, that uncertainty is itself a decision input. It shouldn't be erased by automatically carrying forward the previous release’s acceptance flag.

NIST’s systems-security engineering guidance describes assurance cases as structured claims, arguments and evidence, including assumptions and constraints, maintained as engineering circumstances change (Ross et al., 2022, §4.3). I apply that reasoning here at release scale. A compact argument can be more useful than a large bundle of unexplained logs: the reader should understand which evidence supports which claim, and what would make that support cease to apply.

Resolve contradictions before counting passes

A release can contain both reassuring and adverse evidence. Suppose a scanner reports no relevant findings, but a reproducible test demonstrates an unauthorized export. These aren't equally weighted votes. The observed violation contradicts the stated boundary; the scanner's silence may simply be outside its detection capability. I would block ordinary release until the contradiction is corrected or shown, with evidence, not to apply to the candidate.

Dependency findings require similarly careful interpretation. CISA’s VEX minimum requirements distinguish affected, not affected, fixed and under-investigation statements. For a not-affected status, the document requires a justification or an impact statement explaining applicability (Cybersecurity and Infrastructure Security Agency [CISA], 2023). This supports treating applicability as a reasoned assertion with identifiable product and vulnerability scope. It doesn't make every supplier assertion reliable or make an unresolved investigation equivalent to a clean result.

I would preserve the original finding and record its disposition alongside it. “Not affected in this product configuration” is different from “affected but temporarily accepted.” The former needs evidence for non-applicability; the latter needs an authorized risk decision. A suppression that hides both distinctions prevents a later reviewer from understanding what changed, especially when a new deployment configuration invalidates the original rationale.

Disagreement between tools should produce a bounded investigation. Confirm that they examined the same candidate and component inventory, compare configuration and information dates, then inspect the specific disputed claim. The outcome might be a false positive, a coverage gap or unresolved uncertainty. A tie-breaking rule that always trusts the cheaper or more convenient result is not an evidentiary argument.

I would also distinguish evidence quantity from independence. Several reports may inherit one mistaken permission model or use the same fixture that omits another tenant. Repeating the same assumption doesn't strengthen it. For consequential claims, seek a challenge that can reveal an error in the shared premise: review the intended rule with its business owner, inspect a different observation point, or test an explicitly excluded state. More reports are useful only when they add relevant information.

Use dispositions that preserve the reason for the decision

My proposed method gives each mandatory claim one of four evidentiary states: supported within scope, contradicted, incomplete, or not applicable with a recorded reason. Exceptions are separate authorizations attached to specific requirements; they are not a fifth kind of passing test. This avoids two common distortions: treating missing evidence as failure of the software, and treating permission to proceed as proof that a failed requirement was satisfied.

The following example is fictional. It illustrates the logic rather than a recommended universal release policy, a real product finding or an empirical result.

Claim for one candidate release	Available evidence	Disposition and next action
Intended authorization behavior remains intact	Negative test exports a prohibited record	Contradicted; correct the behavior and rerun affected tests before ordinary release
Required dependency analysis completed	Job succeeded but its report was never produced	Incomplete; establish execution and obtain a usable report
Published artifact matches the reviewed candidate	Verified artifact identity and expected provenance	Supported for identity; does not resolve the authorization defect
Disabled export feature is inaccessible in the proposed restricted deployment	Configuration recorded, but alternate entry path untested	Incomplete; restriction is not yet credible evidence for conditional release
Unchanged isolated helper retains its previously reviewed property	Relevant inputs unchanged and reuse argument reviewed	Supported within stated assumptions; reopen if those inputs change

There is no legitimate percentage to calculate from this table. Two supported claims do not cancel a contradicted boundary or an incomplete deployment restriction. If the organization defines tenant isolation as a non-exceptionable release condition, the demonstrated violation blocks release regardless of the other rows. That rule is a policy choice I recommend making before the deadline, not a universal mandate from the cited standards.

Conditional release might become defensible if a separately verified restriction removes the affected behavior from the proposed exposure and policy permits the remaining risk. The condition must be enforceable, observable and tied to the approved scope. A hidden menu item doesn't establish that an endpoint or queued operation is inaccessible. Changing the release scope also changes the claim being approved; it doesn't retroactively turn the original full-feature release into a pass.

For exceptionable obligations, I would require a named decision authority, the exact gap, consequences, compensating measures, validity conditions and an action that resolves or re-evaluates the decision. The exception-closure paper covers the subsequent lifecycle. Here, the important boundary is that an exception must be visible to the release evaluator and must not extend to other artifacts, environments or future changes by accident.

Put authority and emergency behavior into the mechanism

A review checkbox is useful only if the person selecting it has the right authority and sees the relevant evidence. GitHub documents optional prevention of self-approval and configuration that can prevent administrators bypassing deployment protection rules; availability and behavior depend on the environment and applicable product plan (GitHub, n.d.-a). Those are examples of enforceable workflow choices. They aren't proof that an organization's actual repository is configured that way.

I would separate three responsibilities: engineers produce and explain evidence; an appropriately qualified reviewer challenges consequential claims; an authorized owner accepts residual risk where policy allows. A small team may combine roles, but it should disclose the resulting limitation rather than claim independent review. The release mechanism should preserve who made which decision and what evidence they had at that time.

Emergency changes need a predefined route because delay also has consequences. The right question is whether the available evidence supports the emergency scope given the risk of waiting. I would retain candidate identity, the critical affected claims and an explicit authorization even when less consequential checks are deferred. A deadline doesn't supply missing evidence, and an emergency label should not silently change a non-exceptionable requirement.

Deferred work should remain attached to the release record with an owner and a condition for follow-up. If a compensating restriction is what makes deployment acceptable, the system needs a way to detect its removal or failure and act on that event. Otherwise the temporary decision becomes a permanent assumption. The precise response could be disabling the feature, reducing access or another tested action appropriate to the service; no single rollback rule fits every failure.

Preserve a decision that another reviewer can reconstruct

A useful release record should survive the dashboard changing. I would preserve the candidate identity, policy version, claim dispositions, evidence references and approval scope together. Link to the actual run or immutable record where the system supports it, rather than to a dashboard that always displays the latest build. Record the reason for a disposition in language another engineer can challenge. A screenshot of a green badge is weak evidence if the underlying candidate and evaluation rules are no longer identifiable.

The record also needs a distinction between discovery and correction. If a release is blocked by an invalid test fixture, fixing the fixture establishes that the evaluator can observe the intended property. It does not establish that the application satisfies that property until the relevant test runs again. If the rerun reveals a defect, the release should retain that history instead of presenting only the final successful attempt. Otherwise a future reviewer cannot distinguish a repaired application from a repaired measurement.

Retention should preserve the reasoning without collecting unnecessary secrets or customer data. Use synthetic cases and sanitized identifiers where they support the claim. Restrict sensitive diagnostic material and retain the minimum necessary reproduction context. Evidence that creates a new disclosure risk can undermine the security objective it was supposed to support. These are proposed record-design principles, not a claim that a particular retention period is universally required.

Keep the evidence burden proportional and useful

The strongest objection is that this becomes another approval bureaucracy. It will, if every routine change requires a new document, a committee and an unchanged pile of attachments. I would implement the method as a small set of versioned claims and machine-collected evidence, with human attention focused on changed assumptions, contradictions and exceptions. The ordinary path should be fast because its conditions are explicit, not because uncertainty is hidden.

A second objection is that an external penetration test should settle the issue. Such a test can add valuable evidence within its scope and timing, but the report still needs a relationship to the candidate, configuration and claims being approved. A report produced before a consequential change may need targeted follow-up. Buying another assessment doesn't solve an inability to identify which evidence remains applicable.

A third objection is that absolute confidence is impossible. I agree. Finite testing cannot exhaust every operational state, and even a strong assurance argument may begin with an incomplete threat model. This method makes the remaining uncertainty inspectable; it doesn't eliminate it. Formal analysis can strengthen particular claims under explicit assumptions, while still leaving deployment configuration or organizational authority outside the model.

The sources also have limits. Guidance documents prescribe practices rather than demonstrate this combined method's effectiveness. The industrial paper describes historical experience, and vendor documentation describes particular platform behavior that can change. No cited work validates a universal threshold for sufficient release evidence. Organizations must adapt the proposed dispositions to consequence, architecture and applicable obligations, and evaluate whether the process actually catches decision-relevant gaps.

I would start with one critical product and review a small set of authorized historical releases. Trace each claimed pass to its source, identify missing or invalid evidence, and ask whether the decision would change. Then test the evaluator's failure behavior with controlled fixtures before relying on it. Useful measures include unexplained evidence gaps, invalid carry-forwards and overdue exceptions, interpreted alongside release delay and the consequences of errors. Targets should follow observed baselines; invented improvement percentages would obscure the work.

If you need to know whether your release approvals are supported by what your pipeline actually proves, request a release-assurance assessment. Bring a sanitized pipeline outline, one release record and the security obligations that matter most to the product. I can help define the review scope, trace the evidence to those obligations and identify the changes needed to make a green release mean something specific.