How Do You Rotate an Internal CA Without Breaking Mutual TLS?

For a planned rollover, establish the new trust on both sides of each required mutual-TLS connection before switching the certificates those peers present. Confirm that applications have loaded the change, renew the workload identities, and test fresh connections in both authentication directions. Remove the old trust only after the remaining dependencies have been resolved. Keeping two roots temporarily can support that sequence, but it cannot guarantee uninterrupted service by itself.

I would treat this as a coordinated change in who may authenticate to whom. An issuer can successfully sign a new certificate while the application fails to present it, the remote peer cannot build its chain, or authorization rejects its identity. The migration is successful only when the intended service relationships work under the new authority and the retired authority no longer supplies an unintended path back in.

This analysis concerns a planned internal CA rollover while the old authority remains trusted. A suspected CA compromise changes the risk of overlap and may require containment that sacrifices continuity. Drawing on twelve authoritative sources, I develop a proposed migration model, test matrix and retirement rule. They are analytical tools, not an experimentally validated guarantee of zero downtime or instructions to execute against a live PKI without understanding its configuration.

Define the authority being changed

“Rotate the CA” can mean renewing an intermediate certificate, changing an intermediate's key, replacing the root, changing a trust bundle, or changing the default issuer selected by automation. These are not equivalent changes. Begin with the current and proposed chains, the signing keys involved, the trust anchors applications actually use, and the workload identities the change must preserve. A familiar CA display name does not establish that the underlying public key is unchanged.

RFC 5280 treats trust-anchor information as an input to certification-path validation. A self-signed certificate can carry that information, but its trust comes from a trustworthy delivery procedure, not merely from the fact that it signs itself. The same specification defines CA basic constraints and path-length restrictions (D. Cooper et al., 2008, §§4.2.1.9, 6.1.1). Distributing a certificate is therefore a security-policy operation when that distribution makes the recipient treat its key as authoritative.

An intermediate rollover beneath an unchanged root might require new chain material and workload certificates without replacing every root store. A root replacement can require changing the authority that each verifier accepts. Some deployments deliberately trust an intermediate directly. For those deployments, a nominal intermediate rotation also changes the effective trust anchor. Discover that distinction from deployed validation configuration rather than assuming the organizational PKI diagram describes every application.

Keep a record of the intended difference. If the goal is replacing an expiring CA, do not silently change identity naming, authorization policy and cryptographic algorithms in the same rollout. Sometimes a combined change is unavoidable, but its acceptance tests must cover the additional variables. Separating them where practical makes failures attributable and rollback more intelligible. The post-quantum inventory analysis addresses the broader discovery problem; this paper focuses on the ordering of one trust transition.

Model two authenticators for every connection

For a connection initiated by workload A to workload B, A validates B's server identity while B validates A's client identity. The pair may use different certificate profiles, trust stores or policy implementations. A successful test of server authentication alone cannot establish mutual authentication. Nor does success from A to B necessarily establish success for B initiating a separate connection to A, where their roles and certificate-selection rules can change.

A useful proposed model treats required service calls as directed edges. For each edge, record the identity each endpoint presents, the authorities each side accepts, the identity-matching rule, the authorization decision and the application operation that demonstrates useful service. This is more specific than counting workloads with a new certificate. One workload can participate in several edges whose consumers have different trust and reload behaviour.

In a simplified single-root example, let an old identity be O, a new identity be N, an old-only trust store be {O}, and a dual store be {O,N}. A verifier with {O} cannot be assumed to accept N. A verifier with {O,N} has the necessary root material for either, but acceptance still depends on the chain, identity, validity, usage and policy. The notation is an explanatory abstraction; it does not replace actual path validation.

Local presentation	Remote presentation	Trust needed for a mixed-phase connection
Old identity	Old identity	Each verifier must still accept the other's old path
New identity	Old identity	Remote accepts the new path; local still accepts the old path
Old identity	New identity	Local accepts the new path; remote still accepts the old path
New identity	New identity	Both verifiers accept the intended new paths

The middle rows explain why changing both sides “around the same time” is an inadequate plan. In an asynchronous estate, mixed states are part of the migration. Prepare them deliberately or isolate the relationship until both peers are ready. The matrix describes compatibility conditions; it does not justify making every authority trusted everywhere.

Preserve identity and authorization while changing the chain

A chain terminating at a trusted anchor does not establish that the peer is the service you intended to call. RFC 9525 requires a client to construct acceptable reference identifiers independently of those presented by the server and prohibits using the Common Name RDN to identify a service (Saint-Andre & Salz, 2023, §§2, 6.1.1). That source addresses service identity in its stated TLS context; it is not a universal authorization specification for all internal workload systems.

SPIFFE provides a different, explicit workload profile. Its X.509-SVID specification places one SPIFFE ID in a URI subject alternative name, requires RFC 5280 path-validation semantics, and adds leaf-validation requirements beyond ordinary X.509 processing (SPIFFE, n.d.-c, §§2, 5). Preserve the expected trust domain and workload identity when replacing the chain. Do not treat an arbitrary certificate signed by the new CA as sufficient evidence that the intended workload identity survived.

The migration test should include a valid new-chain identity that is not authorized for the target operation. If it succeeds merely because its issuer is trusted, the exercise has discovered an authorization problem, not successful rollover. Conversely, a correctly rejected unauthorized identity should not be treated as a migration outage. Define the expected peer and operation before interpreting a handshake or application error.

Avoid widening identity matching to overcome an inconvenient certificate profile. For example, switching from a specific workload identity to “anything from this CA” changes the authority boundary. If that is a deliberate architecture decision, it requires its own review. If it is an emergency troubleshooting shortcut, it can leave a persistent security regression after availability returns. The test record should distinguish chain acceptance, identity matching and permission to perform the requested operation.

Distribute overlapping trust before changing issuance

For a planned root rollover, a controlled overlap lets verifiers accept current identities while preparing for their replacements. SPIFFE's trust-domain and bundle specification illustrates adding a replacement root before removing the previous one. It also defines a monotonically increasing bundle sequence when present and a refresh hint whose timing is advisory (SPIFFE, n.d.-a, §4.1 and Appendix A). A published bundle version is useful evidence about intended state; it is not proof that every consumer has received it.

cert-manager's trust-manager documentation makes the sequencing risk concrete. Pointing trust directly at a changing issuer Secret can replace the old root immediately, breaking peers that still depend on it. Its documented approach uses a separately controlled trust source so both roots can coexist while workloads migrate (cert-manager, n.d.-b, “cert-manager Integration: Intentionally Copying CA Certificates”). This is a product example of a wider design need: separate authority publication from the decision to withdraw an authority.

Before enabling new issuance broadly, establish a release gate for the relevant verifiers. The evidence should identify the bundle version, the public keys it contains, the scope of distribution and the effective application state. If one partner cannot accept overlapping trust, treat that as a separate migration branch: a coordinated outage, an independently controlled endpoint, or another supported design. Do not assume dual trust is available just because your own stack supports it.

Overlap should also be scoped. A new internal CA for one workload domain does not automatically belong in every browser, operating-system or unrelated service trust store. Each additional verifier gives the authority another place where its signatures may matter. The proposed migration record should say why that verifier needs the authority, which identity policies constrain its use and when the old authority is scheduled for removal. Temporary trust still grants real capability.

Prove that the running process received the trust

A configuration controller can report success while an application continues using an older in-memory trust store. Kubernetes documents eventually consistent updates for Secret-backed volumes and states that containers using a Secret through a subPath mount do not receive automated Secret updates (Kubernetes Authors, n.d., “Using Secrets as files from a Pod”). That is a precise distribution limitation. It does not establish what an application's TLS library does after a file changes.

Trace the path from the authoritative bundle to the process that validates the peer. The sequence may include an API object, a node cache, a mounted file, a generated keystore and an application reload. Record the relevant version or fingerprint at each meaningful boundary. A file hash proves the file's contents; a fresh connection exercise can supply evidence about the running verifier. Neither should be reported as the other.

SPIFFE's Workload API provides streamed updates and specifies that response messages are complete updates, not patches whose omitted values should be retained. Its specification also warns that rapid widespread updates can cause reload-related unavailability and allows implementation timing choices (SPIFFE, n.d.-b, §§4.3–4.4). A consumer that only accumulates trust and never removes withdrawn material would defeat retirement. A consumer that reloads every instance at once can create a different failure: synchronized loss of service.

Use cohort-based rollout where the architecture permits it. Verify one representative configuration, then expand across independently behaving cohorts with an explicit reason for any sampling. Include startup-only loaders, long-running processes, delayed jobs and disaster-recovery environments. A successful reload on the newest service build cannot stand in for an older partner appliance or a dormant batch worker. What matters is the diversity of update paths, not the visual size of a green deployment dashboard.

Renew identities and verify certificate selection

Trust preparation is necessary but not sufficient. The workloads must eventually present certificates chained to the replacement authority. cert-manager's CA issuer documentation states that it does not automatically rotate the configured CA certificate, that changing the CA Secret does not itself trigger leaf reissuance, and that leaf validity can extend beyond the issuing CA's validity (cert-manager, n.d.-a, “Important Information”). Those are reasons to verify the issuer's actual lifecycle behaviour rather than infer a complete migration from one Secret update.

The issuance inventory should distinguish certificates already replaced, certificates queued for replacement and identities whose owner is unknown. Confirm that the new leaf contains the intended identity and permitted usages, that its chain is available to the presenting endpoint, and that the process selected it. When several certificates are configured, the default or preference rule can matter as much as the presence of the new file.

TLS 1.3 defines a certificate_authorities extension that communicates supported authorities and guides certificate selection. It may appear in ClientHello or CertificateRequest, depending on the sender (Rescorla, 2018, §4.2.4). A stale selection hint can therefore be relevant when diagnosing why a peer did not present the expected certificate. Do not equate the advertised list with the complete effective validation policy; inspect both selection and verification in the implementation being changed.

Keep issuance cutover distinct from trust retirement. Stopping old issuance prevents creation of more old-dependent identities, while keeping old trust temporarily can permit existing identities to finish their planned transition. If an emergency exception allows another old certificate to be issued, it changes the migration's dependency inventory and potentially its retirement condition. The exception must reach the migration owner, not remain invisible inside a certificate-service ticket.

Protect the channel that delivers the replacement

A migration can become circular when a workload needs the new trust to reach the service that distributes new trust. List the control-plane relationships as well as application calls: identity agents, secret-distribution services, enrollment endpoints and the administrative path used to repair failed consumers. A healthy data-plane canary does not establish that a freshly restarted workload can bootstrap its identity after the old root is removed.

Envoy's SDS documentation describes remote certificate delivery and the need to secure the connection between Envoy and its secret-discovery service. Its documented mTLS option uses statically configured client certificates for that SDS connection (Envoy Project, n.d., “Secret discovery service (SDS)”). This is an implementation-specific illustration of the dependency: dynamically delivered application credentials can still rely on separately configured credentials to reach their distributor. The retrieved documentation is a development snapshot, not a statement about every deployed Envoy version.

I would test both warm operation and cold start. The warm test asks whether a running process consumes an update. The cold test asks whether a newly started process can acquire current trust and identity using only its intended bootstrap material. Run those tests against an authorized representative environment. If the cold path still requires a retired anchor, leaving existing processes running merely delays the failure until a restart, scale-out or recovery event.

The repair channel needs an owner and a trust basis that survives the transition being performed. That does not justify an unauthenticated administrative backdoor. It means identifying a supported, protected recovery route before removing the last working path. An architecture that cannot update its own trust without already having the update needs redesign or a deliberately staged intervention; a longer overlap period alone does not resolve the circular dependency.

Test path construction, not just the root list

Rollover often exposes differences in how clients find a certification path. RFC 4158 distinguishes building a candidate path from validating it and discusses more complex cross-certified structures. It is Informational guidance offering implementation approaches, not a protocol requirement that all libraries choose the same path (M. Cooper et al., 2005, §§1–1.2). The practical test must include the chain served, certificates available to the verifier and the actual library's behaviour.

Cross-signing can create a path between old and new trust arrangements, but it adds relationships that must be understood during retirement. A new-looking leaf may still validate through an old authority when an alternate chain is supplied or selected. Likewise, removing one old-root file does not prove that the old authority is unusable if another trusted path remains. Record the path that actually validates in representative clients, not only the filename of the intended root bundle.

Test chain completeness, certificate constraints, algorithm compatibility and clock behaviour separately enough to explain a failure. A verification error after rollover may reflect an absent intermediate or an unintended usage rather than missing root distribution. Repeatedly adding more certificates to the trust store can make the symptom disappear while expanding trust unnecessarily. Diagnose the failed condition before changing the policy that defines acceptable authorities.

Where clients build different valid paths, define which outcomes are acceptable during overlap and after retirement. The objective is not uniform diagnostic output for its own sake. It is that the intended new identity works and that the authority being withdrawn cannot regain acceptance through an overlooked path. Keep cross-signing arrangements in the same dependency record as direct anchors so their eventual removal is not forgotten.

Make retirement a separate acceptance decision

There is no universal number of hours after which an old CA becomes safe to remove. The condition depends on the last old issuance, outstanding certificate validity, application update state, delayed workloads and alternative validation paths. A short nominal leaf lifetime is helpful only if the actual issuance rules, clock behaviour and inventory support the assumption that old-dependent identities cannot still be required.

My proposed retirement gate has three independent parts. First, necessary service calls succeed using intended new identities and trust. Second, no authorized remaining workload requires the old authority, including recovery and bootstrap paths. Third, the old authority is no longer accepted where the change is intended to withdraw it. The first two protect continuity; the third establishes the security effect of retirement. A migration can pass the first and fail the third indefinitely.

Use fresh full-handshake tests when evaluating the new certificate path. TLS 1.3 resumption can establish a subsequent connection using a PSK derived from an earlier connection rather than repeating a full certificate-authenticated exchange (Rescorla, 2018, §2.2). Existing traffic or resumed success therefore cannot alone establish that newly loaded trust accepts the new chain. Product-specific handling of resumption and live connections needs its own acceptance decision if the goal includes ending earlier authenticated relationships.

For negative retirement tests, use isolated, authorized test identities and a configuration that represents the relevant client cohort. Demonstrate rejection of an old-only path while demonstrating acceptance of the intended new path. Also retain the wrong-identity and wrong-authorization tests. A verifier that simply accepts everything can make every availability test pass, which is why successful communication by itself is an insufficient retirement criterion.

Work through an asymmetric migration

Consider a fictional service called Harbor Ledger. A reporting worker calls a processing service using mutual TLS. Both initially use identities under root O. The processing service updates its client-verification trust to {O,N}, but the reporting worker still trusts only O for server authentication. Meanwhile, the processing service starts presenting a server certificate under N. The worker rejects the service even though the service would accept either of the worker's old or new identities.

This is not a contradiction. One direction of authentication was prepared and the other was not. An operator checking only the processing service's trust bundle could truthfully report that dual trust was installed, yet still miss the cause of the failed call. The migration record needs two observations for that edge: what the worker accepts from the service and what the service accepts from the worker.

In the proposed recovery sequence, the worker's trust is updated and its running process is verified. A fresh connection now succeeds with the worker's old identity and the service's new identity. The worker then receives and loads its new identity. Another fresh test verifies new-to-new authentication and the intended reporting operation. The mixed-state test was useful because it localized the earlier failure; it was not treated as the final migration outcome.

A nightly export worker is still absent from the evidence. Its container image has an old-only trust store baked into it, and it starts only when the scheduled job runs. The team must test an authorized representative start or explicitly defer retirement until that dependency is resolved. Waiting until the next real job fails would be an observation, but it would not be evidence that the migration had been safe to close beforehand.

Finally, suppose a cross-signed chain still allows an old-only test identity to validate through a remaining trusted path. Application success under N has not removed that residual authority. The team identifies the alternate path, decides whether it is an intended exception and tests retirement accordingly. This invented case demonstrates the proposed model's logic; it supplies no empirical failure rate, rollout duration or claim about a customer deployment.

Bound overlap, rollback and the exceptions

Dual trust is not free. While the old authority remains accepted, its issuance power can still matter to those verifiers. Control who may issue, monitor exceptions and limit the overlap to a defined migration purpose. The argument for overlap is continuity during a planned change, not a claim that retaining more authorities improves security. If an old CA is suspected compromised, the routine plan must be reconsidered immediately.

NIST's key-management guidance says a compromised key must cease being used to apply protection and distinguishes that from controlled processing of information already protected (Barker, 2020, §5.5). Keeping a compromised CA accepted for new peer authentication cannot be justified merely by calling the period a rollover. The organization needs an incident-specific assessment of containment and continuity; this planned-migration method does not decide that tradeoff automatically.

Rollback should be described as an allowed state transition. Before old trust is removed, returning a cohort to a known valid old identity may remain an approved option. After removal, restoring old trust changes the security decision and needs explicit treatment in the change record. Keep the material and permissions needed for the approved recovery path available under control, without assuming that every historical secret should be retained forever or restored by default.

A legitimate objection is the cost of enumerating every edge in a large estate. Group genuinely equivalent clients by implementation, policy and delivery path, then document the evidence supporting that equivalence. Sampling can reduce effort, but heterogeneous systems should not be merged merely because they share a team name. The broader secret-rotation dependency analysis discusses surviving authority; CA rollover adds the need to coordinate each verifier's trust with each presenter's identity.

What a defensible completion record contains

I would expect the final record to connect the change's purpose to its actual outcome: old and new authority identifiers, affected service relationships, effective trust versions, presented identities, issuer cutover, full-handshake evidence, application authorization results, bootstrap checks and retirement tests. Unobserved cohorts and deliberate old-trust exceptions should remain visible with owners and acceptance conditions. A percentage of certificates renewed cannot replace that account of which relationships still depend on the old authority.

The method's limits matter. The source specifications describe defined semantics; product documentation describes particular mechanisms. Neither establishes a customer's installed versions, trust configuration or application reload behaviour. The cited web documents were retrieved on October 29, 2026, including a clearly labelled Envoy development snapshot. No real cluster, certificate authority, handshake, reload or recovery operation was tested for this paper. The compatibility model and Harbor Ledger scenario are original analysis, not measured research findings.

The practical decision is whether you can move through trust expansion, identity replacement and trust contraction without losing a required service relationship or silently widening authorization. If that sequence is not demonstrable, request an internal PKI rollover review. The work should produce a scoped dependency map, tests for the mixed states you will actually encounter, and a retirement decision supported by evidence rather than a hopeful date in the change calendar.