How Do You Reconstruct an OT Incident When Clocks Disagree?

Preserve the original timestamps, establish what each timestamp actually records, and estimate the clock correction and uncertainty for the incident window. Build the timeline from supported ordering relationships rather than sorting every log entry into an apparently exact sequence. Separately assess whether any operating function depends on the affected clock and what its engineering requirements permit during recovery. Correcting the display time does not, by itself, resolve either question.

I'd start by separating three decisions: what can we say happened first, which functions may be affected, and how can time service be restored without creating another problem? They need related evidence, but they aren't interchangeable. A historian's misleading timestamp can obstruct an investigation without proving that a controller's execution timing was affected. Conversely, a function can depend on synchronized time even when its logs appear orderly.

The framework below combines standards and original research with a worked uncertainty example and a proposed acceptance method. It is an analytical design, not a report of a plant incident or a tested control system. Numerical examples are hypothetical. No operational technology device, network, timing source or physical process was accessed or modified for this article.

Identify the clock that matters to the decision

The phrase “the plant clock” conceals several systems. A device may use wall-clock time to label events, a local counter to measure elapsed intervals, a synchronization service to align measurements, and another mechanism to schedule control execution. Before discussing an offset, I would identify which clock feeds the function under examination.

NIST's OT guidance explicitly connects synchronized clocks with cyber event correlation and functions such as motion control (Stouffer et al., 2023, §6.2.12). That establishes the need to investigate timing dependencies. It does not establish that every PLC scan, safety function or alarm uses the same network time source. The relevant dependency has to be confirmed through the actual design, configuration and equipment documentation.

For software that exposes separate clocks, their semantics can differ materially. The GNU C Library documentation describes CLOCK_MONOTONIC as suitable for elapsed time and unaffected by discontinuous changes to the system's absolute calendar-time setting; it is not an absolute timestamp (Free Software Foundation, n.d.). I would not assume every industrial application uses that interface, or that a local monotonic value can be compared directly with another device's value. A boot or session identifier may be needed to interpret the counter.

Build a requirement for each use: the reference timescale, acceptable offset, relevant interval, required confidence or bound, detection time, and behavior when quality is unknown. A log-correlation requirement and a synchronized measurement requirement may differ by orders of magnitude. NIST's smart-grid timing report discusses application-specific requirements and challenges in distributing reliable time through power systems (Allnutt et al., 2017, §§II–IV). It is context for asking the engineering question, not a tolerance table to impose on an unrelated plant.

This first distinction prevents a costly category error. An investigation team should not turn an unexplained timestamp discrepancy into a claim of unsafe operation without tracing the dependency. An operations team should not dismiss the same discrepancy as cosmetic before determining whether another function consumes that clock. Both conclusions require more than a timestamp screenshot.

Preserve timestamp meaning before normalizing it

I would preserve the raw record and the transformation history. What looks like an event timestamp might represent sensor sampling, controller detection, message transmission, gateway reception, historian ingestion, or eventual database insertion. These moments can differ even when every clock is accurate. A delayed upload can arrive after a later event without reversing the physical sequence.

NIST SP 800-92 warns that inconsistent host clocks can reverse the apparent order of events in cross-system analysis (Kent & Souppaya, 2006, §2.3.1). My extension is to preserve two distinct uncertainties: uncertainty about the clock and uncertainty about the point in the processing path where the timestamp was assigned. Correcting one cannot compensate for an unknown value of the other.

For each important source, retain the device and application version, timestamp field definition, timescale, time zone or numeric offset, display resolution, clock source, synchronization state and relevant correction history. Record whether the timestamp is generated at acquisition or added by a collector. Also retain sequence numbers, transaction identifiers and boot identifiers where available. These may establish relationships that wall-clock time cannot.

RFC 3339 defines an Internet timestamp representation with date, time, optional fractional seconds and a UTC relationship (Klyne & Newman, 2002, §5.6). A parseable string is useful for interchange, but formatting is not calibration. Adding six fractional digits does not establish microsecond accuracy, and attaching “Z” does not repair a device that had the wrong UTC value. Keep the source representation alongside the normalized value and document the conversion.

I would check time-zone and timescale assumptions before fitting an offset correction. A fixed display-zone mistake is different from an oscillator drifting during source loss. A timestamp based on a different timescale requires the appropriate conversion, while mixed leap-second handling can produce disagreement around a specific transition. RFC 8633 describes problems when NTP sources use inconsistent leap-smear behavior (Reilly et al., 2019, §3.2). That is one reason to inspect policy as well as server reachability.

Normalization should be reproducible and reversible. An analyst should be able to recover the original record, see the applied correction, and understand why the correction was valid for that interval. Silent replacement of raw timestamps makes later disagreement about the timeline harder to resolve.

Represent the timeline as intervals, not false precision

A corrected timestamp should carry the uncertainty relevant to the claim. For a simple model, let t be a recorded timestamp, c the estimated amount by which that clock is ahead of the reference, and u a justified error bound after correction. The corresponding interval is [t − c − u, t − c + u]. This is an illustrative interval model; it does not turn a vendor's typical accuracy figure into a guaranteed bound.

The correction may vary across the incident. A measurement taken after recovery does not establish the earlier offset. I'd divide the record into intervals supported by source changes, clock adjustments, restarts and available quality observations. Interpolation between observations needs a justified drift model; it should not cross an unexplained step as though the clock changed smoothly. Preserve uncertainty about the transition time itself. If an event could fall on either side of a correction, retain both possibilities until another record resolves the ambiguity. A single offset applied to the entire incident can conceal the behavior under investigation.

Suppose the following synthetic records use milliseconds after the same nominal second. The assumed bounds include clock correction and the relevant timestamp-capture uncertainty. They are invented inputs for demonstrating the method, not measurements from equipment.

Record	Raw timestamp t	Estimated ahead offset c	Assumed bound u	Corrected interval
A: controller input record	120 ms	8 ms	6 ms	106–118 ms
B: operator command record	125 ms	15 ms	8 ms	102–118 ms
C: historian receipt record	150 ms	2 ms	3 ms	145–151 ms

Sorting the raw values puts A before B before C. The corrected intervals do not establish whether A or B occurred first: they overlap. They do establish that the timestamped receipt event C occurred later than both, under the stated assumptions. They do not establish that the physical event described by C occurred later, because C records receipt. Nor does either ordering show that the command caused the input change.

For two events, I'd assert a strict temporal order from these bounds only when the latest possible time of one is earlier than the earliest possible time of the other. Touching or overlapping intervals do not support that strict conclusion. If the numbers are statistical confidence intervals instead of deterministic bounds, the interpretation must change; simply placing them in the same table does not create certainty.

Correlated errors need attention too. Two devices sharing one biased source may have poor agreement with UTC but good relative alignment. Independent worst-case bounds can then be conservative for comparing those devices. Conversely, treating shared-source agreement as independent confirmation can hide a common error. The report should say which reference the uncertainty concerns and whether the dependence between errors has been characterized.

I'd leave an event's order unresolved when the available evidence cannot support it. That is a useful result: it tells the team which hypotheses remain possible and which additional evidence could discriminate between them. An attractive, fully ordered diagram can be less truthful than a partial timeline with explicit uncertainty.

Use causal records without inventing physical causation

Message identifiers and within-process sequences can establish ordering even when clocks disagree. If a preserved record demonstrates that one process sent a particular message and another received that same message, the send precedes the receipt. The inference depends on trustworthy matching and known logging semantics, not on the numerical order of their wall-clock values.

Lamport's analysis defines a partial ordering through process order and message transmission, and distinguishes that relationship from a total ordering assigned by logical clocks (Lamport, 1978). The clock condition does not license the converse inference that every lower logical-clock value caused every higher one. I would use that distinction to challenge a timeline that presents timestamp order as an explanation of mechanism.

In a practical investigation, first establish the record's identity and role. A command request is not necessarily its acceptance; acceptance is not necessarily execution; execution is not necessarily the expected physical response. A historian entry may summarize an earlier state rather than record a fresh measurement. Identify the relevant acknowledgment, state transition and physical observation before connecting them into an account of what happened.

Some apparent contradictions are diagnostic. If a matched receive appears earlier than its send on the corrected timeline, revisit the timestamp semantics, offset estimate, identifier matching and possible record alteration. Do not immediately move the records until the story looks plausible. The contradiction may reveal a bad assumption or a missing stage in the path.

I would also avoid estimating a device's clock error solely by forcing a suspected cause to precede its supposed effect. That builds the desired explanation into the correction, then uses the corrected timeline as evidence for the explanation. Prefer independent timing observations, documented synchronization state, known test markers, or relationships whose ordering does not depend on the disputed incident hypothesis.

Examine path delay as well as the time source

A time server can be correct while a client estimates the wrong time. NTP's four-timestamp calculation estimates offset from the two directions of an exchange (Mills et al., 2010, §8). Its interpretation depends on the network-delay model. A round trip does not directly reveal how the delay was divided between the forward and reverse paths.

A simplified derivation makes the issue concrete. If the forward delay exceeds the reverse delay by d, the symmetric-path offset estimate has a bias of magnitude d/2, with sign determined by the offset convention. A hypothetical 20 ms directional difference can therefore create a 10 ms offset-estimation error. This is arithmetic under the model, not a measured network result or an assertion that every timing implementation has that exact error.

RFC 7384 identifies packet-delay manipulation as a timing threat: a party on the path can alter delivery timing without necessarily modifying the packet contents (Mizrahi, 2014, §3.2.6). Ordinary routing, congestion or path changes also deserve investigation when observations suggest delay asymmetry. The presence of offset alone does not identify an attacker or distinguish malicious manipulation from an operational fault.

I'd collect the timing service's offset, delay, dispersion or equivalent quality indicators, source-selection history and alarms, together with evidence about the path. Those statistics have implementation-specific meanings. A reported estimate against a selected source is not automatically an independent measurement of end-to-end application timestamp accuracy. The capture point, local clock discipline and application path can add further uncertainty.

This is where comparing independent observations helps. A trusted external measurement at the relevant device boundary may challenge a misleading “synchronized” flag. A nearby server test may not. The measurement design needs a reference and uncertainty appropriate to the actual requirement; otherwise it can validate the time server while leaving the failing endpoint unexamined.

Authentication protects a relationship, not every aspect of time

Network Time Security uses TLS-based key establishment and cryptographic protection for NTP client-server communication (Franke et al., 2020). That strengthens the protocol relationship, but it does not prove that the authenticated server's reference is correct. RFC 8915 also explicitly discusses asymmetric delay attacks that cryptographic packet protection does not eliminate (§8.6).

I'd therefore separate three questions: are we communicating with the intended service, is that service deriving time from an acceptable reference, and is the distribution path meeting its timing requirements? An authenticated service can still have a bad upstream source or an unsuitable path. A correct upstream receiver does not prove that every downstream device applied its information correctly.

Malhotra et al. (2016) demonstrated attacks affecting unauthenticated NTP synchronization in their evaluated implementations and network settings. The research provides evidence that time integrity and availability deserve explicit security analysis. It does not establish that today's particular plant is vulnerable, and this article does not reproduce an attack procedure. Current exposure requires examination of the deployed protocol, software, configuration and trust boundaries.

NIST's PNT profile frames responsible use around identifying dependencies and sources, detecting disruption or manipulation, and managing the resulting risk (Bartock et al., 2023, §§1–3). I would apply that approach to the local reference chain: receiver, antenna or upstream feed, time appliance, network distribution, endpoint and application. Include power, management access and configuration ownership. These shared dependencies can matter as much as the number of servers.

Protocol improvements also require compatibility and operational planning. Some legacy endpoints may not support the preferred protection or may depend on a particular PTP profile. A standard's existence is not evidence that an installed device implements it correctly. The engineering task is to establish an appropriate trust boundary and testable behavior for the actual equipment, not to change a live plant configuration merely to match a checklist.

Treat source diversity and holdover as measured capabilities

Two source addresses are not necessarily two independent references. They may share a receiver, upstream service, network path, power supply or firmware defect. RFC 8633 recommends reference-clock diversity and explicitly warns that devices from different vendors can still share implementation dependencies (Reilly et al., 2019, §3.3). I would document the common dependencies instead of describing redundancy from the IP count alone.

Disagreement creates a decision problem as well as an availability problem. If two sources disagree, a client needs a justified way to choose, reject, or enter a degraded state. A fixed preference can be an intentional operating policy, but it is not proof that the preferred clock is correct. Test the behavior of the actual selection implementation, including what operators see when it cannot establish acceptable time.

Holdover means maintaining time after loss of the disciplining reference, with quality depending on the clock and conditions. The smart-grid timing report discusses holdover and the need to characterize time quality in deployed systems (Allnutt et al., 2017, §§III–IV). For planning, I'd use a validated error model rather than assuming that a source outage immediately makes every clock unusable or that a stable-looking display remains accurate indefinitely.

Consider an intentionally simple bound: initial error no greater than e₀, plus fractional frequency error bounded by ρ during the outage. Ignoring other contributions only because the example assumes they are already bounded, error after elapsed time Δt is no greater than e₀ + ρΔt. If e₀ is 0.2 ms and ρ is 2 parts per million, one hour adds 7.2 ms, yielding 7.4 ms. Under an assumed 5 ms limit, that model reaches the boundary after 2,400 seconds, or forty minutes.

Those are hypothetical engineering inputs, not a recommended oscillator specification or plant limit. Temperature, aging, prior discipline, interruption, restart and the reference measurement may invalidate a simple constant bound. A product's typical drift figure is not automatically the worst-case value needed here. Validate the model in representative conditions and allow time to detect and respond before the functional limit is exceeded.

The monitor also needs a failure analysis. If it compares two clocks disciplined by the same bad reference, agreement may conceal the error it is supposed to detect. If its alert transport depends on the failed network segment, a correctly generated alarm may never reach the operator. I'd test the observation and delivery paths separately, record their shared dependencies, and specify how loss of monitoring itself becomes visible. An independent reference can strengthen the comparison, but independence must be demonstrated through its source and path. It is not a property conferred by buying a second appliance.

I'd make the degraded state visible to the functions that need it. A time source can remain reachable while its upstream reference is lost. If clients receive a timestamp without meaningful quality information, downstream users may continue to treat holdover output as fully established reference time. The design should specify what is signaled, what is monitored, and who owns the response.

Restore time without erasing the incident history

During an incident, correcting clocks can alter the evidence being collected. Preserve available source-selection history, offset observations and raw records before routine maintenance overwrites them, where doing so is compatible with the plant's immediate operating priorities. Document subsequent clock changes so analysts can distinguish the incident condition from the recovery action.

A step correction and gradual adjustment have different implications. A backward step can make wall-clock labels repeat; gradual correction takes time to reduce offset. Which approach is appropriate depends on the endpoint, application and operating state. RFC 5905 describes clock-discipline behavior that includes stepping under specified circumstances (Mills et al., 2010, §§11–12). I would not turn that reference algorithm into a universal command for restoring industrial equipment.

Prepare a recovery sequence for the actual dependency chain. Establish the intended reference, validate the distribution path, observe endpoint convergence, and verify the consuming function against its requirement. Recovered servers may carry old source settings or policies. Certificate-dependent time authentication may also have a startup dependency on the client's initial clock (Franke et al., 2020, §8.5). Resolve such dependencies through documented product behavior and an approved bootstrap procedure, not improvised disabling of checks.

The process owner needs to determine whether a function can continue, must operate with restrictions, or requires a controlled transition while time quality is unknown. A cybersecurity analyst's uncertainty interval is evidence for that decision; it is not a substitute for the equipment's operating requirements or the safety analysis. Avoid inferring an emergency stop requirement from an unexplained log offset alone.

Recovery acceptance should cover the transition, not just the final value. Did scheduled work run twice or get skipped? Did an application interpret old data as fresh? Did a historian preserve both timestamps when the clock moved backward? These are proposed questions for representative testing, not claims about every product. The separate article on testing OT backups without using the live process as the test bed addresses recovery-environment fidelity and hold points.

Test the evidence path and the operating path separately

I would use an isolated representative environment first, with the plant's engineering constraints defining what can be exercised and what remains unproven. The goal is to evaluate a specified timing dependency and its observability. It is not to disturb a running process to demonstrate that time matters.

An evidence-path exercise can use known markers with independently established ordering. Generate and collect them through the intended logging path, then compare raw records, normalized times, sequence identifiers and quality metadata. Deliberately vary a permitted lab clock condition or introduce collector buffering within the test design. Check whether the analysis distinguishes timestamp error from ingestion delay and whether it correctly leaves ambiguous events unordered.

An operating-path exercise needs the actual function requirement and a representative endpoint. Evaluate approved cases such as reference loss, selection of a backup source, restoration of the primary, and restart with stale configuration. Record equipment versions, source settings, measurement reference, uncertainty, traffic conditions, duration and observed transitions. Where a simulator omits physical behavior or timestamp hardware, state that boundary rather than promoting a software-only pass into a plant-wide claim.

Acceptance question	Evidence required	Reason to withhold acceptance
Can we interpret each event time?	Field semantics, source, timescale and capture point	Collector time mislabeled as occurrence time
Can we support the required ordering?	Bounded correction or independent sequence evidence	Unresolved overlap presented as certainty
Does source loss produce an observable degraded state?	Endpoint quality state and delivered operator alarm	Reachable service conceals unknown reference quality
Does recovery meet the functional requirement?	Measured endpoint behavior through the transition	Only the server or final dashboard value checked
Can the result be reproduced?	Versioned configuration, raw evidence and calculation method	Silent timestamp rewriting or missing assumptions

Retain enough evidence to answer these questions without indiscriminate collection. Timing diagnostics can reveal operational activity and network structure. Access, retention and redaction decisions belong in the evidence design. The companion discussion of security telemetry and privacy examines those controls; it should not be necessary to collect every payload merely to establish a clock's state.

Make the conclusion match the strength of the timing evidence

There is a reasonable objection that this approach demands too much for a small discrepancy. I would scale the effort to the decision. If every plausible correction leaves two events far apart and the conclusion unchanged, further calibration may add little value. If the suspected mechanism depends on a millisecond ordering that the records cannot establish, the uncertainty is central. Spend effort where it can change the operational or investigative conclusion.

A better clock architecture also cannot rescue every incident record. Missing events, ambiguous field meanings, shared bias, compromised logs and unobserved capture delays may remain. Cryptographic authenticity and high timestamp resolution address different properties from accuracy and completeness. Be explicit about which properties the evidence supports.

The deliverable I would want is a partial timeline with traceable corrections, an inventory of affected timing dependencies, and a recovery decision tied to functional acceptance criteria. It should separate supported facts, assumptions, unresolved ordering and proposed tests. That allows an engineering team to challenge the reasoning without having to reconstruct how a polished timeline was assembled.

If your incident account depends on clocks that nobody can characterize, request a briefing on OT time dependencies and incident evidence. Bring one disputed sequence and the function it affects. I would help identify the timing path, determine which ordering claims the records support, and define the evidence needed for a controlled recovery assessment. The useful result is a decision you can defend, with the uncertainty still visible.