]> Huawei Canada

christopher.janz@huawei.com

Lancaster University

d.king@lancaster.ac.uk

Network Management Research Group telemetry analog instrumentation optical networks digital twin diagnosis Evolution toward network operations automation requires systems encompassing software-based analytics and decision-making. Network-based instrumentation provides crucial data for these components and processes. However, the proliferation of such instrumentation and the need to migrate the data it generates from the physical network to "off-the-network" software poses challenges. In particular, analog measurement instrumentation, which generates time-continuous real-number data, may generate significant data volumes. This document discusses methodologies for handling analog measurement instrumentation data, informed by the requirements of network digital twins and of automated fault diagnosis. It proposes that instrumentation report, for each measured quantity and each interval, a summary that keeps the level, the rate of change, the departure from a steady trend, the extremes over a stated short window and the number of measurements taken. Such summaries can be combined exactly over any longer span, and they allow trends, brief events and stopped measurements to be detected and reported as they occur. Discussion Venues Discussion of this document takes place on the Network Management Research Group mailing list (nmrg@irtf.org).

Introduction Existing studies of network telemetry typically deal with packet-oriented measurements that generate packet traffic, path, discard, latency and other data , . However, some networking equipment and network operations scenarios feature or use more physically oriented measurement instrumentation that generates data of a different character. Here, the particularities of data generated by such "analog" instrumentation are examined, and telemetry methodologies suitable for such data are considered. This consideration is informed by the requirements of specific use cases, including network digital twins and automated fault diagnosis. Optical networks, which are increasingly rich in analog instrumentation, are used as a specific example here. But the telemetry methodologies discussed may apply to instrumentation and telemetry intersecting a wide variety of networks and their related operational software, for example in support of digital twins that model radio-based transmission, thermal characteristics or energy consumption. This document presents telemetry methodologies tailored to analog measurement instruments, aiming to improve data accuracy, transmission efficiency and real-time monitoring for the software that consumes the data. It summarises the state of the art, identifies gaps and proposes one methodology in detail: interval summaries (), with the measurement count as a basis for liveness ().

Terminology

Network Digital Twin:

A virtual replica of a physical network system that allows the simulation, monitoring and analysis of the network's behavior under various conditions without affecting the actual network.

Network Measurement Instrumentation:

The tools, techniques and systems used to collect, monitor and analyze data about the performance and behavior of a network.

Interval Summary:

A record, computed by or close to an instrument, that summarises the measurements of one quantity taken over one reporting interval ().

Short Window:

A duration, stated per measured quantity and much shorter than the reporting interval, over which measurements are averaged before extremes are taken.

Measurement Count:

The number of measurements an instrument has taken of a quantity, kept by the measuring process itself and reported with each value or summary.

Background Photonic networks, which transmit data through light signals via fiber optic cables, are fundamental to telecommunications, Internet services, data center operations and many other critical aspects of modern digital infrastructure. A range of measurement instruments is routinely used in the deployment and maintenance of these networks. Key examples include:

• Optical Time Domain Reflectometers (OTDRs): These devices test the integrity of fiber optic cables by sending a series of light pulses into the fiber and measuring the light that is scattered or reflected back. OTDRs can detect and locate faults, splices and bends in fiber optic cables, and are crucial for both installation and troubleshooting.
• Optical Spectrum Analyzers (OSAs): OSAs measure the power spectrum of optical devices to analyze the wavelength or frequency distribution of light. They are vital for characterizing the performance of components like lasers and optical amplifiers within the network.
• Optical Power Meters and Light Sources: Used in tandem, these instruments measure the loss or attenuation in optical fibers and verify power levels, to ensure that signals are transmitted with sufficient strength without exceeding the damage threshold of network components.
• Network Analyzers and Bit Error Rate Testers (BERTs): These tools assess the overall performance of the optical network by analyzing parameters such as signal integrity, bit error rates and latency.
• Wavelength Division Multiplexing (WDM) Analyzers: WDM combines multiple optical carrier signals on a single optical fiber by using different wavelengths. WDM analyzers are specialized tools for testing and maintaining these systems, ensuring that each channel is transmitted efficiently without interference.
• Dispersion Analyzers: These measure chromatic and polarization mode dispersion in fiber optic cables, which can affect the quality and speed of data transmission.

These instruments play a critical role in the characterization, deployment, optimization and troubleshooting of optical networks. But their use tends to be restricted to specific operational phases, requires manual operation, and is generally not compatible with application to operating facilities. The term instrumentation refers more properly to "embedded" capability that is both operable on active infrastructure and capable of continuous measurement. Such instrumentation is a necessary foundation for telemetry.

Optical Network Measurement Instrumentation Optical network instrumentation has typically focused on detecting transmission performance degradation, through measurement of error correction rates in FEC engines, counting of errored OTN frames and the like. Such measurements are typically executed on network elements through time-interval-based counting. The resulting counts may be forwarded to or collected by software on a subscription or polling basis. The data consists of series of integer numbers, or series of time stamp and integer number couplets. In recent years, however, the nature and scope of optical network instrumentation has broadened and deepened . The idea has been to instrument the optical network more richly to support more effective operations management, including the use of software-based analytics and modeling. Implicated network operations include network and connection planning and configuration, network and connection fault management (fault and impairment detection, classification, localization, preemption and correction) and others. The optical network is a high-performance analog transmission network, so, unsurprisingly, much of this new instrumentation is analog; that is, it produces time-continuous real-number data or data sets. Examples include optical loss, optical power (total, channel peak, etc.), optical spectra (narrow-band-filtered power measured at a series of center wavelengths), differential group delay (DGD), polarization mode dispersion (PMD), polarization dependent loss (PDL), Stokes vector components reflecting the state of polarization (SOP), linear optical signal-to-noise ratio (OSNR) and generalized optical signal-to-noise ratio (GSNR). Many of these measurements are synthesized by coherent receivers across the network, while some may be synthesized by in-span elements such as amplifiers and ROADMs. Equipment health quantities, such as the drive current of an amplifier's pump laser or the bias current of a transmitter's laser, are analog measurements of the same kind.

Telemetry Use Cases One application of this data in the software realm is with optical network digital twins (NDTs), used for transmission performance modeling , . Such NDTs constitute an important class of analytical engine supporting optical network and service planning and other operations, and they rely heavily on data from network instrumentation to model optical transmission performance accurately on targeted variations of the actual network and service configuration, state and condition. A default expectation would be that all instrumentation measurements are reflected continuously in the software realm for use by optical NDTs. However, at best only an approximation to this can be achieved (for example, only a series of sampled measurements may in fact be streamed from the network), so the imperative is to find efficient ways to support sufficiently accurate approximations. This imperative grows more compelling the greater the scale of the network and the richness of its embedded instrumentation. A second example application lies in the fault management domain, wherein analysis of rich data, concentrated around the time of a detected evolution in transmission conditions, may be used to classify and localize the origin of the observed evolution . Transient evolutions of transmission performance are commonplace on optical networks and have myriad causes, including extrinsic causes such as lightning strikes, earthworks and construction, weather, road and rail traffic and fires, as well as intrinsic causes including continuous or discrete deteriorations of equipment or fiber plant. Detection, classification and localization of transmission performance evolutions permit assessment of the likelihood, expected severity and rate of further deterioration, and planning of timely and cost-effective corrective interventions where indicated. However, successful analysis may depend on the availability in software of richer data sets than continuous streaming supports or other applications require.

Automated Diagnosis: What Is Lost Between Measurement and Use A third application is automated diagnosis, in which software proposes the possible causes of a reported problem and weighs the evidence the network's data hold for each, including the predictions of a digital twin. A simulation study of such an agent on a national-scale optical network carrying OTN services, evaluated on eight kinds of anomaly, found that several of the cases the agent could not settle, or settled wrongly, trace back to what is kept of a telemetry stream between the moment it is measured and the moment a diagnosis needs it.

• Slow drifts. Ageing of an amplifier's pump laser, of a transmitter's laser or of a receiver's margin develops over weeks with no alarm. With only recent samples held, software can read where a quantity stands now but not how it has moved. A level is not a trend: it cannot distinguish a unit that is ageing from one that has run high since it was installed.
• A ramp and a step. A receiver's error ratio rising slowly as its margin eroded was mistaken for a misreporting power instrument. Weeks of data would have shown a ramp for the first and a step for the second. The present level of the reading could not.
• Changes judged on too little. A capability that compared readings before and after an onset was left, for a drift with no onset in the window held, with a single sample after it. Across about a thousand monitored ports, one noisy sample beyond the threshold was then likely, and it was read as a fault.
• Brief events. Fast changes of polarization, mechanical disturbances of fiber and channel loading transients last seconds. Unless a capture was triggered when they happened, nothing in the data held recorded where or when they occurred.
• Stopped measurements. The largest single group of unresolved cases were instruments that had stopped measuring while their reports continued to arrive on schedule, with the last value repeated. Nothing in a report said whether its value was new.

Keeping every raw sample for weeks would address the first three at a cost few operators would accept across a network's many thousands of streams. and propose instead to keep a few numbers per interval, chosen so that the trend, the brief event and the liveness of a measurement all survive at any time scale.

Analog Measurement Requirements provides a framework for considering concepts, constructs and developments in network telemetry. Many of the methods and mechanisms it discusses or suggests are invoked here.

Sampling An analog-to-digital conversion process typically converts analog signals into digital data that can be transmitted, stored and processed more efficiently. This often involves sampling the signal at a certain rate and quantizing the amplitude into digital values. The "mirroring" (transmission for replication at a different place) of continuous-time real-number data, generated by in-network instrumentation, begins with sampling and representing measured values by a scalar or vector of finite-decimal-place numbers. As neither sampling at fixed intervals, nor fixed time alignment or offset among measurement points in the network or between such points and the off-network software realm, can generally be assumed, it is useful that instrumentation should generate, as primary data, a series of couplets or vectors consisting of sample time stamps and corresponding measured values.

Time Precision Inadequate sampling frequency and quantization error are both potential sources of error in the literal or effective "reconstruction" of the original time-continuous measurement in the software realm. Sampling frequencies might be varied in response to evolving temporal characteristics of measured parameters; this is one strategy for data reduction, and one reason why sampling may not occur at fixed intervals. Requirements on the precision of reconstructed data, its time basis and the alignment in time of different reconstructed measurements are determined by the operational role of the analytical functions that consume the data. Some operations of interest, such as network and service planning or fault and impairment management, may impose only relatively relaxed requirements on time synchronization among measurement instruments, and between those instruments and the software realm. Other applications, such as those tending toward closed-loop control, may require tighter temporal alignment among different measurement sources. These considerations have implications for the source and synchronization of the clocks producing time stamps, but in general the requirements on clock synchronization and precision are far from those required for bit-level operations: they are more like "network time" than "digital time". Similarly, requirements on the absolute or relative (among different instruments) precision of reconstructed measured values may be application-dependent. In many cases relative precision, or precision consistency, may be more important than absolute precision.

Reduction and Other Pre-Processing With telemetry data volume a primary potential challenge, methods for reducing the data volume associated with analog measurement instrumentation are of evident interest. Signals may also be filtered to remove noise and unwanted frequencies to improve data quality. Thresholding, triggering and interval summaries are three forms of reduction by pre-processing; the last is described in .

• Thresholding: Data samples are transmitted only if and when a measured value, or a derivative of the measured value, crosses a threshold. Possible examples include: a) exceeding some absolute or proportional variation from the last transmitted sample value; b) exceeding a previously observed and transmitted maximum or minimum value; or c) exceeding some time rate of change of the measured value.
• Triggering: An extension or variation of thresholding, triggering may refer to the transmission of a series of samples, from a defined set of instruments, over a defined period and at defined intervals, on the crossing of a particular threshold (that is, the threshold crossing "triggers" the transmission of the defined data series). The detection by instrumentation of some predefined occurrence, such as an unusually large or rapid change in an optical power level or a channel's SOP, would trigger the transmission of a predefined, rich set of data covering an interval around the triggering observation. That data could then be subjected to "forensic" analysis in software to support detection, classification or localization of events affecting transmission performance. Required pre-processing includes the processing of triggers and the sliding storage of sample values sufficient to cover the targeted capture window, as well as trigger processing and transmission intervals.

Post-processing of threshold-driven data may or may not be required by applications. For example, an application may generate a scenario for behavioral analysis by an NDT that requires the "current" data from network instrumentation. To whatever precision is reflected in the operating thresholding mechanisms, that data is simply the most recently transmitted sample. Another application, perhaps one dealing with fault or impairment management, might require a regular and continuous time series presentation of measured data; in that case, interpolation or other post-processing of received samples might be needed. Other kinds of pre-processing may also be of interest, including normalization of data, frequency domain conversion and computation of statistics.

Interval Summaries Thresholding keeps changes and triggering keeps events, but neither keeps the course of a quantity between them, and both are decided at the instrument once and for all. An interval summary is a form of reduction that keeps, for each measured quantity and each reporting interval (for example, 15 minutes), a small set of numbers from which the level, the trend, brief events and the liveness of the measurement can each be recovered afterwards, for any span of time.

Content of a Summary For each quantity x measured at times t within an interval, the instrument, or a proxy close to it, computes:

Level:

the mean of the measurements;

Rate of change:

the least-squares slope of x against t over the interval, with its standard error, so that a consumer can tell a real trend from noise;

Scatter about the line:

the standard deviation of the measurements about that slope. A steady drift fits a line and leaves a small scatter. A step, a break or another change of course inside the interval does not, and leaves a large one at the same change of mean;

Short-window extremes:

the minimum and maximum of x averaged over a short window stated for the quantity (for example, one second for an optical power, ten milliseconds for a state of polarization), each with the time at which it occurred;

Measurement count:

the number of measurements taken in the interval ().

Extremes of instantaneous samples, as commonly reported today, depend on the sampling rate in use, so the same event may appear in one report and vanish from another. Defining the extreme over a stated short window makes it a measurement of an event of that duration, and makes extremes from different instruments comparable. The times of the extremes allow an event to be placed by comparing when it appeared at successive instruments along a path.

Exact Combination over Any Span If each summary also carries the sums over its measurements of 1, t, x, t*x, t*t and x*x (the first being the measurement count), then the mean, slope, standard error of the slope and scatter about the line over any longer span follow exactly from the sums of the summaries within it, and the extremes over the span from the extremes of those summaries. Times are best expressed relative to a stated reference within each summary, with the reference reported, so that the sums retain precision; sums from summaries with different references are combined after a shift of origin, which is exact. No resampling or refitting of raw data is therefore needed to answer a question at a longer scale. The slope of a quantity over the last hour, over a day, or over the weeks before an event is available, from the same store, as exactly as if every measurement had been kept. A stream measured once per second produces 900 values in 15 minutes; its summary is about a dozen numbers. Summaries are facts about an interval and should be kept with the time they cover and the time they were received, and attributed to the measured element through records that are themselves dated, so that a question about a past moment can be answered as the data stood then. This is a matter for the consuming system, but it bears on what a summary must carry: the identity of the instrument and of the measured element, the interval, the reference time and the reception time.

Illustration Three readings of an optical power over two hours, summarised in 15-minute intervals, show what the summaries keep.

• A slow drift gives the same positive slope in every interval, significant against its standard error, with a small scatter about the line.
• A step, such as an instrument that begins to misreport, gives a flat slope before and after it and, in the interval where it occurs, a large slope together with a large scatter: the line does not fit.
• A brief excursion of a few seconds leaves the mean and the slope almost unchanged, but its short-window minimum stands out, with its time.

Compression Data compression is an obvious candidate methodology for bandwidth reduction. Methods for lossless compression of series of numerical data have been widely studied, for example . Such compression must be implemented as a pre-processing function executed by the instrumentation itself, or some proxy to it, and decompression as a post-processing function within the software realm. Where time stamps are uncompressed, depending on the compression methodology employed, it may be possible to support selective decompression of data, for example only on selected time intervals. This might allow application- driven, "as-required" post-processing of more limited volumes of telemetry data. The compressibility of time-based data depends on its evolution in entropic terms, resulting in streamed data flows of varying volume or rate. The effective transmission and reception rates of data samples thus may vary and differ at any point from the rate of data generation. This is another reason why data samples may require time stamps.

Programmable Streaming As discussed in , in-network pre-processing of telemetry data may usefully be "programmed" by telemetry clients (software applications that consume instrumentation data), including dynamically. The range and nature of software applications and their data requirements may vary among systems, may evolve with time within any given system, based on experience and learning or with the deployment of new capabilities, and may also vary with the instrumentation capabilities available on a given network. Interval summaries give three triggers that instrumentation can evaluate as each interval closes and push as notifications, rather than waiting to be polled:

• Trend: a slope that is significant against its standard error, held over a stated number of intervals, raises an early warning on change, before any threshold on level is reached.
• Brief event: a short-window extreme outside a stated band raises a notification carrying its size and time, and may start a triggered capture () at that instrument.
• Stopped measurement: a measurement count that does not advance raises a liveness notification, whatever the reports themselves continue to carry.

The interval, the short window per quantity, the significance required of a slope, the number of intervals it must persist and the bands for extremes are natural parameters for clients to program.

Measurement Liveness Streaming specifications provide subscription sequence numbers, heartbeats and observation times , . These show that reports are being produced and delivered. None shows that the value in a report is a new measurement. An instrument whose measuring process has stopped, while the process that reports its values continues, yields reports that arrive on time, carry fresh observation times and repeat the last value. To software, such a value is indistinguishable from a quantity that has not changed. A measurement count, kept by the measuring process itself and carried with every value or summary, distinguishes the two: a count that advances shows that measurements are being taken, and a count that stops shows that they are not, regardless of what the reporting process does. In the study of automated diagnosis described in , readings from instruments that had stopped measuring formed the largest single group of unresolved cases, and a measurement count carried with each reading resolved them without other change to the analysis. The count is the first quantity of the interval summary, so summaries carry it without further cost.

Streaming versus Polling Streaming, that is subscription-based push, is, as identified in and other works, and as suggested by the discussion above, expected to be the principal if not exclusive operational modality for telemetry, including analog instrumentation telemetry. Software clients consume data generated by the network and, having identified which data they require and from where within the network, use subscriptions to place themselves in a position to receive it on an ongoing basis without continuing operational steps. Triggered transmission of "batched" data is aligned with a streaming paradigm, as the telemetry server (the instrumentation) must detect the trigger conditions and react by capturing and transmitting data to subscribing clients. The same holds for summaries and their triggers. It is worth considering, however, whether polling can or should be completely dispensed with. The data needs of NDTs can be satisfied by, and are probably best served by, streaming. However, polling could be used if NDT-based analyses are required relatively infrequently, do not require very rapid execution and do not draw arbitrarily on historical data. Polling might also complement streaming. For example, to reduce data transmission and handling volumes, an NDT might unsubscribe from telemetry it has observed to change little with time, and then, for a particularly critical analysis, ensure that all available data is up to date by polling the unsubscribed instrumentation. Further, if certain kinds of compression are used, decompression can enter errored regimes, for example through transmission loss of telemetry data, and periodic polling may be useful to "reset" absolute values. As suggested in , the possibility of transmission loss of streamed telemetry, a concern particularly if unreliable transport such as UDP is used, may provide a general reason to enable polling as a fail-safe mechanism. Interval summaries ease this: a lost summary loses one interval, while the sums of the summaries around it remain exact.

Communication Protocols Communication protocols provide reliable data exchange between telemetry devices and control systems. Depending on the method, streaming or polling, various messaging protocols exist to provide efficient delivery of instrumentation data.

Data Models A complete framework for analog instrumentation telemetry might require data models supporting:

• identification of instrumentation-equipped and telemetry-capable network equipment, its available instrumentation, its available pre-processing and which aspects of that pre-processing are programmable;
• subscription to streaming from specific instrumentation;
• programming of pre-processing on specific subscriptions and instrumentation, including the type of pre-processing, applicable thresholds or triggers, the definition of trigger-associated data sets (included data and start and stop limits relative to triggering events), the summary interval and the short window for each quantity;
• transmission of time stamp and value couplets, vectors or batches;
• transmission of interval summaries: instrument and measured element, interval start and end, reference time, the sums of , the short-window extremes with their times, and the measurement count.

As far as the authors are aware, current performance monitoring models, such as the 15-minute and 24-hour records of and the average, minimum and maximum with their times in the OpenConfig models , carry level and extremes but no slope, no scatter about it and no sums from which longer spans can be computed exactly. Their extremes are of instantaneous samples at a rate not stated, and no model carries a count of measurements taken.

IANA Considerations This document makes no requests for action by IANA.

Operational Considerations Operational considerations for optical network measurement instrumentation involve a range of factors to ensure accurate, reliable and efficient performance of optical networks. Key considerations include:

• calibration and signal integrity;
• dynamic range and sensitivity;
• resolution and accuracy;
• scalability;
• bandwidth and storage of instrumentation data. Interval summaries bound storage per stream to a fixed number of values per interval, independent of the sampling rate, while preserving trends, brief events and liveness at every scale.

Future versions of this document will expand on these topics.

Security Considerations The security implications of optical network telemetry are significant, given the increasing reliance on optical networks for data transmission in many sectors. Ensuring the security and integrity of these networks and of the telemetry instrumentation used to measure and maintain them is necessary to prevent unauthorized access, data breaches, service disruptions, and use as threat vectors and attack surfaces. Key considerations include:

• encryption of sensitive telemetry data;
• secure configuration and management of telemetry functions;
• network monitoring and anomaly detection;
• secure data handling and storage.

Summaries and their triggers raise one further consideration: a falsified summary or a suppressed liveness notification could conceal a deteriorating element, so their integrity matters as much as that of the measurements themselves. Future versions of this document will expand on these topics.

Informative References ITU-T Optical Fiber Communications Conference IEEE/IFIP Network Operations and Management Symposium, Workshop on Technologies for Network Twins Journal of Lightwave Technology vol. 40, no. 10, pp. 3128-3136 IETF OpenConfig IETF Data Compression Conference

Changes from -01

• Added automated diagnosis as a use case, with the problems a simulation study found in what is kept of telemetry ().
• Added interval summaries: content, exact combination over any span and illustration ().
• Added triggers on trends, brief events and stopped measurements ().
• Added measurement liveness and the measurement count ().
• Extended the data model requirements and noted the gaps in current performance monitoring models ().
• Moved thresholding and triggering from Compression to Reduction and Other Pre-Processing, where they belong.
• Removed two paragraphs on network slicing from the Background section, included in -01 in error.
• Added terms; editorial corrections throughout.

Acknowledgements Thanks to the participants in the Network Digital Twin discussions of the Network Management Research Group, which provided input to this work. This work is supported by the UK Department for Science, Innovation and Technology under the Future Open Networks Research Challenge project TUDOR (Towards Ubiquitous 3D Open Resilient Network). The views expressed are those of the authors and do not necessarily represent the project.