IOAM Extensions for On-Path Delay Measurement

Network operators use delay measurements to detect and localize performance degradation, including congestion, affecting customer traffic. RFC 9951 defines IP Flow Information Export (IPFIX) Information Elements for exporting statistical views of the on-path delay measured between an IOAM encapsulating node and subsequent transit and decapsulating nodes. To produce these statistics, the on-path delay needs to be computed at the node performing the aggregation. In postcard mode, this allows delay samples to be aggregated directly in the context of the customer flow that triggered their collection, without requiring subsequent correlation between separately exported telemetry data and customer traffic. IOAM Direct Export (DEX) provides such a mechanism for postcard mode. A packet carrying the DEX Option-Type triggers an IOAM node to collect, locally aggregate, and/or export the IOAM data selected by the IOAM Trace-Type. However, the DEX Option-Type currently does not provide the timestamp reference required by an IOAM transit or decapsulating node to compute the on-path delay defined in RFC 9951. This document defines two IOAM extensions to enable this computation: A new On-Path Delay IOAM data field, which identifies the on-path delay as an IOAM telemetry metric and allows it to be selected by the IOAM Trace-Type. An extension to the IOAM DEX Option-Type carrying a timestamp reference inserted by the IOAM encapsulating node. When processing a packet carrying these extensions, an IOAM transit or decapsulating node computes the on-path delay as the difference between its current timestamp and the timestamp reference carried in the packet. The resulting delay sample can be locally aggregated in the context of the customer flow and subsequently exported, for example using the IPFIX Information Elements defined in RFC 9951. This document defines the IOAM mechanisms required to compute and identify the on-path delay metric. The aggregation and export mechanisms are not redefined by this document.

This document uses the following terms as defined in and . IOAM domain IOAM encapsulating node IOAM transit node IOAM decapsulating node IOAM Direct Export (DEX) Option-Type This document also uses the terms postcard mode and passport mode as described in the Introduction of . In postcard mode, IOAM transit nodes also export on-path telemetry data. In passport mode, only the IOAM decapsulating node of the IOAM domain reports metrics.

The IOAM Direct Export (DEX) Option-Type defined in provides a mechanism for postcard mode in which packets trigger IOAM nodes to collect, locally aggregate, and/or export selected telemetry data. The metrics to be processed by a DEX node are identified using the IOAM-Trace-Type field carried in the DEX Option-Type. Computing the on-path delay requires two pieces of information that are not currently available together in IOAM DEX. First, the on-path delay needs an allocation in the IOAM Trace-Type registry so that DEX can identify it as a metric to be aggregated or exported. Second, the DEX Option-Type needs to carry the timestamp of the IOAM encapsulating node, which provides the reference from which a transit or decapsulating node computes the delay. The IOAM encapsulating node inserts the timestamp reference in the DEX Option-Type. When a transit or decapsulating node processes the packet, it subtracts this timestamp from its current timestamp to obtain the on-path delay.

. . . . D2 . . <--------------------> . . . . D3 . . <-----------------------------------> . . . (H1) ------ (R1) ------- (R2) ------- (R3) -------- (R4) ------ (H2) IOAM IOAM IOAM IOAM Host 1 Encapsulating Transit Transit Decapsulating Host 2 Node Node 1 Node 2 Node . . . . ......................................... ]]> D1, D2, and D3 represent the on-path delay from the IOAM encapsulating node R1 to R2, R3, and R4, respectively. Together, the On-Path Delay data field and the DEX timestamp extension allow an in-flight customer packet to trigger computation and local aggregation of the on-path delay at each participating node. For the per-flow aggregation use case described in this document, the IPFIX mechanism defined in is the preferred export method because the delay samples can be aggregated in the context of the customer flow that triggered their collection, avoiding subsequent correlation of telemetry records with customer traffic. The extensions defined here are not tied to IPFIX export. Other mechanisms can make the resulting on-path telemetry available, including the YANG data model and YANG-Push mechanisms described in . Export mechanisms are outside the scope of this document and are discussed further in .

This document defines a new IOAM Trace-Type data field for the on-path delay. The field represents the delay between the IOAM encapsulating node and the transit or decapsulating node where the value is computed. The allocation in the IOAM Trace-Type registry is required for IOAM DEX because the DEX Option-Type defined in uses the IOAM-Trace-Type field to identify which telemetry data is to be collected, locally aggregated, and/or exported. Allocating an On-Path Delay data field therefore allows a DEX node to select the computed delay for processing in the same way as other IOAM telemetry metrics. The same IOAM Trace-Type allocation can also be used by the Pre-allocated and Incremental Trace Option-Types defined in . Consequently, the allocation enables the on-path delay metric for both postcard mode and passport mode, although the DEX use case motivates the allocation in this document.

The On-Path Delay data field is a 4-octet unsigned integer in the range 0 to 2^31-1. It is the on-path delay between the IOAM encapsulating node and the transit or decapsulating node, measured in microseconds. Note that this unit differs from the nanoseconds used by the transit delay field of . The value is computed as specified in . If the delay exceeds 2^31-1 microseconds (about 35 minutes), the top bit 'O' is set to indicate overflow and the value is set to 0x80000000. If the node cannot compute the delay, for example because the difference is negative as a result of clocks of the IOAM encapsulating node and of the computing node that are not sufficiently synchronized, the field MUST be set to 0xFFFFFFFF to mean not populated. A delay sample with either of these two values MUST NOT be included in local aggregation. The data field is allocated by IANA as defined in .

In postcard mode, a packet carrying the DEX Option-Type can trigger telemetry data to be exported directly or locally aggregated before export, as defined in . The telemetry metrics subject to this processing are selected using the IOAM-Trace-Type field. The On-Path Delay data field defined in this document identifies the delay as one of these metrics. However, computing its value additionally requires the timestamp of the IOAM encapsulating node. This document therefore extends the IOAM DEX Option-Type with a timestamp reference. Two optional fields are defined: Timestamp Seconds and Timestamp Fraction. The fields are enabled using the IOAM DEX Extension-Flags.

As in , the bits of the IOAM DEX Extension-Flags are numbered from 0, with bit 0 being the most significant bit of the field. Bit [TBD1] of the Extension-Flags is used for Timestamp Seconds. Bit [TBD2] is used for Timestamp Fraction. The Timestamp Seconds and Timestamp Fraction fields MUST be used in combination. The Timestamp Seconds and Timestamp Fraction fields use the encoding formats defined in Section 5 of . The timestamp reference is inserted at the IOAM encapsulating node. At each subsequent IOAM transit or decapsulating node, the computing node obtains the on-path delay, represented by the On-Path Delay data field, by subtracting the timestamp reference from its current timestamp, using the seconds and fraction fields of the format in use at the full resolution of that format. The sign of the difference is evaluated at this resolution, before any conversion. The difference is then converted to microseconds and rounded to the nearest microsecond. For the PTP truncated format, the fraction is in nanoseconds. For the NTP 64-bit format, the fraction is in units of 2^(-32) seconds. For the POSIX format, the fraction is already in microseconds. A packet that carries only one of the Timestamp Seconds and Timestamp Fraction fields does not conform to this specification. A node receiving such a packet MUST NOT use the timestamp reference to compute the On-Path Delay data field value, and MUST set the field to 0xFFFFFFFF as described in . Each of these fields is an optional 4-octet field as defined in , so the length of the DEX Option-Type increases by 4 octets for each field present. The Extension-Flags explicitly indicate this length, as specified in . For example, when the DEX Option-Type is carried in IPv6 as specified in , the Opt Data Len field MUST reflect the resulting length, the 4-octet alignment is preserved because each field is 4 octets long, and the total length of the IOAM Option-Types remains subject to the 255-octet limit defined in that Other IOAM encapsulations, as presented in have their own constraints, which this document does not change. A node that does not support these fields ignores them as specified in . The computed value is meaningful only if the clocks of the IOAM encapsulating node and of the computing node are synchronized, and the accuracy of the result is bounded by the accuracy of that synchronization. Nodes participating in this measurement MUST use the same timestamp format, as determined by the management plane (see ). Deployment guidance on clock synchronization is provided in .

This document does not define an export mechanism. It defines the IOAM extensions required for an IOAM node to compute the on-path delay and to identify that value as a telemetry metric that can be locally aggregated and/or exported. defines IPFIX Information Elements and mechanisms for exporting on-path telemetry, including statistical views of on-path delay. This is particularly suitable when DEX processing aggregates delay samples on a per-flow basis, because the resulting telemetry can be exported in the context of the customer flow that triggered the measurements. Other export mechanisms can also be used. The YANG data model defined in provides on-path telemetry data that can be published using YANG-Push. In addition, describes a mechanism for exporting raw IOAM data. The choice of export mechanism is outside the scope of this document.

The security considerations for the IOAM Trace-Type and IOAM DEX Option-Type are defined in and . The additional considerations below apply to the extensions defined in this document. The on-path delay is computed from a timestamp reference carried in the packet and from the local clock of the computing node. An attacker that can modify the timestamp reference in a packet, or that can disturb the time synchronization of the IOAM nodes, can corrupt the delay samples. The consequences include delay values that are too small or too large, samples rejected as not populated, and locally aggregated statistics that hide or exaggerate a performance problem. Operators SHOULD protect the time source and the synchronization protocol used by the IOAM nodes, as discussed in , and use the extensions only within a limited IOAM domain, with packets filtered at the domain boundary. The Timestamp Seconds and Timestamp Fraction fields add 8 octets to the DEX Option-Type for each packet that carries them. This increases the per-packet overhead and the MTU and size considerations of the encapsulation that carries the option, for example for IPv6, and the rate-limiting guidance of continues to apply.

The on-path delay is computed from the timestamps of two different nodes, so the accuracy of the result depends on how closely the clocks of the IOAM encapsulating node and of the computing node are synchronized. Nodes participating in this measurement SHOULD synchronize their clocks using a protocol such as PTP or NTP. Operators need to ensure that the synchronization accuracy is small compared to the delays being measured. When synchronization degrades, the difference between the timestamps can become negative, and the node then reports the On-Path Delay data field as not populated, as described in . A higher rate of such values can indicate a synchronization problem, so operators are advised to monitor the clock status of the IOAM nodes that take part in the measurement. The protection of the time source is discussed in .

This document requests IANA to allocate the following bit in the "IOAM Trace-Type" registry.

This document also requests IANA to allocate the following two bits in the "IOAM DEX Extension-Flags" registry.

The authors would like to thank Greg Mirsky for the review and valuable comments. The authors also thank Haoyu Song for the review and valuable comments.