IPFIX Export of RoCEv2 Sequence Observations

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 3 May 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

Two metering implementations can see the same PSNs yet report different gap counts if they use different history, initialization, or closure rules. A common export profile is useful when a collector consumes such observations from multiple implementations and must distinguish unresolved history from observed packets. The interoperability target is the interpretation of exported measurements, not an implementation-specific packet-processing architecture.¶

[I-D.lin-opsawg-ipfix-rocev2] defines raw RoCEv2 BTH Information Elements. Those fields remain useful for packet traces and validation. This document reuses its destination QP and partition identifiers and adds an explicitly specified summary. A raw-field export need not export every packet; however, a sampled packet trace cannot reconstruct the exact state of this profile. No throughput or export-volume improvement is claimed without measurement.¶

A shared collector may correlate queue and sequence observations, but such correlation alone does not establish which link lost a packet or whether an ECN policy is effective. A sequence position may be observed after reordering, a retry, duplication, or a capture fault. These causes cannot in general be distinguished by the counters defined here.¶

2. Requirements Language

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

3. Applicability and Provisioning

The profile covers only a provisioned, unidirectional RoCEv2 Reliable Connection request stream using BTH OpCodes 0x00 through 0x0B: SEND and RDMA WRITE, including the immediate-data variants in that range. Originally transmitted requests advance by one PSN modulo 2^24. Other RC operations, UC, UD, XRC, and inferred READ spans are outside this profile. Provisioning MUST establish the operation restriction for the whole connection, rather than merely filter the listed opcodes.¶

RDMA READ can reserve sequence space for response packets; [LINUX-RXE] provides a public implementation example. Filtering READ from a mixed stream does not make its remaining request PSNs consecutive. An observed unsupported request or malformed request attributable to the provisioned connection MUST close the epoch with profileViolation and disable metering for that connection until explicitly reprovisioned. The offending packet is not counted. A subsequent SEND or WRITE MUST NOT automatically re-enable the profile. Response packets and CNPs do not change this request state.¶

The observation point MUST process all request packets in the provisioned scope, without sampling, selective omission, or known capture loss, and MUST cover all paths carrying that stream. A single ECMP branch without complete coverage is not sufficient. Deployment provisioning MUST establish namespace separation and path coverage. If capture completeness cannot be established, this profile MUST NOT be enabled. A capture gap closes affected epochs with captureDiscontinuity; metering resumes only after complete observation has been re-established and starts fresh state. An unclassifiable capture fault affecting several contexts closes all contexts that may be affected.¶

Serial interpretation also requires that a packet cannot legitimately be separated from the current high-water position by half or more of the PSN serial space, including very old retries. This is an applicability condition, not something a 24-bit field alone can verify. If endpoint behavior or retention bounds cannot establish it, this profile MUST NOT be used for that connection. Known violations close the epoch. An exactly half-space distance is always treated as ambiguous.¶

4. Identity and Epoch Lifecycle

The record key comprises the IPFIX Transport Session, Observation Domain ID, observationPointId, source and destination IP addresses of the RoCEv2 packet, rdmaDestinationQP, rdmaPartitionKey, any additional tenant or tunnel namespace required to avoid collisions, and rdmaSequenceEpochId. A physical interface identifier alone is not a tenant namespace. Address family and encapsulation layer MUST be fixed and unambiguous in each Template. UDP source port MUST NOT split a connection context.¶

The Exporting Process MUST allocate a nonzero, monotonically increasing epoch ID across all contexts in each Transport Session and Observation Domain when the first report of that epoch is serialized. Thus first reports are serialized in increasing epoch-ID order; the ID does not order the first packet times. An ID MUST NOT be reused in that scope. IDs need not be consecutive at a collector. A new Transport Session starts fresh metering state. An epoch is closed on a known connection-generation change, idle expiration, policy or window change, capture discontinuity, profile violation, resource eviction, serial discontinuity, or impending counter/report-ID exhaustion.¶

Unless connection generation is independently established by trusted provisioning or endpoint integration, every report MUST set generationUnverified. A local epoch is not proof of a connection lifetime. Neither an idle timer nor an apparently new initial PSN proves that destination QP reuse was detected. Known connection restarts MUST NOT be joined even if addresses, QP, and PSN repeat.¶

A serial discontinuity caused by a valid eligible request closes the old epoch before counting that request, then starts a new epoch with that request. The request is counted exactly once. Other closure causes require the applicable validity conditions to be re-established before restart. A final snapshot abandons all pending positions. If it cannot be exported because of failure or resource exhaustion, the collector sees an incomplete epoch; a final record is never fabricated by the collector.¶

5. Bounded Sequence Observation Algorithm

Let M=2^24 and 1<=W<=2^22. The exporter maintains a local extended integer H, a fixed initial extended position S, and a retained interval [L,H], where L=max(S,H-W+1). Retained positions are either SEEN or PENDING. Positions before S or before L are unclassified, including positions numerically within W of H but preceding the first observed request. No initialized prefix is presumed seen or pending. Local extended integers are not exported PSN values.¶

For an arriving PSN p, compute u=(p-(H mod M)) mod M. If u=M/2, comparison is ambiguous. Otherwise d=u for u<M/2, and d=u-M for u>M/2. This uses the serial comparison boundary in [RFC1982]. Implementations MUST use checked arithmetic for counters and local extended positions; they MUST close and restart before overflow rather than wrap a cumulative counter.¶

At most W position labels are retained. Ordinary reports do not reset counters or pending state. On closure, add the number of pending positions to abandoned and clear pending. A position can be filled or expired or abandoned, never more than one of these. Expiry is caused by high-water advance, not by an elapsed-time estimate of network loss.¶

skipped = filled + expired + abandoned + pending
0 <= pending <= W - 1
repeated + unclassifiedLate + filled <= requests - 1

These relationships MUST hold at every snapshot. Abandoned is zero in an active epoch and pending is zero in a final epoch. Their validity establishes internal accounting only. A malicious exporter can satisfy them with fabricated measurements.¶

6. Export Profile and Template

This profile uses IPFIX over TCP [RFC7011] to order reports from an Exporting Process. A profile implementation MUST support this binding. Selecting TCP for this profile does not remove the general implementation requirements of RFC 7011, including its SCTP requirement, and does not update that RFC. Reconnection creates a new session and new epochs. TCP reliability does not guarantee delivery of a final report after process or connection failure.¶

Each Data Record MUST contain every field listed in Table 2, plus any required namespace keys. It MUST be an atomic snapshot of a single epoch. All counters are cumulative from the first accounted request. The report ID starts at one and increments once per serialized snapshot; retransmission of an identical snapshot retains its ID. The ID and each total counter MUST NOT wrap. flowStartMilliseconds and flowEndMilliseconds are the first and last accounted packet arrival times; they do not measure the export delay or the final closure time.¶

A Template MUST use exactly one address family. A field is not conditionally omitted within a record. All named fields occur once in this profile. The exporter MUST advertise the Flow Keys using the Flow Keys Options Template in RFC 7011, including record-level namespace and epoch identifiers but excluding report ID. The Transport Session and Observation Domain are outer context. An observationPointId change requires a new epoch. The unassigned identifiers in this draft and its dependency are not usable wire numbers.¶

With assigned standard IEs, no extra namespace fields, and full-length encodings as shown, there are 19 fields: 120 octets per IPv4 Data Record or 144 per IPv6 Data Record. The Template Record is 80 octets (4+19*4); its containing Template Set is 84 octets before any padding. These are distinct from the IPFIX Message length. An enterprise-specific field adds a four-octet PEN to its Field Specifier; therefore these Template lengths do not apply unchanged to experimental PEN mappings. No Template ID is reserved by this profile. Other legal integer lengths require explicit agreement and MUST retain the same semantics.¶

The destination QP uses the unsigned32 definition from [I-D.lin-opsawg-ipfix-rocev2], with its upper eight bits zero. rdmaPartitionKey is unsigned16. Both remain dependent on the referenced individual proposal; its progression or allocation is not assumed. Prior to publication, the definitions and assignments must be resolved with that work or replaced through explicit review, rather than silently inventing competing fields.¶

7. Collector Processing

The collector MUST validate Template bindings, field ranges, required keys, and the arithmetic relationships in Section 5. It compares cumulative counters only for an identical full epoch key. Window, connection-generation qualification, flow start time, and namespace are immutable within the epoch. Closure-reason flags appear only in final records. Each final record has pending zero; each non-final record has abandoned zero. Unknown flag bits make the record an unsupported qualification; the collector MUST preserve the raw record and MUST NOT derive unqualified measurements from it.¶

For a retained epoch, a larger report ID is processed only if total counters do not decrease and immutable fields agree. A report gap, including a first received ID greater than one, marks missing report coverage. It does not invalidate an otherwise self-consistent cumulative total, but differences across the gap MUST NOT be assigned to the missing subintervals. An identical repeat of the latest report ID is ignored; different decoded content for that ID invalidates the epoch. A smaller report ID is stale and MUST NOT modify already published values. An implementation MAY keep older raw records for offline validation.¶

After an accepted final record, no larger report ID can extend that epoch. A conflicting or extending record received while closure state is retained invalidates it. Collectors MUST set finite resource and retention limits and expose those limits to their operators. On retirement, they MUST preserve an outcome of complete, incomplete, or invalid in the stored result. Offline recomputation from an explicit archive is a separate operation.¶

For bounded retirement, the collector retains an epoch-admission high-water ID per Transport Session and Observation Domain. It first looks up retained context by epoch ID, rejecting a changed full key for that ID. For an unknown epoch, an ID at or below the high-water ID is an orphan and MUST NOT create or reopen context. An unknown larger ID advances the high-water ID, including when admission is refused because of resource limits. It can create a new context with missing initial reports explicitly qualified. This rule relies on the specified ordering of first reports. Retirement therefore needs no unbounded tombstone set. The high-water ID is cleared only with a new Transport Session. A collector MUST NOT silently forget it while continuing that session.¶

Silence alone is not a final report. A session failure or configured inactivity timeout without finalEpoch leaves the epoch incomplete. Complete report coverage requires reports from ID one through a valid final record without report gaps; metering qualifications still apply and this does not imply coverage of an entire connection lifetime. A missing final does not erase useful observed counters, but outstanding positions remain unresolved. Collectors MUST distinguish metering qualifications, report gaps, and collector resource drops from network loss.¶

Expired positions MUST NOT be labeled lost packets. Repeated PSNs MUST NOT be labeled confirmed retransmissions. Filled positions MUST NOT be labeled proven reorderings. Aggregate counters from different observation points MUST NOT be subtracted to claim a loss location: the records lack common event identities, and windows, paths, and epochs may differ.¶

8. Information Element Definitions

The following definitions follow [RFC7012] and [RFC7013]. Each requested IE has status current and reference RFC XXXX. TBD labels denote unassigned Element IDs. The units none are intentional for sequence-position quantities; packets is used only for counts of observed packet arrivals.¶

9. Verification Examples

With W=4 and PSNs 100,102,101,103,102: requests=5, skipped=1, filled=1, repeated=1, pending=expired=abandoned=unclassifiedLate=0. These observations are compatible with multiple network histories.¶

With W=4 and PSNs 100,99,100: position 99 precedes S even though it lies inside the nominal width. The result is requests=3, unclassifiedLate=1, repeated=1, skipped=filled=pending=0. Treating 99 as an initialized missing or previously seen position is incorrect.¶

With W=4 and PSNs 100,102,106: skipped=4, expired=1, pending=3. Closing produces abandoned=3 and pending=0. A later packet cannot fill the closed result.¶

PSNs 16777214,16777215,0,1 advance continuously with no skipped positions. A jump of 2^23 closes as ambiguous. With W=1, any forward advance greater than one starts a fresh epoch; no pending positions exist.¶

An observed READ disables the provisioned profile, including after subsequent SEND packets. A capture fault requires a fresh epoch after capture validity returns. Neither mechanism carries pending state across the interruption.¶

10. Security Considerations

The IPFIX security considerations in [RFC7011] apply. Exporters and collectors for this profile MUST support TLS 1.3 [RFC8446] over TCP and MUST use mutually authenticated, confidential export sessions with explicitly authorized peers. Current TLS deployment guidance is in [RFC9325]. A transport reconnect changes the epoch scope even when certificates identify the same exporter.¶

Addresses, QP identifiers, partition identifiers, and times can reveal tenant activity. Operators SHOULD restrict observation and retention to the required scope and authorize collector access. Malformed packets and fabricated identities can exhaust state or induce closure storms. Exporters MUST bound concurrent contexts and window size, restrict provisioning to authorized scopes, and expose capture/resource failures. Resource controls can lose reports; they MUST NOT silently convert incomplete observations into complete results.¶

An authenticated exporter may still be faulty or compromised. Arithmetic validation does not establish truth. This profile supplies no instructions to endpoints and MUST NOT be used as authoritative evidence to force retransmission or disable congestion feedback.¶

11. IANA Considerations

IANA is requested to allocate the twelve IEs in Section 8 in the IPFIX Information Elements registry [IANA-IPFIX]. No numeric IDs are requested. Experimental implementations MUST use enterprise-specific IEs under an assigned PEN and an explicitly shared mapping; they MUST NOT use TBD labels or unassigned standard numbers as wire values.¶

IANA is also requested to create an rdmaSequenceFlags registry within IPFIX assignments. Entries are individual unsigned16 masks, description, and reference. Initial entries are masks 0x0001 through 0x0020 in Table 15; individual masks 0x0040 through 0x8000 are unassigned. The registration policy is Specification Required [RFC8126]. Review must verify a nonoverlapping mask and define epoch/collector compatibility. A change to the observation algorithm or a new operation profile requires explicit version or IE separation; a flag alone MUST NOT silently redefine existing counters.¶

Authors' Addresses