SCONE-Based Rate Advice for RoCEv2 Networks

Introduction Remote Direct Memory Access (RDMA) enables data to be read from and written to remote memory without involving the CPU, which effectively reduces latency, improves throughput and lowers computing overhead of devices. It is widely deployed in high-performance computing and artificial intelligence scenarios. As a native lossless network for HPC and AI, InfiniBand natively supports RDMA. The RDMA over Converged Ethernet (RoCE) standard introduces RDMA capabilities to Ethernet. Among its versions, RoCEv2 carries the InfiniBand transport layer over UDP/IP and has become the most widely adopted version in the industry. Data Center Quantized Congestion Notification (DCQCN) is the default congestion control algorithm for RoCEv2 networks . When a DCQCN flow starts transmitting, it sends data at the physical line rate by default. It reduces the transmission rate upon receiving Congestion Notification Packets (CNP) and restores the rate quickly when the network is free of congestion, forming an end-to-end transmission rate adjustment mechanism. However, the binary nature of the Explicit Congestion Notification (ECN) mechanism can only indicate the presence or absence of network congestion, which limits the timeliness and effectiveness of rate adjustment on the sender side. Based on the concept of SCONE , providing advised transmission rates for RoCEv2 endpoints is of great practical significance. This document defines a Rate Advice mechanism adapted to the RoCEv2 protocol, to deliver quantitative rate recommendations. Two implementation solutions with different granularities are specified:

Solution 1 (Out-of-Band UDP, Path-Granularity): Network elements directly send Rate Advice messages over UDP to senders. The advice is identified by the source IP address and applies to all active QPs originating from that source IP. This solution provides sub-RTT feedback and is particularly suitable for inter-datacenter scenarios.
Solution 2 (On-Path Endpoint Feedback, Per-QP Granularity): A Rate Signal is carried in RoCEv2 data packets and filled by network elements along the path on a per-QP basis; the receiver returns the per-QP advised rate to the sender via a feedback loop, following the SCONE protocol model .

Terminology The following terms are used in this document:

Rate Advice: A quantitative rate adjustment recommendation generated by network nodes (e.g., switches) and delivered to the sender.
Rate Advice Message: A message that carries Rate Advice information to be conveyed toward the sender. In Solution 1, it is an out-of-band UDP packet carrying a path-granularity advised rate as generated by network nodes; in Solution 2, it is a feedback message carrying the per-QP on-path Rate Signal as generated by the flow’s peering receiver.
Rate Signal: A field carried in RoCEv2 data packets, filled by network elements along the forwarding path to indicate the advised rate for a specific QP.
Path-Granularity (Solution 1): The advised rate is identified by the source IP address and applies to all active QPs originating from that source IP.
Per-QP Granularity (Solution 2): Each Queue Pair (QP) receives its own advised rate, identified by the Destination QP ID.
Out-of-Band: The Rate Advice signaling is separated from the RoCEv2 data flow, using independent UDP packets.
On-Path: The Rate Advice information is carried within the RoCEv2 data packets and returned to the sender via an endpoint feedback loop.

Deployment Scenarios

Deployment within Intelligent Computing Data Centers Inside an intelligent computing data center, RoCEv2 flows are generally exchanged within a single data center, and the network is mainly built with two-tier Leaf-Spine or three-tier Clos architectures. Network nodes such as Spine switches and Leaf switches can monitor the egress queue depth, queuing delay and packet loss rate. When an uplink link gets congested, the switch generates Rate Advice Messages based on real-time monitoring results and sends them directly to the senders, acting as a supplement to DCQCN. In this deployment scenario, SCONE-enabled network nodes reside on the data plane of the data center network and coexist with the existing DCQCN protocol stack. Senders can receive both CNP congestion notifications from DCQCN and Rate Advice from SCONE. The recommended implementation policy is that senders give priority to responding to emergency congestion indications carried in CNP, and conduct gradual rate adjustment with reference to the advised rate in SCONE Rate Advice. For Solution 1 (path-granularity out-of-band UDP), the advised rate is derived from the aggregate behavior of all active QPs sharing the same source IP, providing a lightweight, low-overhead guidance that is easy to implement on the network side. For Solution 2 (per-QP on-path endpoint feedback), the existing RoCEv2 data path can be reused. Spine/Leaf switches fill the Rate Signal on a per-QP basis when forwarding data packets, and the receiver returns the per-QP advised rate to the sender via a reverse feedback loop. This approach provides fine-grained control and is suitable for scenarios where the end-to-end protocol stack is controllable.

Deployment across Intelligent Computing Data Centers For interconnection of distributed intelligent computing clusters, multiple intelligent computing data centers are interconnected via WAN links (e.g., dedicated lines, SRv6 tunnels) to form a cross-domain logical intelligent computing pool. With the continuous expansion of large language models, a single data center is restricted by power supply, heat dissipation and physical space. Cross-data center collaborative training has become an inevitable industry demand. For latest progress, Google DeepMind adopted Decoupled DiLoCo to train a 12-billion-parameter model across four regions in the United States, achieving more than 20x faster training speed compared with traditional synchronous methods. In this deployment model, data center gateways or WAN edge routers act as aggregation nodes for RoCEv2 traffic. The Round-Trip Time (RTT) of WAN links is much longer than that inside data centers, which can reach tens of milliseconds. The long RTT leads to delayed rate reduction. In extreme cases, when the RTT is 20 ms and the packet loss rate is 0.1%, the throughput may drop to nearly zero . Deploying SCONE-enabled network nodes on central gateways or WAN edge routers to generate Rate Advice provides valuable reference for the transmission rate of endpoints. Two deployment modes are available:

Near-source Gateway Generation: The source gateway sends Rate Advice to senders, reflecting the congestion status of the WAN ingress path.
Remote Gateway Generation: The WAN egress gateway sends backpressure signals to the source gateway, and then the source gateway forwards Rate Advice to senders.

For Solution 1 (path-granularity out-of-band UDP), the network node (DCI gateway or WAN edge router) directly constructs a UDP Rate Advice message and sends it to the sender. The advice is identified by the source IP address and applies to all active QPs originating from that source IP. The feedback latency is single-hop level (sub-RTT) and is not affected by the long inter-DC RTT. This solution is particularly advantageous in inter-DC scenarios where per-QP state is costly to maintain. For Solution 2 (per-QP on-path endpoint feedback), the sender carries a Rate Signal in each data packet. The network node along the path updates the per-QP advised rate based on egress queue status, and the receiver returns it to the sender via a feedback loop. This solution provides precise per-QP control and is suitable for scenarios where the end-to-end protocol stack is controllable and the data path is preferred for reuse.

Overview of Rate Advice Framework This section describes the overall architecture of the Rate Advice framework and defines two implementation solutions. The Rate Advice mechanism introduces a signaling path from network nodes to senders. Depending on whether the signaling is separated from the data flow, two implementation solutions are defined.

Solution 1: Path-Granularity Out-of-Band UDP Rate Advice

Out-of-Band UDP Rate Advice Signaling Path +----------+ +-----------+ +----------+ | <---------> Network <---------> | | Sender | RoCEv2 | Element | RoCEv2 | Receiver | | | Data | | Data | | +-----^----+ +-----+-----+ +----------+ | | | Rate Advice Msg | +--------------------+ The network node (switch, DCI gateway, or WAN edge router) generates the Rate Advice locally, constructs an independent UDP Rate Advice packet, and sends it directly to the sender. This packet is separated from the RoCEv2 data flow (out-of-band), does not modify the RoCEv2 BTH, and does not rely on the receiver for forwarding. Key characteristics:

Path granularity: The advised rate is identified by the source IP address and applies to all active QPs originating from that source IP. No per-QP state is required at the network node.
The network node directly sends a UDP packet to the sender. The feedback path is "network node to sender" single-hop.
In inter-DC links, the feedback latency is sub-RTT level and is not constrained by the end-to-end long RTT.
The packet's destination address is the sender's IP address, and the destination UDP port is a newly registered IANA port (TBD), distinct from the RoCEv2 data port 4791.
The solution is entirely within the IETF UDP/IP layer, maintaining a clear standards ownership boundary.

Applicable scenarios: Inter-DC intelligent computing interconnection, DCI gateway aggregation points, and scenarios where per-QP state is costly to maintain and a lightweight, path-level guidance is sufficient.

Solution 2: Per-QP On-Path Endpoint Feedback

On-Path Endpoint Feedback Rate Advice Signaling Path +--------+ +---------+ +----------+ | RoCEv2 | | Network | | RoCEv2 | | Sender | | Element | | Receiver | +---+----+ +----+----+ +----+-----+ | | | +--- SCONE(?) --->| SCONE(advice) | | +RoCEv2 +--- +RoCEv2 --->| | | | Validate RoCEv2 Packet | | | and record advice |<----------------+----------------+ | Rate Advice Msg | The sender carries a Rate Signal field in RoCEv2 data packets (initially set to "no advice" or a probe value). Network nodes along the path fill or update the Rate Signal based on egress queue status or policy when forwarding the packet. Upon receiving the packet with the Rate Signal, the receiver returns the advised rate to the sender via a reverse feedback loop. Key characteristics:

Per-QP granularity: Each QP receives its own advised rate. The QP association is derived from the Base Transport Header (BTH) of the RoCEv2 packet: the SrcQP field in the forward data packet identifies the sender's QP, and the DestQP field in the feedback packet identifies the same QP at the sender.
The Rate Advice is carried along the data path (on-path), written by network nodes into data packets on a per-QP basis.
The receiver acts as the endpoint of the feedback loop, returning the per-QP advised rate to the sender via a reverse message (e.g., ACK, CNP, or a dedicated feedback packet).
The sender adjusts the transmission rate of each QP independently based on the per-QP advised rate in the feedback loop.
The solution can reuse the existing RoCEv2 data path and BTH extension mechanism without requiring a new UDP port.

Feedback loop design (referencing ):

The sender carries a SCONE Rate Signal field in transmitted data packets (e.g., in a BTH extension header).
When forwarding, a network node encodes the per-QP advised rate into the Rate Signal field if congestion or policy rate limiting is detected for that QP.
Upon reception, the receiver carries the per-QP advised rate in a reverse feedback message (e.g., ACK/CNP/SCONE feedback packet).
The sender parses the feedback message, obtains the per-QP advised rate, and adjusts the transmission rate of that QP.

Applicable scenarios: Scenarios where fine-grained per-QP control is required, the end-to-end protocol stack is controllable, and the data path is preferred for reuse; suitable for intra-DC and inter-DC compatible deployments.

Rate Advice Message This section defines the message formats for the two implementation solutions.

Solution 1: Path-Granularity Out-of-Band UDP Rate Advice Packet

UDP Encapsulation The Rate Advice packet is sent as an independent UDP packet, encapsulated as follows: Ethernet Header | IP Header | UDP Header (dport = TBD) | Rate Advice Payload

UDP Destination Port: A newly registered IANA port (TBD), dedicated to SCONE-RoCEv2 Rate Advice, distinct from the RoCEv2 data port 4791.
UDP Source Port: Assigned by the network node.
IP Destination Address: The sender's IP address (obtained from the source IP of the original RoCEv2 data packet).

Payload Format SCONE-RoCEv2 Rate Advice Payload { Version (32), Rate (32), Source IP (32/128), Reserved (32), } Field definitions:

Version (32 bits): Version number. The initial value is 0x00000001.
Rate (32 bits): The advised rate, measured in Mbps. This value applies to all active QPs originating from the Source IP address.
Source IP (32/128 bits): Path identifier, i.e., the sender's IP address, used by the sender for matching. The advised rate is applied to all QPs originating from this source IP.
Reserved (32 bits): Reserved. The sender sets it to 0; the receiver ignores it.

Note that this payload does not carry a QP ID, since the advised rate is path-granularity (source-IP-based) and is shared by all QPs under the same source IP.

Sending and Processing The network node determines the sender's IP address based on the path identifier, constructs the packet, and sends it. The sender listens on the UDP port, matches the Source IP, and uses the Rate as the global rate upper limit shared by all active QPs under that source IP.

Solution 2: Per-QP On-Path Rate Signal with Endpoint Feedback

Rate Signal Bearer The Rate Signal can be carried in one of the following locations:

BTH Extension Header: A new extension field is added after the RoCEv2 BTH, carrying the per-QP Rate Signal (e.g., 6-bit or 32-bit encoding).
IP Header Reserved Field: Reuse reserved bits in the IP header or fields other than DSCP/ECN (to be used with caution to avoid conflicts with existing semantics).
RoCEv2 Packet Payload Prefix: A Rate Signal extension is inserted between the BTH and the Payload.

The recommended approach is the BTH Extension Header. A new BTH extension type RATE_SIGNAL is defined as follows: Rate Signal Extension { Type (8) = RATE_SIGNAL, Length (8), Rate (32), Version (32), }

Feedback Loop

The sender carries the Rate Signal extension in each data packet. The initial Rate can be set to 0 (indicating no advice) or a probe value. The QP to which the advice applies is identified by the SrcQP field in the BTH of the data packet.
When forwarding, the network node updates the Rate field of the Rate Signal for the corresponding QP based on egress queue status or policy.
Upon reception, the receiver carries the per-QP Rate value in a reverse feedback message. The QP association in the feedback message is provided by the DestQP field in the BTH of the feedback packet. The reverse feedback message can be:
- An extended CNP packet (with a Rate field added to the CNP Payload);
- A dedicated SCONE feedback packet over UDP without a BTH; in this case, the Rate Signal Extension MUST explicitly carry an optional Destination QP ID field;
- A reserved field of an existing ACK/confirmation message, where the QP association is derived from the BTH DestQP.
The sender parses the reverse feedback message, obtains the per-QP advised rate for the QP identified by the BTH DestQP, and updates the rate upper limit of that QP.

Relationship with SCONE Protocol The feedback loop of Solution 2 follows the basic model of the SCONE protocol:

Network elements set the per-QP Rate Signal in data packets;
The endpoint (receiver) returns the per-QP advice to the sender via feedback;
The sender uses the per-QP advice as a reference for rate adjustment of that QP.

The difference from SCONE over QUIC is that the RoCEv2 feedback loop can reuse CNP or RoCEv2 confirmation messages instead of QUIC ACK frames, the Rate Signal is carried in a BTH extension rather than the QUIC long header, and the advice is applied per QP rather than per path.

Acquisition of Advised Rate The advised rate can be obtained in two ways: acquiring the advised rate or rate upper limit according to the rate policies configured on network elements, or calculating the value based on the egress queue depth, queuing delay and packet loss rate of network elements.

Rate Policy Based on Network Element Configuration Network nodes directly obtain the advised rate or rate upper limit according to pre-configured rate policies set by administrators, such as per-path (source-IP-based) rate limiting, per-QP rate limiting, and priority bandwidth allocation. This method applies to scenarios where operators have explicit Service Level Agreement (SLA) and traffic engineering policies.

Calculation Based on Egress Bandwidth Congestion Status Network nodes calculate the advised rate based on the real-time status of local egress queues, including queue depth, queuing delay and packet loss rate. This method is applicable to Rate Advice scenarios that require real-time response to network congestion. For Solution 1 (path-granularity), the advised rate is computed from the aggregate behavior of all active QPs under the same source IP, and is encapsulated into a UDP packet sent to the sender. For Solution 2 (per-QP granularity), the advised rate is computed on a per-QP basis, encoded as a Rate Signal into passing data packets, and returned via a feedback loop. The calculation logic (policy-based or congestion-based) is common to both solutions; the granularity and the signaling bearer differ.

Sender-Side Rate Coordination TBD.

Security Considerations

Solution 1: UDP Rate Advice packets may be forged, causing the sender to be tricked into reducing the rate of all QPs under a given source IP. It is recommended to use a shared key between the network node and the sender or rely on trusted domain boundaries. The sender can perform sanity checks on the Rate value (e.g., not lower than the minimum rate).
Solution 2: The Rate Signal is modified by network nodes along the data path on a per-QP basis, and man-in-the-middle tampering must be prevented. The RoCEv2 ICRC or upper-layer authentication mechanisms can be combined. The receiver should verify the legitimacy of the Rate Signal source before feedback, and the sender should validate the DestQP in the BTH of the feedback message against its local QPs to prevent the advised rate from being applied to the wrong QP.

Further security considerations are TBD.

IANA Considerations

Solution 1: IANA is requested to allocate a new UDP port number (TBD) for SCONE-RoCEv2 Out-of-Band Rate Advice.
Solution 2: If the BTH extension type is used, IANA/IBTA-related registration is requested to allocate a new BTH extension type value (to be coordinated with IBTA). If existing CNP extension fields are reused, no new registration is required.

Contributors The following people have substantially contributed to this document: China Mobile

Beijing China yanghongwei@chinamobile.com

China Mobile

Beijing China lizhiqiangyjy@chinamobile.com