Fast Network Notification for Plane Protection in Multi-Plane Networks

Introduction Multi-plane networking is one of the common design choices in AI data centers. A high-capacity NIC is partitioned into multiple interfaces at port or sub-port granularity, and each interface is attached to an independent network fabric, referred to as a plane. describes such a topology for RoCE NICs in scale-out networks, and Multipath Reliable Connection (MRC) uses multiple parallel planes with per-packet spraying and source- controlled path selection. When a plane becomes unreachable or degraded, the following methods are used to infer the failure:

Local detection at the NIC, e.g., link down or persistent link- level errors on the local port.
Active path health management, where the NIC continuously measures loss and latency across its paths using in-band measurements of data traffic and active probes, and steers traffic away from degraded paths, as done by MRC .
Transport timeout or retransmission timeout, e.g., the RDMA timeout and retransmission mechanisms of RoCEv2 , which is end-to-end and RTT-bound.

Local detection covers the NIC's own attachment well, but by construction does not extend to the remote side of a plane. Probing and transport timeouts observe the path end-to-end and therefore operate at RTT scale. Routing convergence detects and repairs reachability reliably, but completes on control-plane timescales. This document discusses the use of fast network notifications for plane protection in multi-plane networks. In-plane path switching or local repair performed by the switches within a plane is out of scope.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Plane:: An independent forwarding domain consisting of a set of network nodes and links, with no links to any other plane.
Plane Attachment Node:: A network node in a plane that is directly connected to one or more NIC interfaces. In a Clos-based deployment, a plane attachment node is a leaf switch.
Plane-Internal Node:: A network node in a plane that is not directly connected to any NIC interface. In a Clos-based deployment, these are spine or super-spine switches.
In-Plane Repair:: A forwarding or path adjustment performed inside a plane by the nodes of that plane.
Plane Evacuation:: The action taken by a source NIC to move traffic away from an unreachable or degraded plane to one or more other planes.
Plane Exclusion:: The state in which a plane is removed from the candidate plane set for a given destination.
Source NIC:: The NIC that originates traffic and performs plane selection for that traffic.
Destination NIC:: The NIC that receives traffic in a given flow.

Multi-Plane Failure Scenarios This section describes two representative failure scenarios in a multi-plane network.

Example Multi-Plane Topology Figure 1 illustrates an example multi-plane topology in a data center. Each plane is a two-tier Clos fabric with spine and leaf switches. A NIC is attached to a leaf switch in every plane. The figure shows three planes for illustration.

Scenario 1: NIC-to-Leaf Link Failure Figure 2 shows a single plane with two leaf switches and two spine switches. Leaf1 is attached to NIC N1 and N2. Leaf2 is attached to NIC N3 and N4. Leaf and spine switches are fully meshed.

NIC-to-Leaf Link Failure in a Plane In Plane 1, the link between Leaf2 and N4 fails. As a result, other NICs in Plane 1 cannot reach N4 through this plane. Within this plane, N4 is reachable only through Leaf2. Therefore, an in-plane forwarding adjustment, such as Leaf2 notifying a spine to change its path, cannot solve the problem: the spine can still reach Leaf2, but Leaf2 has lost its link to N4. The source NICs need to be informed so that they can evacuate traffic destined to N4 from this plane.

Scenario 2: Leaf-to-Spine Link Failure Figure 3 shows a single plane with three leaf switches and two spine switches. Leaf1 is attached to N1 and N2. Leaf2 is attached to N3 and N4. Leaf3 is attached to N5 and N6. Leaf and spine switches are fully meshed.

Leaf-to-Spine Link Failure in a Plane In Plane 1, all links between Leaf1 and the spine switches fail. N1 and N2 can still communicate with each other through Leaf1, but N3 through N6 cannot reach N1 or N2 through this plane, and N1 and N2 cannot reach N3 through N6 through this plane. The failure is bidirectional: traffic from N1/N2 to remote NICs and traffic from remote NICs to N1/N2 are both affected. The detection points are Leaf1 and the spines, but Leaf1 can no longer send any packet to remote nodes within this plane, and remote nodes do not know that N1/N2 are unreachable through this plane.

Scenario 3: Plane Performance Degradation In addition to hard failures, the source NIC may need to be informed about performance degradation at the plane granularity, where the plane is still reachable, but it is no longer optimal for carrying the affected traffic. For example, a link inside the plane may experience severe congestion or microbursts, causing high latency and packet loss, while the paths remain physically connected. The topology in this case is the same as in Figure 2 or Figure 3. However, unlike Scenario 1 and Scenario 2 where the links are physically down, here the links remain operationally up, but the available capacity is reduced or the path quality is degraded.

Scenario Observations The failure scenarios share the following characteristics:

The failure is scoped to a single plane, in-plane forwarding adjustment cannot resolve the failure.
The impact of a single failure is set-level and bidirectional: the failure affects a set of NICs attached to the same leaf switch, and it affects both directions: the NICs in the set cannot reach remote NICs through the plane, and remote NICs cannot reach the set through the plane.
Detection and mitigation are performed by different nodes, and the detector cannot always reach the nodes that need to act.

Requirements In addition to the considerations already described in the FANN documents , the multi-plane evacuation scenario imposes the following requirements.

Scope Requirements R-S1: A consumer MUST be able to determine, from the notification message, which plane the failure occurred in. R-S2: A consumer MUST be able to distinguish, from the notification message, whether the message is a plane failure notification or an ordinary link failure notification. R-S3: A plane-scoped notification MUST NOT leak across planes. When propagated across planes, the content MUST preserve the semantics of "about one plane only". R-S4: In addition to plane failure notifications, plane degradation notifications MAY be supported. In this case, a consumer MUST be able to distinguish three states: plane unreachable, plane degraded, and destination unreachable within a plane. R-S5: The information elements of the notification MUST be sufficient to drive plane-granular exclusion or quarantine, rather than only path-granular adjustment.

Consumer and Action Requirements R-C1: The consumers of the notification MUST include at least the source NIC of the affected flow.

Delivery Path Requirements R-D1: The delivery path of the notification MUST NOT share fate with the failed link or the failed plane uplink. R-D2: The solution MUST explicitly address the case where the in- plane path from the detection point to some consumers has been cut. For example, the notification may be delivered through another plane or an out-of-band channel, or the affected consumers may be required to rely on other mechanisms such as local end-to- end probing or routing convergence.

Security Considerations TBD.

IANA Considerations This document has no IANA actions.