Internet-Draft Fast Notification for Multi-Plane October 2026
Liu, et al. Expires 29 April 2027 [Page]
Workgroup:
FANN
Internet-Draft:
draft-lqg-fann-multi-plane-protection-00
Published:
Intended Status:
Standards Track
Expires:
Authors:
Y. Liu
ZTE Corporation
F. Qin
China Mobile
C. Gao
ZTE Corporation

Fast Network Notification for Plane Protection in Multi-Plane Networks

Abstract

This document discusses the use of fast network notifications for plane evacuation in multi-plane networks.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 29 April 2027.

▲

Table of Contents

1. Introduction

Multi-plane networking is one of the common design choices in AI data centers. A high-capacity NIC is partitioned into multiple interfaces at port or sub-port granularity, and each interface is attached to an independent network fabric, referred to as a plane. [I-D.xu-idr-fare-in-mpson] describes such a topology for RoCE NICs in scale-out networks, and Multipath Reliable Connection (MRC) [MRC] uses multiple parallel planes with per-packet spraying and source- controlled path selection.

When a plane becomes unreachable or degraded, the following methods are used to infer the failure:

Local detection covers the NIC's own attachment well, but by construction does not extend to the remote side of a plane. Probing and transport timeouts observe the path end-to-end and therefore operate at RTT scale. Routing convergence detects and repairs reachability reliably, but completes on control-plane timescales.

This document discusses the use of fast network notifications for plane protection in multi-plane networks. In-plane path switching or local repair performed by the switches within a plane is out of scope.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Plane:
An independent forwarding domain consisting of a set of network nodes and links, with no links to any other plane.
Plane Attachment Node:
A network node in a plane that is directly connected to one or more NIC interfaces. In a Clos-based deployment, a plane attachment node is a leaf switch.
Plane-Internal Node:
A network node in a plane that is not directly connected to any NIC interface. In a Clos-based deployment, these are spine or super-spine switches.
In-Plane Repair:
A forwarding or path adjustment performed inside a plane by the nodes of that plane.
Plane Evacuation:
The action taken by a source NIC to move traffic away from an unreachable or degraded plane to one or more other planes.
Plane Exclusion:
The state in which a plane is removed from the candidate plane set for a given destination.
Source NIC:
The NIC that originates traffic and performs plane selection for that traffic.
Destination NIC:
The NIC that receives traffic in a given flow.

3. Multi-Plane Failure Scenarios

This section describes two representative failure scenarios in a multi-plane network.

           Plane 1           Plane 2           Plane 3
   +---------------+   +---------------+   +---------------+
   |      +--+     |   |      +--+     |   |      +--+     |
   |      |S1|     |   |      |S1|     |   |      |S1|     |
   |      +--+     |   |      +--+     |   |      +--+     |
   |     /    \    |   |     /    \    |   |     /    \    |
   |   +--+  +--+  |   |   +--+  +--+  |   |   +--+  +--+  |
   |   |L1|  |L2|  |   |   |L1|  |L2|  |   |   |L1|  |L2|  |
   |   +--+--+--+  +   |   +--+--+--+  +   |   +--+--+--+--+
   +-------|-------+   +-------|-------+   +-------|-------+
           |                   |                   |
           |                   |                   |
           |                   |                   |
   +-------|-------------------|-------------------|--------+
   |                          NIC1                          |
   +--------------------------------------------------------+
Figure 1: Example Multi-Plane Topology

Figure 1 illustrates an example multi-plane topology in a data center. Each plane is a two-tier Clos fabric with spine and leaf switches. A NIC is attached to a leaf switch in every plane. The figure shows three planes for illustration.

Figure 2 shows a single plane with two leaf switches and two spine switches. Leaf1 is attached to NIC N1 and N2. Leaf2 is attached to NIC N3 and N4. Leaf and spine switches are fully meshed.

                         +------+       +------+
                         |Spine1|       |Spine2|
                         +--+---+       +---+--+
                            |    \     /    |
                            |     \   /     |
                            |      \ /      |
                            |       X       |
                            |      / \      |
                            |     /   \     |
                         +--+---+     +---+--+
                         |Leaf1 |     |Leaf2 |
                         +-+--+-+     +--+-+-+
                           |  |          | |
                    +------+  +---+      + +----+
                    |             |      |      |
                  +-+--+        +-+--+ +-+--+ +-+--+
                  | N1 |        | N2 | | N3 | | N4 |
                  +----+        +----+ +----+ +----+
Figure 2: NIC-to-Leaf Link Failure in a Plane

In Plane 1, the link between Leaf2 and N4 fails. As a result, other NICs in Plane 1 cannot reach N4 through this plane. Within this plane, N4 is reachable only through Leaf2. Therefore, an in-plane forwarding adjustment, such as Leaf2 notifying a spine to change its path, cannot solve the problem: the spine can still reach Leaf2, but Leaf2 has lost its link to N4. The source NICs need to be informed so that they can evacuate traffic destined to N4 from this plane.

3.2. Scenario 2: Leaf-to-Spine Link Failure

Figure 3 shows a single plane with three leaf switches and two spine switches. Leaf1 is attached to N1 and N2. Leaf2 is attached to N3 and N4. Leaf3 is attached to N5 and N6. Leaf and spine switches are fully meshed.

                         +------+                       +------+
                         |Spine1|                       |Spine2|
                         +--+---+                       +---+--+
                            |                               |
                            | \                           / |
                            |   \                       /   |
                            |     \                   /     |
                            |       \               /       |
                            |         \           /         |
                            |           \       /           |
                            |             \   /             |
                            |              X                |
                            |             / \               |
                            |           /     \             |
                            |         /         \           |
                            |       /             \         |
                            |     /                 \       |
                            |   /                     \     |
                            | /                         \   |
                       +----+----+       +-------+       +--+-----+
                       |  Leaf1  |       | Leaf2 |       |  Leaf3 |
                       +----+----+       +---+---+       +--+-----+
                            |                |                |
                  +---------+----+      +----+----+      +----+---------+
                  |              |      |         |      |              |
                +-+--+         +-+--+ +-+--+    +-+--+ +-+--+         +-+--+
                | N1 |         | N2 | | N3 |    | N4 | | N5 |         | N6 |
                +----+         +----+ +----+    +----+ +----+         +----+
Figure 3: Leaf-to-Spine Link Failure in a Plane

In Plane 1, all links between Leaf1 and the spine switches fail. N1 and N2 can still communicate with each other through Leaf1, but N3 through N6 cannot reach N1 or N2 through this plane, and N1 and N2 cannot reach N3 through N6 through this plane. The failure is bidirectional: traffic from N1/N2 to remote NICs and traffic from remote NICs to N1/N2 are both affected. The detection points are Leaf1 and the spines, but Leaf1 can no longer send any packet to remote nodes within this plane, and remote nodes do not know that N1/N2 are unreachable through this plane.

3.3. Scenario 3: Plane Performance Degradation

In addition to hard failures, the source NIC may need to be informed about performance degradation at the plane granularity, where the plane is still reachable, but it is no longer optimal for carrying the affected traffic. For example, a link inside the plane may experience severe congestion or microbursts, causing high latency and packet loss, while the paths remain physically connected.

The topology in this case is the same as in Figure 2 or Figure 3. However, unlike Scenario 1 and Scenario 2 where the links are physically down, here the links remain operationally up, but the available capacity is reduced or the path quality is degraded.

3.4. Scenario Observations

The failure scenarios share the following characteristics:

  • The failure is scoped to a single plane, in-plane forwarding adjustment cannot resolve the failure.
  • The impact of a single failure is set-level and bidirectional: the failure affects a set of NICs attached to the same leaf switch, and it affects both directions: the NICs in the set cannot reach remote NICs through the plane, and remote NICs cannot reach the set through the plane.
  • Detection and mitigation are performed by different nodes, and the detector cannot always reach the nodes that need to act.

4. Requirements

In addition to the considerations already described in the FANN documents [I-D.ietf-fann-problem-statement], the multi-plane evacuation scenario imposes the following requirements.

4.1. Scope Requirements

R-S1: A consumer MUST be able to determine, from the notification message, which plane the failure occurred in.

R-S2: A consumer MUST be able to distinguish, from the notification message, whether the message is a plane failure notification or an ordinary link failure notification.

R-S3: A plane-scoped notification MUST NOT leak across planes. When propagated across planes, the content MUST preserve the semantics of "about one plane only".

R-S4: In addition to plane failure notifications, plane degradation notifications MAY be supported. In this case, a consumer MUST be able to distinguish three states: plane unreachable, plane degraded, and destination unreachable within a plane.

R-S5: The information elements of the notification MUST be sufficient to drive plane-granular exclusion or quarantine, rather than only path-granular adjustment.

4.2. Consumer and Action Requirements

R-C1: The consumers of the notification MUST include at least the source NIC of the affected flow.

4.3. Delivery Path Requirements

R-D1: The delivery path of the notification MUST NOT share fate with the failed link or the failed plane uplink.

R-D2: The solution MUST explicitly address the case where the in- plane path from the detection point to some consumers has been cut. For example, the notification may be delivered through another plane or an out-of-band channel, or the affected consumers may be required to rely on other mechanisms such as local end-to- end probing or routing convergence.

5. Security Considerations

TBD.

6. IANA Considerations

This document has no IANA actions.

7. References

7.1. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.

7.2. Informative References

[I-D.ietf-fann-problem-statement]
Dong, J., Ed., McBride, M., Ed., and F. Clad, Ed., "Fast Network Notifications Problem Statement", Work in Progress, Internet-Draft, draft-ietf-fann-problem-statement-00, , <https://datatracker.ietf.org/doc/html/draft-ietf-fann-problem-statement-00>.
[I-D.xu-idr-fare-in-mpson]
Xu, X., Patel, K., Wang, H., Sun, W., and G. Ma, "Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks", Work in Progress, Internet-Draft, draft-xu-idr-fare-in-mpson-01, , <https://datatracker.ietf.org/doc/html/draft-xu-idr-fare-in-mpson-01>.
[IBTA-ROCEv2]
InfiniBand Trade Association, "InfiniBand Architecture Specification Volume 1, Release 1.4", , <https://www.infinibandta.org/ibta-specification/>.
[MRC]
Araujo, J., Chow, A., Handley, M., Lu, G., and L. Yuan, "Resilient AI Supercomputer Networking using MRC and SRv6", , <https://arxiv.org/abs/2605.04333>.

Authors' Addresses

Yao Liu
ZTE Corporation
Fengwei Qin
China Mobile
Chenqiang Gao
ZTE Corporation