]> CMCC

No. 58 Kunlunshan Road Suzhou 215000 China lluozy@yeah.net

CMCC

No. 58 Kunlunshan Road Suzhou 215000 China yanhaishuang_yewu@cmss.chinamobile.com

Applications Network Working Group Internet-Draft This document describes an experimental protocol, the Available Session Recovery Protocol (ASRP), for high-availability clusters that provide stateful network services - load balancing, Network Address Translation (NAT ), and other layer-4 stateful network functions, including their virtualized (NFV) forms. ASRP backs up the session state at the session's own endpoints, so that the cluster nodes hold no replicated state: no inter-node synchronization, no backup node, and no state warm-up for a newly added node. The document defines the procedures and message formats for session backup and recovery. In contrast to traditional high-availability techniques that back up session state within the cluster itself, the core innovation of ASRP lies in its distributed backup of state information to the client or server side. This approach offers multiple advantages: theoretically unlimited elastic scaling capacity; support for rapid recovery from multi-point failures; reduction of resource redundancy through the elimination of centralized backup nodes; and significant simplification of cluster implementation complexity. The ASRP protocol provides a standardized method for constructing elastic service clusters, facilitating broader participation from software and hardware developers in building elastic cloud network service clusters.

Introduction Many network functions are stateful: load balancers, NAT gateways, connection trackers, and other layer-4 network functions, in physical and virtualized (NFV) forms alike, each maintain per-session state for the traffic they forward, and are deployed at scale in every data center and enterprise edge. That state is the root cause of the complexity of operating these services at scale - it is what master/backup replication, intra-cluster session synchronization, and failover machinery all exist to protect. This document takes a different starting point: the state of a session does not need to live in the cluster at all. The Available Session Recovery Protocol (ASRP) backs up the session state at the session's own endpoints - the client or the server - so that the cluster nodes hold no replicated state: no inter-node synchronization, no backup node, and no state warm-up for a newly added node; a node that receives a packet of a session it does not hold recovers the session from the endpoint on demand. The protocol is not specific to load balancing or NAT. Any network function that maintains per-session state can use ASRP to keep that state at the endpoints and gain the same properties: sessions survive node restarts, failovers, and elastic scaling without inter-node machinery. The availability boundary is the same as that of conventional high-availability designs - ASRP protects against failures of the network function itself; the failure of an endpoint takes down its sessions under any design.

Comparison with Existing Approaches High-availability mechanisms evolved together with the deployment model they serve. The industry's answer for elastically scaling deployments is the Elastic Stateful Cluster (ESC): a group of network nodes, all active, that simultaneously provide the same stateful network function, such as load balancing (SLB), Network Address Translation (NAT), and other layer-4 functions. In such a cluster the hard problem is session state: a design has to decide where the state lives, and how a node that receives a packet of a session it does not hold recovers that session. Master/backup HA (e.g., VRRP ) solved the failover problem of the master/backup era: it provides fast takeover of a shared address, with session continuity typically left to private synchronization between the active node and its backup (e.g., the LVS connection synchronization daemon, DPVS). The drawback is the single-active model: for a given service address, exactly one node forwards while the others stand by, so a service is capped at the capacity of one node, the standby nodes sit idle, and load cannot be met by adding nodes. The conventional Elastic Stateful Cluster keeps session state inside the cluster: its nodes communicate with each other to synchronize session state, and redundant backup nodes are kept within the cluster for failover. The architecture is typically layered - a Fast/Slow Path design that separates session management from packet forwarding:

| Fast Path Node | <--------> | Server | | | | | +----------------+ | | | | +----------+ | | ... | | +----------+ | | ... | | | +----------------------+ | +--------------------------+ ]]>

The slow path nodes are responsible for session creation and synchronization, while the fast path nodes are responsible for rapid packet forwarding. The drawback is the weak elastic scaling capability of the slow path nodes: new sessions must be propagated to other nodes so that asymmetric traffic can be handled anywhere, the synchronization traffic grows with the session creation rate and the node count, and a newly added node starts without state. A typical implementation reference is the AWS Hyperplane NFV platform. Stateless scheduling (e.g., Maglev-style consistent hashing, ECMP ) avoids session state on the nodes entirely by hashing each flow to a node. Node failure is handled by rehashing flows, which works only when the service can tolerate a flow landing on a node that holds no state for it; stateful services such as NAT cannot. These mechanisms are what make elastic clusters work in practice, but they are private. The contrast with the previous generation is telling: the master/backup architecture is served by a standard (VRRP), while the Elastic Stateful Cluster - the architecture that large deployments actually use today - has no standard protocol for its session-state layer. Routing, failure detection, and address takeover are individually standardized; session synchronization and recovery are not, and every production cluster ships its own. ASRP fills this gap by removing the session state from the cluster altogether: the backup of a session is stored at one of the session's own endpoints, nodes share no protocol state with each other, and recovery state is pulled on demand (QS/RS) only when a node actually needs a session it does not have. This structure - state held at an endpoint and recovered against an opaque token - mirrors the pattern proven at Internet scale by TLS session tickets and QUIC address validation tokens . This addresses the hardest problem of the elastic cluster and, as a side effect, simplifies the cluster itself: no inter-node protocol, no synchronization traffic, and no state warm-up for a newly added node. The tradeoff is that the client or server side must support the protocol. Within a single administrative domain - a data center for load balancing, an enterprise edge for SNAT - this is a deployment decision rather than a protocol limitation. summarizes the comparison of master/backup HA with the two Elastic Stateful Cluster designs, conventional and ASRP: Architecture Elasticity Implementation Standard protocol Master/backup HA No Pairwise sync VRRP Conventional cluster Partial Complex None ASRP cluster Yes Simple This document

ASRP Elastic Stateful Cluster

| ASRP Node | <--------> | Server | | | | +----------------+ | | | +----------+ | ... | +----------+ | ... | +----------------------+ ]]>

The Available Session Recovery Protocol (ASRP) provides a standardized method for constructing elastic service clusters, facilitating broader participation from software and hardware developers in building elastic cloud network service clusters. Its core idea is to distribute session state information to the client or server. The lifecycle of the backup state is synchronized with the real session, eliminating the need for independent keepalive and timeout mechanisms. This design ensures the timeliness and availability of the backup information. ASRP defines corresponding session backup and recovery mechanisms. The protocol allows protocol messages to be transmitted together with the original service data packets, thereby reducing control overhead for state synchronization. In an Elastic Stateful Cluster built on ASRP, network nodes possess atomic and mutually independent properties. There is no need for communication between nodes, nor is session synchronization required within the cluster. This fundamental design provides theoretically unlimited scaling capability and supports rapid recovery from multi-point failures.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the following terms: Elastic Stateful Cluster (ESC): A group of network nodes, all active, that simultaneously provide the same stateful network function; any node may receive any flow of the service, and nodes can be added or removed at any time. In this document, "cluster" refers to an ESC. Network Node: A device that provides stateful network services (e.g., load balancing, NAT) and creates and maintains sessions for the traffic it forwards. Client / Server: The two endpoints of a session. Depending on the operational mode, the server (PSV mode) or the client (ACT mode) also holds the backup of the session. Backup Holder: The client or server that stores the backed-up state of a session. Session: The state a network node maintains for a flow, keyed by the flow's 5-tuple. Explicit First Packet: A packet whose characteristics explicitly indicate the start of a session, e.g., a TCP SYN or a DNS query. Inline Message: An ASRP message carried within a packet of the flow itself - at the head of the segment payload marked by the ASRP option, in the datagram's option area, or in a separate packet that copies the original packet header. ASRP option: The transport-layer option that marks the presence of an ASRP message. On TCP it carries no data (a Length of 2) and marks the message at the head of the segment payload; on UDP it carries the message as its option data, in the datagram's option area. Standalone Message: An ASRP message carried in an independent UDP packet whose destination port is ASRP-PORT. ASRP-PORT: The UDP destination port used for standalone ASRP messages.

Protocol Overview

Two Operational Modes For the ASRP protocol to function correctly, two prerequisites must be met. First, all network nodes within the cluster MUST run service software supporting the ASRP protocol. Second, the server or client responsible for backing up sessions MUST deploy a kernel module or an eBPF module that supports ASRP. Depending on whether this module is deployed on the server or the client, the protocol operates in one of two corresponding modes: Passive (PSV) Mode and Active (ACT) Mode.

PSV Mode In PSV mode, the network node is typically deployed in the same network domain as the server (e.g., inside a data center). Load balancing is the typical example; any stateful function serving that domain can use the protocol.

ACT Mode In ACT mode, the network node is typically deployed in the same network domain as the client (e.g., an enterprise intranet). Source NAT is the typical example; any stateful function serving that domain can use the protocol. The operational mode (PSV or ACT) is determined by the deployment configuration of the network node and is not carried in ASRP messages.

Two Routing Behaviors

Symmetric Routing

| node X | <----------> | Server | | | | +------------+ | | | +----------+ | ... | +----------+ +------------------+ ]]>

Symmetric routing refers to the path mode where bidirectional traffic of the same session between a client and a server is always routed to the same node within the cluster.

Asymmetric Routing

| node X | -----------> | | | | | +------------+ | | | | Client | | ... | | Server | | | | +------------+ | | | | | <----------- | node Y | <----------- | | +----------+ | +------------+ | +----------+ | ... | +------------------+ ]]>

Asymmetric routing refers to the scenario where bidirectional traffic of the same session may be routed (e.g., by mechanisms such as ECMP , ) to different nodes within a cluster. In cloud networking environments, asymmetric routing is a common phenomenon, which imposes higher demands on the implementation of elastic stateful clusters.

Protocol Message ASRP achieves distributed backup and recovery of session state information by exchanging specific protocol messages among the client, server, and network nodes (such as load balancers or NAT gateways). In a load-balancing scenario, session state is distributed and backed up to individual servers; in an SNAT scenario, session state is distributed and backed up to individual clients. ASRP defines the following messages: Message Generated by Full name NS Network node New Session NA Backup holder New Session Acknowledge NC Backup holder New Session Conflict QS Network node Query Session RS Backup holder Recover Session RX Backup holder Recovery no-session HS Client (ACT mode) Hello Session PS Server (PSV mode) Push Session The network node sends an NS message to deliver the state of a newly created session to the backup holder - the server in PSV mode, or the client in ACT mode; the backup holder acknowledges it with an NA message, or reports a conflict with an existing local session with an NC message. When a received packet matches no local session and none can be created directly, the network node queries the backup holder with a QS message; the holder answers with an RS message carrying the state required to recover the session, or an RX message when the queried session is not found. In ACT mode, a client sends an HS message to announce its support of the protocol and trigger the network node to return an NS message; in PSV mode, a server sends a PS message to push session state to the network node, e.g. to create or update a session under asymmetric routing.

Transmission Modes ASRP messages can be transmitted in two modes: Inline Message and Standalone Message.

Inline Message The Inline Message is transmitted with the same 5-tuples as the flow. The carriage differs by transport. For TCP flows, the ASRP option is a TCP option with Kind TBD and a Length of 2; it carries no data and marks the presence of the message. The message occupies the leading octets of the segment payload. When the total packet length does not exceed the MTU, NS, HS, and PS messages SHOULD be inserted into the original packet; otherwise, they are carried in a separate packet that copies the original packet header, including the ASRP option, without the original packet data. For UDP flows, the ASRP message is carried as the option data of the ASRP option, a UDP option placed in the datagram's option area. The datagram's own payload is untouched. When the datagram leaves no room for the option within the MTU, the message is carried in a separate packet that copies the original datagram header. HS messages are sent after forwarded packets, and PS messages are sent before forwarded packets. Traveling with the flow's 5-tuple is a structural requirement for HS and PS: both are sent from an endpoint towards the cluster, and under asymmetric routing the sender cannot predict which network node handles the corresponding direction of the flow. Since per-flow scheduling (e.g., ECMP hashing) maps one 5-tuple to one node, a message carried with the flow's 5-tuple - whether inserted into a packet of the flow or in a separate packet copying its header - is delivered to the same node as the flow itself, which is the node that holds (for HS) or needs (for PS) the session. NS, by contrast, is sent from a network node to a known endpoint (the selected server in PSV mode, or the client in ACT mode) and can also be transmitted as a Standalone Message; carrying it inline is a performance optimization that piggybacks on a packet about to be forwarded. Inserting a message into the payload of the first packet of a TCP session makes that packet carry data (e.g., a TCP SYN with a non-empty payload). Some middleboxes drop such packets; this behavior has been observed in the deployment of TCP Fast Open . Where such interference is known or detected on a path, the implementation SHOULD use the separate-packet form instead of inserting into the original packet, since the separate packet carries the message without altering the characteristics of the session's own packets. The UDP form is not affected: an option in the datagram's option area does not change the datagram's payload or its UDP length.

Standalone Message The Standalone Message is transmitted in an independent packet with its own 5-tuples. It is encapsulated using IP/UDP , with IP addresses configured to ensure mutual reachability and a fixed destination port ASRP-PORT (e.g., 51200). QS, RS, RX, NA and NC messages typically use this transmission mode.

Session Creation/Recovery Scenarios This section elaborates on, through a series of typical scenarios, how the ASRP protocol achieves session backup and recovery via message interaction in the event of network node failures under different operational modes. Each scenario details the involved protocol message flows and the processing steps of each entity.

PSV-Scenario-1

| | -----2:NS----> | | | | | | <----3:NC----- | | | client | | ASRP Nodes | | server | | | | | | | | | <--5:PKT--- | | <----4:PS----- | | +----------+ +--------------+ +----------+ ]]>

This scenario describes that, in PSV mode, a network node receives an explicit first packet (i.e., a packet whose characteristics explicitly indicate the start of a session) and directly creates a session flow. Common examples of explicit first packets include TCP SYN and DNS query, among others. The processing flow is as follows: Session Creation: Upon receiving a packet from the client (e.g., TCP SYN), the network node first creates a new session and then sends an NS message to the selected server. Server Response: Upon receiving the NS message, the server stores the session state information and associates it with its local session. If a conflict occurs, it sends an NC message. In the case of asymmetric routing, when sending its first response packet, the server sends a PS message to the network node. Session Recovery: Upon receiving the PS message, the network node restores the session and forwards packets accordingly. The session state information backed up by the server is released upon local session termination, without requiring any additional teardown message.

PSV-Scenario-2

| server | | | | | | | | | | | <---3:RS/RX--- | | +----------+ +--------------+ +----------+ ]]>

This scenario describes the session recovery flow triggered by a server packet. The processing flow is as follows: Session Query: Upon receiving a packet from the server, the network node searches its local session table. If no matching session is found, the node SHOULD first buffer the packet for forwarding, then sends a QS message back to the server. Server Response: After receiving the QS message, the server looks up the backed-up session state information and sends an RS/RX message. Session Recovery: Upon receiving an RS message, the network node creates a new local session and forwards packets accordingly. Upon receiving an RX message, the node SHOULD discard the pending packets.

PSV-Scenario-3

| ... | | | ---1:PKT--> | | <---3:RS/RX--- | +--------+ | | | | | ... | | server | | | | | | ... | +--------+ | | client | | ASRP Nodes | ---4:PKT/NS--> | ... | | | | | | +--------+ | | | | | | | server | | | | <--7:PKT--- | | <--5:NA/NC---- | +--------+ | | | | | <----6:PS----- | ... | +----------+ +------------+ +------------+ ]]>

This scenario describes the session creation/recovery flow triggered by a client packet. The processing flow is as follows: Query Local Session: Upon receiving a packet from the client, if no matching session is found, the network node first obtains a list of candidate servers (possibly multiple) for querying. Query Backup Session: The network node sends QS messages to each candidate server. Each server replies with an RS or RX message. Process Query Results: If a session is found, the network node restores the session and forwards the packet. Otherwise, for TCP packets: drop the packet. For UDP packets: create a new local session and send an NS message to the selected server. Server Creates New Session: Upon receiving an NS message, the server stores the session state information and associates it with its local session. If a conflict occurs, it sends an NC message; otherwise, if the F-NA flag is set, it immediately replies with an NA packet. Session Recovery: In an asymmetric routing environment, when sending its first response packet, the server sends a PS message to restore the session at the network node. In this scenario, obtaining the list of candidate servers is a key challenge. Two solutions are proposed: The network node derives an ordered list of candidate servers from the packet's 5-tuple using a deterministic server-selection algorithm, such as consistent hashing or history-aware consistent hashing, and sends QS messages to the candidates in order. Enhance the client so that client packets carry backend server information, allowing the network node to directly extract the target server from the client packet. One such enhancement is specified in , which carries the selected backend identity in a transport-layer option. ASRP works with either solution and does not depend on the SR option. For the first solution, the specific algorithm is a local implementation choice; what the protocol relies on are three properties of it: Determinism: The mapping from a session's 5-tuple to its candidate list MUST be identical on every node of the cluster, so that any node receiving a packet of a session it does not hold computes the same candidates to query. History awareness: When the backend pool changes (e.g., a server is added), the algorithm SHOULD keep the servers that previously served the affected hash region in the candidate list, so that sessions created before the change can still locate their backup holder. A removed server simply fails to answer; the recovery attempt moves on to the next candidate or is abandoned. Bounded querying: The number of candidates queried for one recovery MAY be limited to a small constant, bounding the QS traffic and the recovery latency. When the limit is reached without an RS response, the node stops querying and falls back to its local handling of packets without a session ().

ACT-Scenario-1

| | | | | | <----2:NS----- | | ---3:PKT--> | | | client | | ASRP Nodes | | server | | | <----5:QS----- | | <--4:PKT--- | | | | ---6:RS/RX---> | | | | +----------+ +--------------+ +----------+ ]]>

This scenario describes session creation at a network node and server-initiated session restoration. The processing flow is as follows: Session Backup: When sending a packet without receiving an NS message, the client sends an HS message to the network node to request session backup (the client MAY rate-limit the HS messages it generates). Upon receiving the HS message, the node replies with an NS message. Session Lookup: For packets from a server with no matching session, the network node identifies the target client and sends it a QS message. Session Recovery: Upon receiving QS, the client sends an RS/RX message to the network node to restore the session. In step 2, identifying the target client is challenging. Two solutions are proposed: Use static mapping (e.g., map destination port to client). For SNAT, client IP addresses can be statically mapped to distinct port ranges. Enhance the server to embed client information in its packets, allowing the network node to extract the client address directly.

ACT-Scenario-2

| | | | | | | | | | | client | <----2:QS----- | ASRP Nodes | ---4:PKT--> | server | | | | | | | | | ---3:RS/RX---> | | | | +----------+ +--------------+ +----------+ ]]>

This scenario describes the client-packet-triggered session recovery. The processing flow is as follows: Session Query: Upon receiving a packet from a client with no local session and without an HS message, the network node sends a QS message to the client. Session Recovery: The client responds with an RS/RX message; the RS message enables session restoration.

State Consistency ASRP does not need a distributed consistency protocol, because the session state has a single writer at any time and the backup is a read-only copy. The backup of a session is created by the NS message at session creation and is never modified afterwards; it is released when the local session terminates. It therefore reflects the session state at creation time. For the state that ASRP targets - the flow-defining state of a session, keyed by its 5-tuple - this snapshot remains valid for the lifetime of the session, since the key does not change. Implementations that maintain additional mutable state (e.g., timeouts or counters) recover a usable approximation of it. Recovery is a local action of one network node: the node rebuilds the session from the backup delivered in an RS message and thereafter owns the session. If, under asymmetric routing, two nodes receive packets of the same session and each recovers it independently, both hold the session locally and each forwards the packets that reach it - which is exactly how the cluster treats any session it holds. No node writes to another node's session state, so there is no split-brain condition to resolve. In this sense, a node's session table is a cache rather than a source of truth. A session that a node does not hold locally can always be rebuilt from the endpoint backup - including the complete address-and-port mapping that a full-NAT load balancer maintains - so nothing is lost when a table is dropped, a node restarts, or traffic shifts to a new node. If recovery fails (an RX response, or no response before the recovery attempt is abandoned), the network node falls back to its local handling of packets without a session, e.g., discarding them or creating a new session.

Protocol Details

Message Format An ASRP message consists of a fixed 12-octet header followed by a variable-length body. The header comprises the first eight fields below; the body comprises the Session-Tuple and Session-Data fields and, for the NS message, the Node-Local-IP field. The fields are defined as follows:

Rules for validating these fields on receipt are specified in "Message Validation".

Message Header Every ASRP message begins with the following 12-octet header:

Session Tuples ST4 (length 12) Format:

ST6 sequence) ST64(length 48) Format: Mixed ST pair(ST6->ST4 sequence) ]]>

For messages carrying two Session-Tuples (NS, NA, NC, RS, and PS), the first Session-Tuple (ST1) MUST be the client-side tuple, and the second Session-Tuple (ST2) MUST be the server-side tuple. For messages carrying a single Session-Tuple (QS and RX), the Session-Tuple MUST be the 5-tuple of the triggering packet.

NS Message The NS message contains a Node-Local-IP field and two Session-Tuples. The Node-Local-IP field is located at the end of the NS message and carries the local IP address of the network node that sends the NS message. Its length is 16 octets when the F-LIP6 flag is set, and 4 octets otherwise. The client or server uses this address as the destination IP to send the NA message back to the network node. The body of the NS message follows the header:

The two Session-Tuples represent the network node's connections with the client and the server, respectively.

QS Message The body of the QS message follows the header:

HS Message The HS message consists of the message header followed by a 4-octet Reserved field.

Other Messages The NA message has the same structure as the NS message, except that the Node-Local-IP field is not present. The NC, RS, and PS messages have the structure of the NA message. In the NC message, the client-side Session-Tuple remains unchanged, while the server-side Session-Tuple may be modified to reflect the conflicting session; the F-CC flag indicates whether the conflict is on the client side or the server side. The RX message has the structure of the QS message.

ASRP Packets A packet that carries a single ASRP message is referred to as an ASRP packet. An inline-ASRP packet is a packet of the flow itself: how the message is placed in it - at the head of the segment payload marked by the ASRP option, or in the datagram's option area - is specified in . A standalone-ASRP packet is an independent packet with its own 5-tuples, encapsulated using IP/UDP with the destination port ASRP-PORT; its payload is the message.

Transport-Layer Transparency An inline ASRP message exists only on the wire between the sending side's ASRP module (which inserts it) and the receiving side's ASRP module (which removes or reads it). Neither the application nor either transport stack observes it. On a TCP flow, an inline message is never part of either endpoint's TCP sequence space: the sender's transport stack generated the segment without it, and the receiver's transport stack processes the segment without it. The receiving ASRP module removes the message from the head of the payload and the ASRP option from the header before the segment is handed to the transport stack. Implementations MUST NOT modify TCP sequence or acknowledgment numbers when inserting or removing an inline message; the fields requiring adjustment are the IP total length, the TCP data offset, and the transport checksums, which MUST be recomputed. After removal, the byte stream each endpoint's transport stack observes is identical to that of a session that never carried ASRP messages. On a UDP flow, no such machinery is required. The ASRP Option is carried in the datagram's option area, beyond the length indicated by the UDP header: a receiving stack that does not process UDP options ignores it by definition, and the receiving ASRP module reads the option without modifying the datagram. The datagram delivered to the application is the one the sender's application wrote. A separate packet that copies the original packet header carries no data of the flow: the receiving ASRP module processes the message and discards the packet, on both TCP and UDP flows.

Message Processing Standalone-ASRP packets are identified by the UDP destination port, while Inline-ASRP packets are identified by the ASRP option. Once an ASRP packet is identified, the ASRP message within it is then parsed and processed.

Message Validation Before processing any ASRP message, a receiver MUST perform the following checks and silently discard the message if any of them fails: The Version field is a version the receiver implements. The Length field matches the number of ASRP message octets actually present in the packet. The STType field matches the MsgType: QS and RX carry a single Session-Tuple (ST4 or ST6); NS, NA, NC, RS, and PS carry a Session-Tuple pair (ST44, ST66, ST46, or ST64); HS carries no Session-Tuple (NULL). The Reserved field and any unassigned flag bits are ignored. A message that passes these checks is processed according to its MsgType as described in the following subsections.

Cookie Handling Every ASRP message carries the 4-octet Cookie field defined in . The cookie associates a response with the request that triggered it. The sender of a request message (QS, NS, or HS) SHOULD fill the Cookie field with a value that identifies the request; a per-request random value is a typical choice. A responder (the sender of RS/RX, NA/NC, or the NS replying to HS) MUST copy the Cookie field of the request into the Cookie field of its response, unchanged. The Cookie is opaque on the wire and echoed verbatim, so interoperability does not depend on how it is generated; the security properties of different generation methods are discussed in the Security Considerations. A node that receives a response SHOULD verify that the Cookie field matches the outstanding request it is applying the response to, and discard the response otherwise. A backup holder MUST store the Cookie of the NS message that created the backup, and MUST echo that Cookie in every PS message it sends for that session. A network node that receives a PS message SHOULD verify the echoed Cookie when it can - e.g., by matching its local record for the session, or by recomputing a Cookie whose derivation it is configured with - and silently discard the PS message if verification fails. A PS message whose Cookie the node cannot verify is processed according to the node's local policy; see the Security Considerations. An implementation that does not use cookie validation MAY set the Cookie field of its requests to zero and skip the verification above; see the Security Considerations for how this affects message validation. Stronger message authentication, e.g., based on a message authentication code, can be introduced as a future extension carried in the variable-length Session-Data field, without any change to the message format.

NS Message When a client or server receives an NS message, it MUST first check for session conflicts on the client side and then on the server side. A client-side conflict occurs when the client-side tuple matches an existing backed-up session on either side. If a conflict is found, it MUST clear the F-NA flag and send an NC message, with the F-CC flag set to indicate a client-side conflict. If no conflict is found, it MUST back up the session state information together with the Cookie of the NS message, and hand the embedded packet (if present) over to the system. If the F-NA flag is set, it MUST send an NA message. The F-NA flag SHOULD be set when the session is created from a non-explicit first packet; explicit first packets (e.g., TCP SYN or DNS query) do not require it. If an NS message is lost, in PSV mode a TCP SYN retransmission or subsequent packets regenerate NS messages until an NA message is received; in ACT mode, subsequent packets generate HS messages, prompting the network node to retransmit the NS message. NS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , and .

NA Message The destination IP of the NA packet MUST be set to the Node-Local-IP field carried in the received NS message, and the destination port MUST be set to ASRP-PORT. Upon receiving an NA message, the network node considers that the corresponding NS message has been processed by the responder, and MUST NOT send any further NS messages for this session.

NC Message The destination IP of the NC packet MUST be set to the Node-Local-IP field carried in the received NS message, and the destination port MUST be set to ASRP-PORT. Upon receiving an NC message, the network node considers that the session conflicts with an existing local session on the responder. If the F-CC flag is set, the conflict is on the client side; otherwise, it is on the server side. The network node SHOULD handle the conflict according to its local policy.

QS Message The source IP of the QS packet MUST be set to the network node's local IP (obtainable from configuration), and the destination IP MUST be set to the client's or server's IP (obtainable from the forwarded packet as described in and , or derived via algorithmic mapping to the client or server as described in and ). The source port MUST be randomly generated, and the destination port MUST be set to ASRP-PORT. The backup holder MUST index the backed-up session by both the client-side and the server-side 5-tuples. When a client or server receives a QS packet, it MUST look up the backed-up session whose client-side or server-side 5-tuple matches the Session-Tuple in the QS message. If a match is found, it MUST return an RS message; otherwise, it MUST return an RX message. If a QS message is lost, subsequent packets will trigger the generation of new QS packets, continuing the attempt to recover the session. QS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , and .

RS Message The RS packet MUST reuse the protocol header of the QS packet, with the source and destination IP addresses swapped. When a network node receives an RS packet, it MUST recover the session and forward packets according to the session. If an RS message is lost, subsequent QS messages will continue the attempt to recover the session, thereby triggering retransmission of the RS message. RS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , , , and .

RX Message Upon receiving an RX message, the network node MUST follow the processing procedures described in the respective scenarios under PSV/ACT mode.

HS Message The network node identifies the session by the 5-tuple of the packet carrying the HS message. If a local session is found, it MUST return an NS message for that session to the client, addressed to the source IP of that packet; otherwise, no NS message is returned. HS messages are only generated in ACT mode. The handling procedure is described in .

PS Message The source IP, destination IP, and source port of the PS packet MUST be copied from the packet sent by the server. When a network node receives a PS message, it SHOULD verify the echoed Cookie as described in "Cookie Handling" before recovering the session; it extracts the forwarded packet (if present) and forwards it according to the session. PS messages are only generated in PSV mode. The handling procedure is described in .

Security Considerations ASRP messages may be subject to forgery, replay, or flooding by attackers who are able to inject traffic into the network that carries them. ASRP deployments can counter this with two layers of message validation. Both draw only on information available to the validating node - including, where deployed, a cluster-wide key, which is deployment configuration rather than protocol state - so no protocol-level trust relationship between nodes is required. Cookie-based validation: Every ASRP message carries a 4-octet Cookie field. For a request/response pair (QS and RS/RX, NS and NA/NC, HS and the NS replying to it), a node that sends a request fills the Cookie and verifies that the response echoes it; this binds the response to an outstanding request of the receiving node, so that a forged response that guesses the Cookie, or a replayed response from an earlier request, can be recognized and discarded. A backup holder likewise echoes in its PS messages the Cookie of the NS message that created the backup, and a network node verifies the echoed Cookie before recovering a session; this binds a PS message to a session the cluster actually created. This document specifies the echo and verification framework and intentionally leaves Cookie generation to the implementation and deployment, since the Cookie is opaque and echoed verbatim: a per-request random value resists guessing and replay, and a value derived with a key shared across the cluster additionally lets any node verify a PS message, which matters under asymmetric routing, where the node receiving a PS message need not be the node that created the session. Session-legitimacy validation: Nodes SHOULD validate the session referenced by a message against application-level policies, such as verifying that the session points to a known and healthy server in a load-balancing deployment, or that the address translation complies with configured rules in a NAT deployment. Nodes that perform this validation MUST discard messages that fail it and MUST NOT create a new session for them. Implementations MAY apply rate limiting to ASRP messages as appropriate, e.g., to contain QS floods . ASRP responses are comparable in size to the requests that trigger them, so the protocol itself provides no amplification.

Privacy Considerations ASRP messages carry the 5-tuple of the session they describe. This is the same information that is visible in the session's own packets; observing ASRP traffic reveals the existence of a session between two endpoints and their addresses, but nothing beyond what the carried traffic already exposes. The Node-Local-IP field of an NS message discloses the address of the network node that created the session to the backup holder. This is information the backup holder requires in order to return NA/NC messages, and the network node address is in any case observable in the forwarded traffic of the session. ASRP introduces no new identifiers that persist across sessions and no tracking mechanism. Operators who wish to limit the visibility of session metadata can apply the usual operational measures to ASRP traffic, e.g., restricting routing of the ASRP-PORT destination to the serving domain.

IANA Considerations This document defines an application-layer protocol (ASRP). All values of the MsgType, STType, and Flags fields used by this version of the protocol are defined in this specification. New values MUST be defined via an update to this document or a successor specification; a future standards-track version of ASRP SHOULD request the creation of IANA registries for these fields. The implementation of this protocol requires the allocation of a TCP option Kind, a UDP option Kind, and a UDP destination port:

TCP Option Kind The ASRP option (see ) requires the allocation of a TCP option Kind number from the TCP option kind registry: Kind Length Meaning Reference TBD 2 ASRP This document

UDP Option Kind The ASRP option requires the allocation of a UDP option Kind number from the "UDP Option Kind Numbers" registry : Kind Length Meaning Reference TBD variable ASRP This document

UDP Destination Port QS/RS/RX/NA/NC messages are encapsulated within UDP datagrams for transmission. A fixed UDP destination port number is required so that the receiving end can identify and process such encapsulated packets. Service Name: asrp Port Number: 51200 (proposed value for current experimentation) Transport Protocol: udp Description: Used for receiving UDP-encapsulated ASRP protocol messages. For experimental implementations and interoperability testing prior to IANA assignment, UDP port 51200 MAY be used as a temporary default. This port falls within the dynamic/private port range (49152-65535) reserved for local or temporary use and documentation examples . IANA is requested to assign a permanent port number in the "User Ports" range (1024-49151) for the "asrp" service in the "Service Name and Transport Protocol Port Number Registry", with a reference to this document.

Experimental Use Prior to IANA assignment of permanent option Kind values, implementations deployed for experimentation or interoperability testing SHOULD use the option Kind values reserved for experimental use, together with an Experiment Identifier (ExID), as described in for TCP and for UDP.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Transport protocols are extended through the use of transport header options. This document updates RFC 768 (UDP) by indicating the location, syntax, and semantics for UDP transport layer options within the surplus area after the end of the UDP user data but before the end of the IP datagram. This RFC is the revised basic definition of The Domain Name System. It obsoletes RFC-882. This memo describes the domain style names and their used for host address look up and electronic mail forwarding. It discusses the clients and servers in the domain name system and the protocol used between them. This RFC is the revised specification of the protocol and format used in the implementation of the Domain Name System. It obsoletes RFC-883. This memo documents the details of the domain name client - server communication. The effect of multipath routing on a forwarder is that the forwarder potentially has several next-hops for any given destination and must use some method to choose which next-hop should be used for a given data packet. This memo summarizes current practices, problems, and solutions. This memo provides information for the Internet community. Equal-cost multi-path (ECMP) is a routing technique for routing packets along multiple paths of equal cost. The forwarding engine identifies paths by next-hop. When forwarding a packet the router must decide which next-hop (path) to use. This document gives an analysis of one method for making that decision. The analysis includes the performance of the algorithm and the disruption caused by changes to the set of next-hops. This memo provides information for the Internet community. This document defines basic terminology for describing different types of Network Address Translation (NAT) behavior when handling Unicast UDP and also defines a set of requirements that would allow many applications, such as multimedia communications or online gaming, to work consistently. Developing NATs that meet this set of requirements will greatly increase the likelihood that these applications will function properly. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document describes TCP SYN flooding attacks, which have been well-known to the community for several years. Various countermeasures against these attacks, and the trade-offs of each, are described. This document archives explanations of the attack and common defense techniques for the benefit of TCP implementers and administrators of TCP servers or networks, but does not make any standards-level recommendations. This memo provides information for the Internet community. This memo defines the Virtual Router Redundancy Protocol (VRRP) for IPv4 and IPv6. It is version three (3) of the protocol, and it is based on VRRP (version 2) for IPv4 that is defined in RFC 3768 and in "Virtual Router Redundancy Protocol for IPv6". VRRP specifies an election protocol that dynamically assigns responsibility for a virtual router to one of the VRRP routers on a LAN. The VRRP router controlling the IPv4 or IPv6 address(es) associated with a virtual router is called the Master, and it forwards packets sent to these IPv4 or IPv6 addresses. VRRP Master routers are configured with virtual IPv4 or IPv6 addresses, and VRRP Backup routers infer the address family of the virtual addresses being carried based on the transport protocol. Within a VRRP router, the virtual routers in each of the IPv4 and IPv6 address families are a domain unto themselves and do not overlap. The election process provides dynamic failover in the forwarding responsibility should the Master become unavailable. For IPv4, the advantage gained from using VRRP is a higher-availability default path without requiring configuration of dynamic routing or router discovery protocols on every end-host. For IPv6, the advantage gained from using VRRP for IPv6 is a quicker switchover to Backup routers than can be obtained with standard IPv6 Neighbor Discovery mechanisms. [STANDARDS-TRACK] This document defines the procedures that the Internet Assigned Numbers Authority (IANA) uses when handling assignment and other requests related to the Service Name and Transport Protocol Port Number registry. It also discusses the rationale and principles behind these procedures and how they facilitate the long-term sustainability of the registry. This document updates IANA's procedures by obsoleting the previous UDP and TCP port assignment procedures defined in Sections 8 and 9.1 of the IANA Allocation Guidelines, and it updates the IANA service name and port assignment procedures for UDP-Lite, the Datagram Congestion Control Protocol (DCCP), and the Stream Control Transmission Protocol (SCTP). It also updates the DNS SRV specification to clarify what a service name is and how it is registered. This memo documents an Internet Best Current Practice. This document describes how the experimental TCP option codepoints can concurrently support multiple TCP extensions, even within the same connection, using a new IANA TCP experiment identifier. This approach is robust to experiments that are not registered and to those that do not use this sharing mechanism. It is recommended for all new TCP options that use these codepoints. This document describes an experimental TCP mechanism called TCP Fast Open (TFO). TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged. However, TFO deviates from the standard TCP semantics, since the data in the SYN could be replayed to an application in some rare circumstances. Applications should not use TFO unless they can tolerate this issue, as detailed in the Applicability section. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. CMCC CMCC This document defines the Session Recovery (SR) option for TCP. The option lets the endpoints of a connection exchange identifiers: during the handshake each endpoint carries its own identifier, and designated segments afterwards carry the identifier of the peer. This places the knowledge of which endpoint a connection belongs to inside the TCP header, where network functions on the path - load balancers, NAT gateways, firewalls - can read it without per-flow state and without payload inspection. The primary use is session recovery in SNAT and load-balancing clusters; further uses include reduced-state forwarding, connection-tracking recovery, and per- backend telemetry. The option is carried in the SYN, the SYN-ACK, and retransmitted segments, or in every segment when so configured; the default adds no overhead to normal data segments. Endpoints that do not support it behave as if the option did not exist.

Implementation Status Note to the RFC Editor: Please remove this section before publication. ASRP has been implemented twice, through two independent technology stacks, and the implementations are interoperable: A Linux kernel module extending IPVS, implementing the network-node role in PSV mode: session creation with inline NS insertion, QS/RS exchange for session recovery, NA/NC handling, cookie verification of responses, and the ASRP-PORT UDP listener. An eBPF implementation based on traffic-control programs, implementing the backup-holder role on the endpoints: NS backup from session first packets, session reconstruction from PS messages, and response generation with cookie echoing. The two stacks were interoperability tested on a three-node testbed (client, load-balancing node, backend server), covering direct session creation, recovery after cluster session-table loss simulating node failure, and the fallback when no backed-up session exists. Recovery is driven by the retransmission behavior of the recovered session, as described in the scenarios. The small footprint of both implementations - a kernel module and a set of eBPF programs - indicates that endpoint support for ASRP is a modest engineering effort.

Acknowledgments The authors would like to thank all individuals who have provided valuable feedback and contributions during the development of this document.