]>

dragos.oancea.3.141592@gmail.com

Web and Internet Transport Audio/Video Transport Core Maintenance Opus FEC ULPFEC LBRR RTP packet loss The Opus codec can carry low-bitrate redundancy for the preceding frame inside each packet ("in-band FEC"). The RTP payload format for generic Forward Error Correction (RFC 5109, commonly called ULPFEC) protects groups of RTP packets with XOR parity packets. The two mechanisms recover different loss patterns, have different delay and overhead characteristics, and interact in ways that are not described in either specification. This document gives informational guidance on when the combination is worthwhile, how a sender should configure each mechanism, how a receiver should order recovery, how to signal the combination in SDP, how middleboxes affect it, and how to keep the combined redundancy consistent with congestion control. It complements, and does not change, the WebRTC FEC requirements in RFC 8854. It defines no new protocol elements. About This Document Discussion of this document is expected to take place on the AVTCORE Working Group mailing list (avt@ietf.org). This is an individual submission and has not been adopted by any working group.

Introduction Packet loss on the path between an audio sender and receiver can be repaired in several ways: retransmission , redundant encoding , codec-internal redundancy, and separate parity-based FEC streams such as and . For Opus , the codec-internal mechanism is the Low-Bit-Rate Redundancy (LBRR) of the SILK layer, which RTP usage refers to as "in-band FEC". In-band FEC and ULPFEC are complementary. In-band FEC adds at most one packet of delay and can be decoded from a single received packet, but it can only repair the frame immediately preceding a received packet, it is not available in every Opus mode, and it yields a lower-fidelity reconstruction. ULPFEC reconstructs lost RTP packets bit-exactly, independent of the codec and mode, and can be arranged to cover bursts, but it adds delay proportional to the size of the protected group, and its overhead does not shrink with the codec's own bitrate decisions. Neither nor discusses using the two together, and (which targets WebRTC) recommends Opus in-band FEC for audio and does not recommend a separate FEC stream for audio because of its overhead. That recommendation remains appropriate for typical interactive WebRTC calls. There are, however, deployments where the assumptions behind it do not hold: music or other content coded in Opus modes without LBRR, links with frequent burst loss, one-way or latency-tolerant distribution, multicast without a feedback path, and systems that must recover RTP header extensions or contributing-source lists along with the payload. This document describes how to combine the mechanisms well in those deployments.

Scope This document is informational. It does not update , , or , and it defines no new RTP payload formats, SDP attributes, or media type parameters. Where this document uses requirement language, it describes good practice for implementations that choose to combine the two mechanisms; it does not impose requirements on implementations that do not. Flexible FEC provides a superset of ULPFEC's capabilities and most of the guidance here applies to it equally. This document concentrates on ULPFEC because it is widely implemented and because is the format named in many existing deployments. The Opus Deep Redundancy (DRED) extension under development in the MLCODEC Working Group is discussed in .

Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

In-band FEC:

The Opus LBRR mechanism , in which an Opus packet carries a lower-bitrate re-encoding of the audio of the previous packet, as signalled by the "useinbandfec" media type parameter .

ULPFEC:

The RTP payload format for generic forward error correction , including its uneven level protection (ULP) extension.

Media packet:

An RTP packet carrying an Opus payload.

FEC packet:

An RTP packet, or a RED block , carrying ULPFEC data.

Protection group:

The set of media packets covered by one ULPFEC level of one FEC packet, as identified by its mask. The group size is denoted k.

Pre-repair loss:

The fraction of media packets not received from the network.

Residual (post-repair) loss:

The fraction of media packets that could not be reconstructed by either mechanism and therefore required packet loss concealment.

Background

Opus In-Band FEC Opus operates in SILK-only, Hybrid, and CELT-only modes . LBRR frames are a SILK-layer feature, so in-band FEC is only available in SILK-only and Hybrid modes. Audio coded in CELT-only mode, which encoders commonly select for music and for high bitrates, carries no in-band redundancy. When enabled, an Opus packet can carry an LBRR re-encoding of the SILK frames of the previous packet. A decoder that has lost packet N but received packet N+1 can decode the LBRR data in packet N+1 instead of concealing the loss. The decision to include LBRR data is made entirely by the encoder ; in the reference implementation it depends on the application enabling in-band FEC and on the expected packet loss rate configured by the application. A receiver indicates that it can make use of in-band FEC with "useinbandfec=1" ; the default is 0, and requires WebRTC receivers that can use incoming FEC to signal it. Two properties matter for the combination:

• In-band FEC can only repair a loss whose following packet arrives. Two consecutive losses leave the first one unrepairable by LBRR. notes this limitation and reports an overhead of roughly 20-30% when it is active.
• Because LBRR data is inside the Opus payload, it travels with the payload through any RTP element that forwards the payload unchanged, including encryption with SRTP or end-to-end encryption with SFrame .

ULPFEC An ULPFEC packet carries a 10-octet FEC header and one or more level headers, each with a protection length and a 16- or 48-bit mask selecting the protected media packets relative to a base sequence number . The FEC data is the XOR of the protected packets' RTP header fields and of the octets following the fixed 12-octet RTP header, which include any CSRC list, header extension, and payload. If exactly one packet of a protection group is missing, it can be reconstructed exactly, including its marker bit, timestamp, CSRCs, and header extensions. RED-based redundancy cannot recover these header elements . allows FEC to be sent either as a separate stream with the same SSRC as the protected stream, in which case it recommends a separate RTP session, or as a secondary encoding inside RED. The registered media type audio/ulpfec has a required "rate" parameter; matching the protected media's clock rate, 48000 for Opus , is recommended.

Comparison Properties of the Two Mechanisms for Opus Audio

Property	Opus in-band FEC	ULPFEC
Losses repaired	Single loss followed by a received packet	One loss per protection group per level; bursts if interleaved
Added receiver delay	One packet duration	Up to the time span of the protection group
Fidelity of repair	Lower-bitrate approximation	Bit-exact packet, including header extensions
Opus modes	SILK-only and Hybrid	All
Overhead	Chosen per packet by the encoder	About 1/k of the largest protected packet, plus headers
Survives payload-preserving forwarding	Yes	Only if sequence numbers and headers are preserved ()
Signalling	"useinbandfec" parameter	Separate payload type, and session or RED configuration

When the Combination Is Worthwhile For interactive, unicast calls with isolated losses, Opus in-band FEC alone (optionally supplemented with RED as recommended by ) is usually the better trade-off, and senders SHOULD NOT add ULPFEC for audio in that case. The combination is worth considering when one or more of the following holds:

1. Content without LBRR. The encoder is expected to use CELT-only mode for significant periods (for example, music or high-bitrate full-band audio), so in-band FEC provides no protection. ULPFEC then protects those periods, and in-band FEC continues to protect speech periods at lower cost.
2. Burst loss. Measured losses often occur in runs of two or more packets (for example, on wireless or satellite links). LBRR alone cannot repair the first loss of a run; ULPFEC with an interleaved mask can, and a packet recovered by ULPFEC can in turn supply LBRR data for its predecessor ().
3. Latency tolerance. The application's delay budget can absorb the span of a protection group, as in one-way distribution, broadcast contribution, or announcements. Larger groups lower the overhead.
4. No feedback path. Multicast or unidirectional delivery rules out retransmission .
5. Header recovery. The receiver depends on information carried in RTP header extensions or CSRC lists, such as audio level indications , which in-band FEC and RED cannot restore.

If none of these holds, the added overhead and delay of ULPFEC are unlikely to be justified for Opus audio.

Sender Guidance

Enabling and Adapting Each Mechanism As recommends, FEC should only be activated when network conditions warrant it. A sender combining the mechanisms SHOULD adapt them independently:

• In-band FEC is cheap to adapt because the encoder decides per packet. The sender typically provides the encoder with an expected loss rate. When ULPFEC is active, the loss that the Opus decoder experiences is the residual loss after ULPFEC recovery, not the network loss. Configuring the encoder with the pre-repair loss rate will over-provision LBRR. Where a post-repair loss estimate is available (), the sender SHOULD base the encoder's expected loss rate on it.
• ULPFEC should be enabled only when the observed loss pattern matches one of the conditions in , and disabled when loss subsides. Its group size and mask are the main levers ().

A sender SHOULD NOT send ULPFEC to a receiver that has not accepted the ULPFEC payload type in the offer/answer exchange , and SHOULD NOT include in-band FEC data for a receiver that signalled "useinbandfec=0" unless it has other reasons to do so. Any compliant Opus decoder ignores in-band FEC data it does not need , so this is an efficiency concern, not an interoperability one.

ULPFEC Parameters for Opus The following considerations apply when choosing ULPFEC parameters for an Opus stream:

• Use a single, full-length level. ULP allows lower levels to protect only the first octets of each packet. An Opus packet is entropy-coded, and a partially recovered Opus packet is generally not decodable, so protecting only a prefix of the payload gives little benefit. The protection length SHOULD cover the longest protected packet, and a single level is usually sufficient (see the "onelevelonly" parameter in ).
• Size the group to the delay budget. A missing packet can only be reconstructed after the last packet of its group and the FEC packet have arrived. With 20 ms packets and consecutive masks, a group of k=4 spans about 80 ms, and an interleaved arrangement that covers every second packet across eight packets spans about 160 ms. For two-way conversation, the total one-way delay should stay within the guidance of ; a small k (2 to 4) with consecutive masks is typically the limit. Latency-tolerant applications can use larger k and interleaving.
• Interleave for bursts. If losses commonly come in pairs, masks that protect alternate packets (two FEC packets per eight media packets, each covering every second packet) let each FEC packet repair one loss of a pair at the same overhead as consecutive groups of four.
• Account for variable packet size. Opus is usually used in variable-bitrate mode, and LBRR data increases the size of the packets that carry it. An FEC packet must be as long as the longest packet it protects, so the actual overhead exceeds 1/k of the average media bitrate. Grouping packets of similar size, or using smaller groups when sizes vary widely, reduces this effect.
• Avoid protecting redundancy twice without reason. ULPFEC protects the whole payload, including any LBRR data it contains. When ULPFEC is active and the loss pattern is mostly isolated losses, in-band FEC adds little and can be reduced; when the pattern is bursty, keeping both gives the largest benefit ().

Bitrate Budget The sender MUST keep the total of primary Opus bitrate, in-band FEC, ULPFEC, and packet headers within the rate allowed by its congestion controller (). When FEC is enabled or increased, the primary Opus bitrate should be reduced to make room for it, rather than adding FEC on top of a rate that already fills the allowed budget. The worked example in shows that ULPFEC with k=4 adds about 27% to the on-the-wire rate of a 32 kbit/s Opus stream with in-band FEC, under the stated assumptions.

Receiver Guidance

Recovery Order When media packet N is missing at its playout deadline, a receiver supporting both mechanisms SHOULD attempt recovery in the following order:

1. ULPFEC: if an FEC packet covering N and all other packets of its protection group have arrived (or been recovered), reconstruct N exactly and decode it normally.
2. In-band FEC: otherwise, if packet N+1 is available, whether received or reconstructed by ULPFEC, and contains LBRR data, decode the in-band FEC data from N+1 to produce audio for N.
3. Concealment: otherwise, apply the Opus decoder's packet loss concealment.

ULPFEC recovery comes first because it produces the original packet and restores its header information. Receivers SHOULD run ULPFEC recovery iteratively: a packet recovered from one FEC packet can complete another protection group or provide the N+1 packet needed for in-band FEC.

Jitter Buffer Interaction ULPFEC only helps if the jitter buffer holds missing packets' slots long enough for the FEC packet to arrive. An adaptive jitter buffer that targets a delay shorter than the protection group's span will rarely benefit from ULPFEC and still pays its overhead. Receivers SHOULD take the signalled or observed protection span into account when choosing a target delay, or report to the application that ULPFEC is ineffective at the current delay so that the sender can be asked to reduce k. In-band FEC needs only one packet duration of additional delay. A packet that is reconstructed by ULPFEC and later arrives late from the network is a duplicate. Receivers MUST discard such duplicates and MUST NOT decode the same audio twice from LBRR data and from a recovered or late packet.

Loss Accounting Receivers should keep separate counts of packets lost before repair, packets recovered by ULPFEC, losses covered by in-band FEC, and losses concealed. Pre-repair counts are the input to congestion control (); post-repair counts are the input to FEC adaptation (). RTCP receiver reports are conventionally computed from packets received from the network, before repair.

Signalling The combination needs no new signalling. Opus in-band FEC is indicated with "useinbandfec" ; ULPFEC is negotiated as its own payload type . Two configurations are possible.

ULPFEC Inside RED When media is multiplexed with BUNDLE , a separate RTP session for FEC is not available. ULPFEC can instead be carried as a secondary encoding inside RED , as allowed by . The RED encapsulation then carries the Opus primary block in every packet and an ULPFEC block in the packets the sender chooses. The following SDP fragment offers this configuration: In this configuration, implementers should be aware that RED adds a per-block header, and that peers which support RED for audio do not necessarily support ULPFEC as a RED secondary encoding for audio. An answerer that does not support it removes payload type 100 from its answer, and the sender then uses in-band FEC (and RED redundancy, if negotiated) alone.

ULPFEC as a Separate Session Outside WebRTC, ULPFEC can be sent in its own RTP session, as recommended by , and associated with the media session using the FEC-FR grouping semantics of . The FEC stream uses the same SSRC as the media stream. advises against offering FEC-only "m=" lines in WebRTC; the example below is intended for other environments such as SIP-based or broadcast systems. The "maxaveragebitrate" parameter limits the average bitrate of the Opus stream . It does not cover the separate FEC stream, so the sender must still account for ULPFEC in its overall budget ().

Middleboxes RTP topologies affect the two mechanisms differently:

• Selective forwarding units (SFUs). An SFU that forwards Opus payloads unchanged preserves in-band FEC. ULPFEC, however, protects the RTP sequence number, timestamp, marker bit, and header extensions, and its base sequence number refers to the original numbering. An SFU that rewrites sequence numbers or SSRCs, drops packets for a given receiver, or adds or removes header extensions MUST NOT forward the original ULPFEC packets unchanged, because receivers would reconstruct incorrect packets. Such an SFU should terminate ULPFEC from the sender (using it to repair losses on the uplink) and, if useful, generate new ULPFEC per downstream receiver based on that receiver's loss conditions.
• Mixers. A mixer that decodes, mixes, and re-encodes terminates both mechanisms. It should apply recovery on each input as in before mixing, and choose redundancy for each output independently.
• End-to-end encryption. With SFrame , the Opus payload, including LBRR data, is opaque to the SFU. Because ULPFEC is an XOR over octets, an SFU can still generate and apply ULPFEC over the encrypted payloads on each hop without access to the media keys.

Congestion Control Considerations FEC must not hide congestion. recommends that, from a congestion control standpoint, a packet recovered by FEC be treated as lost. requires that implementations not substantially increase total bandwidth as losses increase, and notes that redundancy sent during congestion reduces the bandwidth available for the primary encoding and can make losses worse. For the combination, this means:

• Congestion control MUST be driven by pre-repair loss and delay signals, not by residual loss after ULPFEC and in-band FEC.
• The combined rate of Opus (including LBRR), ULPFEC, and headers MUST stay within the allowed sending rate. Under congestion, the sender should reduce the total rate first and then divide what remains between primary coding and redundancy; increasing redundancy in response to congestion-induced loss is counter-productive.
• Applications using RTP over UDP remain subject to the congestion control expectations of and , and to RTP circuit breakers , which apply to the combined media and FEC traffic.

Using 's classification, both mechanisms as described here operate above the transport, within the rate given to the application by its congestion controller.

Feedback and Measurement Adapting the combination requires both pre-repair and post-repair loss information. Standard RTCP receiver reports provide the former. The latter can be conveyed with the RTCP Extended Report blocks for post-repair loss run-length encoding or for post-repair loss counts . A sender that receives these can tell whether ULPFEC is effective at the receiver's jitter buffer depth () and can set the Opus encoder's expected loss rate from the residual loss (). Without post-repair feedback, the sender has to rely on models such as the one in , which can be substantially wrong for real loss patterns.

Relationship to Opus Deep Redundancy (DRED) The MLCODEC Working Group is specifying a Deep Redundancy (DRED) extension for Opus . DRED carries very low bitrate redundancy, covering up to a second or more of past audio, inside the Opus padding. When both peers support it, DRED addresses much of the burst-loss case for which this document suggests ULPFEC, at lower overhead. ULPFEC would remain relevant where bit-exact recovery, recovery of RTP header information, codec independence, or support for peers without DRED is needed. Guidance on combining DRED with ULPFEC, and with in-band FEC, is left for a future revision once the DRED specification is stable.

IANA Considerations This document has no IANA actions.

Security Considerations Neither in-band FEC nor ULPFEC provides any security. The security considerations of , , and apply. In particular:

• When SRTP is used, FEC packets SHOULD receive the same confidentiality and integrity protection as the media packets. An attacker able to inject or modify unauthenticated FEC packets can cause a receiver to reconstruct incorrect packets. Receivers should apply sanity checks, such as length and timestamp plausibility, to recovered packets, as advises.
• FEC recovery consumes receiver processing. Large or numerous FEC packets, including ones that can never complete a group, can be used to increase receiver load; receivers should bound the processing and memory devoted to pending protection groups.
• Variable bitrate audio can leak information about the content of encrypted speech through packet sizes . Whether the encoder includes LBRR data in a packet is itself signal-dependent and so affects packet size. ULPFEC packet sizes follow the largest packet of each group. Deployments concerned with this leakage should follow the guidance of , for example by using constant bitrate coding, and should not assume that FEC traffic hides the pattern.

References Normative References Informative References ITU-T Work in Progress.

Worked Overhead Example This example illustrates the bitrate cost of each mechanism for a single stream. Assumptions: Opus at 32 kbit/s with 20 ms packets (80 octets of payload per packet, 50 packets per second); in-band FEC adding 25% to the payload when active (20 octets), the midpoint of the range reported in ; IPv4 and UDP headers (28 octets), RTP fixed header (12 octets), and a 10-octet SRTP authentication tag; ULPFEC sent as a separate stream with one FEC packet per four media packets (k=4), one 4-octet level header, and a protection length equal to the 100-octet media payload. Illustrative On-the-Wire Bitrates

Configuration	Packet size (octets)	Packets/s	Rate (kbit/s)
Opus only	130	50	52.0
Opus with in-band FEC	150	50	60.0
ULPFEC stream, k=4	164	12.5	16.4
Opus with in-band FEC and ULPFEC, k=4	-	62.5	76.4

With k=2 the ULPFEC stream would cost 32.8 kbit/s, bringing the total to 92.8 kbit/s. Real overhead depends on Opus packet-size variation (), on IPv6 (20 additional octets per packet), and on RED encapsulation if used.

Illustrative Residual Loss The following table gives residual loss from a simple simulation, to show the qualitative behaviour of each mechanism. It is not a performance claim. Assumptions: 20 ms packets; in-band FEC present in every media packet and able to repair packet N whenever packet N+1 is available; ULPFEC with one FEC packet per four media packets, either consecutive (k=4) or interleaved (two FEC packets per eight media packets, each covering every second packet); FEC packets subject to the same loss process as media packets; losses generated by a two-state Gilbert model with the stated average loss and mean burst length, where a mean burst length of 1 means independent losses; 400,000 media packets per run. Receivers apply ULPFEC first and then in-band FEC, as in . Values are residual loss in percent. Simulated Residual Loss (%)

Mean burst	Network loss	None	In-band FEC	ULPFEC k=4	Both, k=4	ULPFEC interleaved	Both, interleaved
1	2%	1.99	0.04	0.15	0.03	0.15	0.00
1	5%	4.97	0.24	0.92	0.18	0.93	0.01
1	10%	9.98	0.99	3.39	0.77	3.43	0.12
1	20%	20.04	4.02	11.81	3.35	11.90	1.38
2	2%	1.96	0.97	1.35	0.78	0.81	0.33
2	5%	5.02	2.49	3.53	2.03	2.27	0.89
2	10%	9.92	4.93	7.22	4.07	5.16	1.99
2	20%	19.91	9.92	15.33	8.37	12.96	5.08

Under these assumptions, in-band FEC alone is very effective against independent losses, and adding consecutive ULPFEC improves on it only modestly; interleaving helps even here, because two adjacent losses then fall into different protection groups. With bursts, in-band FEC alone halves the loss at best, while interleaved ULPFEC combined with in-band FEC reduces it several-fold, because ULPFEC breaks up bursts and the packets it recovers then supply LBRR data for their predecessors. The interleaved arrangement doubles the recovery delay (). Real in-band FEC is not present in every packet, is absent in CELT-only mode, and repairs at lower fidelity, so measurements with real encoders and loss traces are needed before drawing quantitative conclusions.

Acknowledgments This document builds on the analysis of FEC for WebRTC in and on the FEC and congestion control guidance of .