transport layer doug young suh [email protected] last update : august 1, 2009 transport layer
TRANSCRIPT
ContentsTransport layer QoSUDP : User Datagram (connection-less)RTP : Realtime Transport (connection-oriented)RFC 3984
RTP Payload format for H.264 Video
Transport Layer
Transport layer and Realtime Multimedia
Transport Layer
Transport layer : end-to-endFlow control : “Send as much as it receives.”Error control : packet lossNothing to do with the routers
Networked Video
Transport layer
Call setup : path, resourcesQoS control : RTP/RTCP QoS enhancement (not standardized, yet)
FEC with erasures Interleaving Dejittering buffer Retransmission if allowed
Upper Layers
Transport Layer
Network Layer
Data Link Layer
Physical Layer
Error resilience, scalable coding, rate control
FEC, retransmission, flow controlUpper Layers
Transport Layer
Network Layer
Data Link Layer
Physical Layer
Network Layer
Data Link Layer
Physical Layer
Network Layer
Data Link Layer
Physical Layer
Resource allocation
……
Mobileterminal
Base station
VOD Server
Routers
Networked Video
the avt WG?
8 Areas in IETF (http://ietf.org)Applications Area - General Area Internet Area - Routing Area Security Area - Sub-IP Area
Operations and Management Area Transport Area
Among 27 working groups in Transport Area ([email protected])
RTCP (RTP control)
In every interval(e.g. 1 sec), exchange RTCP-SR (Sender Report) and RTCP-RR (Receiver Report).
Report Packet loss ratio (PLR) by using SNThe receiver counts number of lost RTP packets.
Measure RTT (Round Trip Time, but, not one way delay) by using Time Stamp
RTT = T4 – T1 – (T3 – T2)
Networked Video
senderreceiv
er
RTCP-SR
RTCP-RR
T1 T2
T4 T3
Networked Video
Time varying end-to-end QoS (Suwon Pusan)
delay & loss rate
0
50
100
150
200
250
300
350
400
450
time
dela
y(m
s)
0
0.05
0.1
0.15
0.2
0.25
0.3
loss r
ate
(%)
loss rate(%) mean delay(ms)
Networked Video
FEC (forward erasure correction)
data parity
lossy channel
recovery
• RFC2733 : XOR-based (one parity packet) • IETF : draft-ietf-avt-uxp-02.txt
Networked Video
FEC with erasure
Erasures : RTP Sequence numbers of lost packets Recover up to N-k lost packets by using RS(N,k) code
How to overcome?
1.E-11
1.E-09
1.E-07
1.E-05
1.E-03
1.E-01
1 3/4 1/2 1/4
coding rate R = K/N (PLR=10%) .
Res
idua
l PLR
N=4 N=8 N=16
Networked Video
Interleaving (RFC 3267)
to combat against bursty loss : loss rate vs. delay Example) 1) No Interleaving : 2 out of each set of 6 packets (1 2 3 4 P1 P2) (5 6 7 8 P3 P4)(9 10 11 12 P5 P6) (13 14 15
16 P7 P8)...... 2) Interleaving : up to 12 consecutive packets
1 5 9 13 17 21 2 6 10 14 18 22 3 7 11 15 19 23 4 8 12 16 20 24 P1 P3 P5 P7
P9 P11 P2 P4 P6 P8 P10 P12
How to overcome?
2
1
P2
P1
4
3
6
5
P4
P3
8
7
10
9
P6
P5
12
11
14
13
P8
P7
16
15
18
17
P10
P9
20
19
22
21
P12
P11
24
23
RFC 3409 : FEC with UED and UEP
XOR based ULP
0 1 0 1 1 1 0 0 1 0 1
0 0 1 1 0 1 1
1 0 1 0 1 0 1 1 0 0 1 0 1
1 1 0 0
Increase in data rate
P 1
P 2
P 3
RS code with UEP
Parity bytes
10 packets
Pack
et
size
Introduction to UDPThe UDP header (connection-less).
Transport Layer
Real-time Transport Protocol
(a) The position of RTP (connection-oriented) (b) Packet nesting.
Transport Layer
RTP (RFC 3550, 2003)RTP Use Scenarios
Simple Multicast Audio Conference Audio and Video ConferenceMixers and TranslatorsLayered Encodings
RTP Header, RTCP
Transport Layer
SSRC
CSRC
Semantics of fields in RTP HeaderMarker bit (M): 1 bit
Set for the very last packet of the access unit
Payload type (PT): 7 bitsMPEG-1 Video, MPEG-2 Video, AAC etc.
Sequence number (SN): 16 bitsdecoding order for the NALU.DON (Decoding Order Number)
Timestamp: 32 bitsthe sampling timestampA 90 kHz clock rate MUST be used.
Transport Layer
RFC3550: RTCP(RT Control Protocol, 2003)
RTCP SR(Sender Report), RR(Receiver Report)
The server monitors round trip time, jitter, and packet loss ratio.Exchanged periodically (<5% bitrate of total)
23年 4月 21日 MediaLab , Kyunghee University
16
Networked Video
RFC3611 RTCP XR(Extended Report, 2003)
Block types 1. Loss RLE Report Block 10. post repair RLE Block
(2009)2. Duplicate RLE Report Block 3. Packet Receipt Times Report Block 4. Receiver Reference Time Report Block 5. DLRR Report Block 6. Statistics Summary Report Block 7. VoIP Metrics Report Block Against burst loss (Internet,
wireless)255. reserved for future extensions
SDP attributesAttribute name : rtcp-xr Long form: RTP Control Protocol Extended Report ParametersType of attribute: session and media level
loss Run Length
Encoding
RFC3550 RTP Translators and Mixers
Intermediate systems between “network clouds”
Multicast audio and video via firewalls and low bandwidth connectionsOr IPv4 IPv6
The notation "M1: 48(1,17)" designates a packet originating a mixer M1, identified by M1's (random) SSRC value of 48 and two CSRC identifiers, 1 and 17, copied from the SSRC identifiers of packets from E1 and E2.
23年 4月 21日 MediaLab , Kyunghee University
18
Networked Video
RFC 3984 : RTP Payload Format for H.264 Video
RTP header + NAL header + slice header + coded sliceDecoupling sampling/transmission/decoding/presentation timeRTP header
V(2)+P(1)+X(1)+CC(4)+PT(7)+SN(16)+TS(32)+SSRC(32)+SCRC
MANE (Media Aware Network Element) with NAL headerF(1) + NRI (2) + NAL Type (5)
Packetization 1. A single NALU, 2. Aggregation, 3. Fragmentation
Order of packetsFEC with RFC 2733 (1999)
An RTP Payload Format for Generic Forward Error Correction
RFC 3984 (Feb. 2005)
RTP Payload format for H.264/AVCITU-T Rec. H.264 ISO/IEC IS 14496-10 (MPEG-4 Part 10 AVC)
Advanced Video Coding
Packetization of one or more Network Abstraction Layer Units (NALUs)Applications
simple low bit-rate conversational usageInternet video streaming with interleaved transmissionhigh bit-rate video-on-demand.
Transport Layer
RTP and H.264/AVC
VCL (Video Coding Layer)NAL (Network Abstraction Layer)RTP (Realtime Transport Protocol)
Transport Layer
Slice dataNAL
headerRTP
header
NAL header
F (1 bit) : forbidden_zero_bit. 1 as a syntax violationset by MANE
NRI (2 bits) nal_ref_idc00 not used to reconstruct reference
Type (5 bits) : nal_unit_typeSPS (7), PPS (8), slice data
Transport Layer
Type (5)NRI (2)F (1)
NAL Types
Transport Layer
NALU
NALU
NALU
RTP
RTP
RTP
NALU
NALU
NALU
RTP
NALU
NALU
NALURTP NALU
RTP
RTP
RTP
Single NAL Unit packet
STAP (Single Time Aggregation Packet)
MTAP (Multi Time Aggregation Packet)
FU (Fragmentation Unit)
picture
NALU
pictures
NAL Types
0 undefined 1-23 NAL unit Single NAL unit packet per H.264
Single NALU per one RTP packetLow delay conversational, non-interleaved
24 STAP-A Single-time aggregation packet 25 STAP-B Single-time aggregation packet 26 MTAP16 Multi-time aggregation packet 27 MTAP24 Multi-time aggregation packet 28 FU-A Fragmentation unit 29 FU-B Fragmentation unit 30-31 undefined
Transport Layer
Type (5)NRI (2)F (1)
An example of MTAP16 packet
16 bits for time stamp offset
Transport Layer
RTP Header 4 bytesMTAP NAL HDR Decoding order number base NALU 1 Size
NALU 1 Size NALU 1 DOND NALU 1 TS Offset
NAL 1 DATA
NALU 1 HDR
NALU 2 Size NALU 2 DOND
NALU 2 TS Offset NALU 2 HDR
NAL 2 DATA Optional RTP padding
32 bits
NAL Types and NRI
NAL Type Content of NALU NRI ----------------------------------1 non-IDR coded slice 102 Coded slice data partition A 103 Coded slice data partition B 014 Coded slice data partition C 015 Primary coded picture 117, 8 SPS, PPS 11
Sets of SPS and PPS are selected by slice header.
6, 9-12 00
Transport Layer
Packetization modes
yes = allowed, no = disallowed, ig = ignore
Type Packet Single NAL Non-Interleaved Interleaved Unit Mode Mode Mode --------------------------------------------- 0 undefined ig ig ig 1-23 NAL unit yes yes no 24 STAP-A no yes no 25 STAP-B no no yes 26 MTAP16 no no yes 27 MTAP24 no no yes 28 FU-A no yes yes 29 FU-B no no yes 30-31 undefined ig ig ig
Transport Layer
Packetization modes (1)
Single NAL Unit PacketSingle-time aggregation packet (STAP)
aggregates NAL units with identical NALU-timewithout DON (STAP-A),including DON (STAP-B).
Multi-time aggregation packet (MTAP):aggregates NAL units with potentially differing NALU-time16, 24 differing in the length of the NAL unit timestamp offset
Transport Layer
Packetization modes (2)
Fragmentation Units (FUs)a NAL unit into several RTP packetsBig NAL units (> 64 kbytes) over an IPv4 network
Because of a limit of the number of slices per picture
For FEC (Forward Error Correction)Especially for SPS, PPS, or Partition A as in RFC 2733
without DON (FU-A), with DON (FU-B).
Interleaving (sprop_deint_buf_req MIME parameter)
Interleaved (2) : STAP-Bs, MTAPs, FU-As, FU-BsNon-interleaved (1) : single, STAP-As, FU-As
Transport Layer
MANE (Media Aware Network Element)
Installed in application layer gateway
Parsing RTP header and payload
Drop packets due to congestion
No duplication
Transformation between Single mode and aggregation mode RTP packets
An RTP translator is required to handle RTCP as per RFC 3550.
Considering path MTU size, UEP
Transport Layer
Signaling for an RTP Session
MIME registration (or SDP)
Media Type name: video
Media subtype name: H264
Required parameters: none
OPTIONAL parameters:
profile-level-id
max-mbps, max-fs, max-cpb, max-dpb, and max-br
redundant-pic-cap, packetization-mode
sprop-parameter-sets : stream properties, SPS, PPS
max-rcmd-nalu-size: to signal the capabilities of a receiver
Transport Layer
SDP in SIP Offer/Answer
Offerer -> Answer SDP message:m=video 49170 RTP/AVP 100 99 98
a=rtpmap:98 H264/90000 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA== a=rtpmap:99 H264/90000 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA== a=rtpmap:100 H264/90000 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==; sprop-interleaving-depth=45; sprop-deint-buf- req=64000; sprop-init-buf-time=102478; deint-buf-cap=128000
Transport Layer
Transporting parameter set NALUs 3 Ways
1.Using a session control protocol (out-of-band) prior to the actual RTP session.
2.Using a session control protocol (out-of-band) during an ongoing RTP session.
3.Within the RTP stream in the payload (in-band) during an ongoing RTP session.
Transport Layer
Security Issues
Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.
data origin authentication data integrity protection
Denial of serviceH.264 is vulnerable to such attack.complex to decodecause the receiver to be overloaded
Transport Layer
Congestion Issues
In real-time encoding Rate control
Pre-coded videoSwitching by using SI/SP slices of the Extended ProfileSwitching streams at IDR picture boundariesTerminate and re-start
MANE removes certain unusable packets.
Transport Layer
Application Examples
Video Telephony According to ITU-T Rec. H.241 in H.323
No Slice Data Partitioning, No NALU Aggregationone-picture-per-one-packet
Interleaved using NALU aggregation in H.324M/H.223 AL3 with 100 byte SDU With Data Partitioning : DP and UEP (2733)
Video Telephony or Streaming With FUs and Forward Error Correction
StreamingLow Bit-Rate StreamingRobust Packet Scheduling
Transport Layer
Video Telephony or Streaming with FUs and FEC
RFC 2733 repetition or XOR based FEC
Residual Packet Loss Ratio 10% (2,1) 1%Extended Golay (24, 12) 0.01%
To reduce delay, FEC applied toFUs or slices of similar length
Transport Layer
Decoding Order Number
With/without interleaving11 steps of video processing camera & capturing / pre-encoding bufferencoder / encoded picture buffertransmitter / transmission channel / receiverreceiver bufferdecoder / decoded picture bufferdisplay
Transport Layer
Interleaving and DON
Early IDR pictures in sending order... I00 N58 N59 R03 N01 N02 R06 N04 N05 ...... 62 63 64 65 66 67 68 69 70 ...
Receiver buffer ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...... 63 64 65 66 67 68 69 70 71 ...
Receiver buffer after reordering... N58 N59 I00 N01 N02 R03 N04 N05 ...... 64 65 66 67 68 69 70 71 ...
I : intra, R : predictive, N : bi-directional
Transport Layer
Networked Video
RFC5109 ULP(Unequal Loss Protection, 2007)
XOR based
Networked Video
RFC4588 Feedback/Retransmission (2006)
Against burst loss (Internet, wireless) Unicast or (small) multicastreal-time applications with relaxed delay bounds
(e.g. 2-3 sec in streaming) RTP/AVPF0: unassigned 1: Picture Loss Indication (PLI)2: Slice Lost Indication (SLI)3: Reference Picture Selection Indication (RPSI)…………..
Abbreviations
DON: Decoding Order NumberDONB: Decoding Order Number BaseDOND: Decoding Order Number DifferenceFEC: Forward Error CorrectionFU: Fragmentation UnitIDR: Instantaneous Decoding RefreshIEC: International Electrotechnical CommissionISO: International Organization for StandardizationITU-T: International Telecomm. Union, Telecomm. Standardization SectorMANE: Media Aware Network ElementMTAP: Multi-Time Aggregation PacketMTAP16: MTAP with 16-bit timestamp offsetMTAP24: MTAP with 24-bit timestamp offsetNAL: Network Abstraction LayerNALU: NAL UnitSEI: Supplemental Enhancement Information
Transport Layer
AU Access Unit IDR Instantaneous Decoding
Refresh MIME Multipurpose Internet Mail
Extensions RTSP Real-Time Streaming
Protocol SAP Session Announcement
Protocol SDP Session Description Protocol SIP Session Initiation Protocol SPS (PPS) Sequence (Picture)
Parameter Set STAP: Single-Time
Aggregation Packet STAP-A: STAP type A STAP-B: STAP type B TS: Timestamp VCL: Video Coding Layer