399 lines
17 KiB
Plaintext
399 lines
17 KiB
Plaintext
This directory contains some RTP payloaders/depayloaders for different payload
|
|
types. Use one payloader/depayloder pair per payload. If several payloads can be
|
|
payloaded/depayloaded by the same element, make different copies of it, one for
|
|
each payload.
|
|
|
|
The application/x-rtp mime type
|
|
-------------------------------
|
|
|
|
For valid RTP packets encapsulated in GstBuffers, we use the caps with
|
|
mime type application/x-rtp.
|
|
|
|
The following fields can or must (*) be specified in the structure:
|
|
|
|
* media: (String) [ "audio", "video", "application", "data", "control" ]
|
|
Defined in RFC 2327 in the SDP media announcement field.
|
|
Converted to lower case.
|
|
|
|
* payload: (int) [0, 127]
|
|
For audio and video, these will normally be a media payload type as
|
|
defined in the RTP Audio/Video Profile. For dynamically allocated
|
|
payload types, this value will be >= 96 and the encoding-name must be
|
|
set.
|
|
|
|
* clock-rate: (int) [0 - MAXINT]
|
|
The RTP clock rate.
|
|
|
|
encoding-name: (String) ANY
|
|
typically second part of the mime type. ex. MP4V-ES. only required if
|
|
payload type >= 96. Converted to upper case.
|
|
|
|
encoding-params: (String) ANY
|
|
extra encoding parameters (as in the SDP a=rtpmap: field). only required
|
|
if different from the default of the encoding-name.
|
|
Converted to lower-case.
|
|
|
|
ssrc: (uint) [0 - MAXINT]
|
|
The ssrc value currently in use. (default = the SSRC of the first RTP
|
|
packet)
|
|
|
|
timestamp-offset: (uint) [0 - MAXINT]
|
|
The RTP time representing time npt-start. (default = rtptime of first RTP
|
|
packet).
|
|
|
|
seqnum-offset: (uint) [0 - MAXINT]
|
|
The RTP sequence number representing the first rtp packet. When this
|
|
parameter is given, all sequence numbers below this seqnum should be
|
|
ignored. (default = seqnum of first RTP packet).
|
|
|
|
npt-start: (uint64) [0 - MAXINT]
|
|
The Normal Play Time for clock-base. This is the position in the stream and
|
|
is between 0 and the duration of the stream. This value is expressed in
|
|
nanoseconds GstClockTime. (default = 0)
|
|
|
|
npt-stop: (uint64) [0 - MAXINT]
|
|
The last position in the stream. This value is expressed in nanoseconds
|
|
GstClockTime. (default = -1, stop unknown)
|
|
|
|
play-speed: (gdouble) [-MIN - MAX]
|
|
The intended playback speed of the stream. The client is delivered data at
|
|
the adjusted speed. The client should adjust its playback speed with this
|
|
value and thus corresponds to the GStreamer rate field in the NEWSEGMENT
|
|
event. (default = 1.0)
|
|
|
|
play-scale: (gdouble) [-MIN - MAX]
|
|
The rate already applied to the stream. The client is delivered a stream
|
|
that is scaled by this amount. This value is used to adjust position
|
|
reporting and corresponds to the GStream applied-rate field in the
|
|
NEWSEGMENT event. (default = 1.0)
|
|
|
|
maxptime: (uint) [0, MAX]
|
|
The maxptime as defined in RFC 4566, this defines the maximum size of a
|
|
packet. It overrides the max-ptime property of payloaders.
|
|
|
|
Optional parameters as key/value pairs, media type specific. The value type
|
|
should be of type G_TYPE_STRING. The key is converted to lower-case. The
|
|
value is left in its original case.
|
|
A parameter with no value is converted to <param>=1.
|
|
|
|
Example:
|
|
|
|
"application/x-rtp",
|
|
"media", G_TYPE_STRING, "audio", -.
|
|
"payload", G_TYPE_INT, 96, | - required
|
|
"clock-rate", G_TYPE_INT, 8000, -'
|
|
"encoding-name", G_TYPE_STRING, "AMR", -. - required since payload >= 96
|
|
"encoding-params", G_TYPE_STRING, "1", -' - optional param for AMR
|
|
"octet-align", G_TYPE_STRING, "1", -.
|
|
"crc", G_TYPE_STRING, "0", |
|
|
"robust-sorting", G_TYPE_STRING, "0", | AMR specific params.
|
|
"interleaving", G_TYPE_STRING, "0", -'
|
|
|
|
Mapping of caps to and from SDP fields:
|
|
|
|
m=<media> <udp port> RTP/AVP <payload> -] media and payload from caps
|
|
a=rtpmap:<payload> <encoding-name>/<clock-rate>[/<encoding-params>]
|
|
-> when <payload> >= 96
|
|
a=fmtp:<payload> <param>=<value>;...
|
|
|
|
For above caps:
|
|
|
|
m=audio <udp port> RTP/AVP 96
|
|
a=rtpmap:96 AMR/8000/1
|
|
a=fmtp:96 octet-align=1;crc=0;robust-sorting=0;interleaving=0
|
|
|
|
Attributes are converted as follows:
|
|
|
|
IANA registered attribute names are prepended with 'a-' before putting them in
|
|
the caps. Unregistered keys (starting with 'x-') are copied directly into the
|
|
caps.
|
|
|
|
in RTSP, the SSRC is also sent.
|
|
|
|
The optional parameters in the SDP fields are case insensitive. In the caps we
|
|
always use the lowercase names so that the SDP -> caps mapping remains
|
|
possible.
|
|
|
|
Mapping of caps to NEWSEGMENT:
|
|
|
|
rate: <play-speed>
|
|
applied-rate: <play-scale>
|
|
format: GST_FORMAT_TIME
|
|
start: <clock-base> * GST_SECOND / <clock-rate>
|
|
stop: if <ntp-stop> != -1
|
|
<npt-stop> - <npt-start> + start
|
|
else
|
|
-1
|
|
time: <npt-start>
|
|
|
|
|
|
Timestamping
|
|
------------
|
|
|
|
RTP in GStreamer uses a combination of the RTP timestamps and GStreamer buffer
|
|
timestamps to ensure proper synchronisation at the sender and the receiver end.
|
|
|
|
In RTP applications, the synchronisation is most complex at the receiver side.
|
|
|
|
At the sender side, the RTP timestamps are generated in the payloaders based on
|
|
GStreamer timestamps. At the receiver, GStreamer timestamps are reconstructed
|
|
from the RTP timestamps and the GStreamer timestamps in the jitterbuffer. This
|
|
process is explained in more detail below.
|
|
|
|
= synchronisation at the sender
|
|
|
|
Individual streams at the sender are synchronised using GStreamer timestamps.
|
|
The payloader at the sender will convert the GStreamer timestamp into an RTP
|
|
timestamp using the following formula:
|
|
|
|
RTP = ((RT - RT-base) * clock-rate / GST_SECOND) + RTP-offset
|
|
|
|
RTP: the RTP timestamp for the stream. This value is truncated to
|
|
32 bits.
|
|
RT: the GStreamer running time corresponding to the timestamp of the
|
|
packet to payload
|
|
RT-base: the GStreamer running time of the first packet encoded
|
|
clock-rate: the clock-rate of the stream
|
|
RTP-offset: a random RTP offset
|
|
|
|
The RTP timestamp corresponding to RT-base is the clock-base (see caps above).
|
|
|
|
In addition to setting an RTP timestamp in the RTP packet, the payloader is also
|
|
responsible for putting the GStreamer timestamp on the resulting output buffer.
|
|
This timestamp is used for further synchronisation at the sender pipeline, such
|
|
as for sending out the packet on the network.
|
|
|
|
Notice that the absolute timing information is lost; if the sender is sending
|
|
multiple streams, the RTP timestamps in the packets do not contain enough
|
|
information to synchronize them in the receiver. The receiver can however use
|
|
the RTP timestamps to reconstruct the timing of the stream as it was created by
|
|
the sender according to the sender's clock.
|
|
|
|
Because the payloaded packet contains both an RTP timestamp and a GStreamer
|
|
timestamp, it is possible for an RTP session manager to derive the relation
|
|
between the RTP and GST timestamps. This information is used by a session
|
|
manager to create SR reports. The NTP time in the report will contain the
|
|
running time converted to NTP time and the corresponding RTP timestamp.
|
|
|
|
Note that at the sender side, the RTP and GStreamer timestamp both increment at
|
|
the same rate, the sender rate. This rate depends on the global pipeline clock
|
|
of the sender.
|
|
|
|
Some pipelines to illustrate the process:
|
|
|
|
gst-launch-1.0 v4l2src ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink
|
|
|
|
v4l2src puts a GStreamer timestamp on the video frames base on the current
|
|
running_time. The encoder encodes and passed the timestamp on. The payloader
|
|
generates an RTP timestamp using the above formula and puts it in the RTP
|
|
packet. It also copies the incoming GStreamer timestamp on the output RTP
|
|
packet. udpsink synchronizes on the gstreamer timestamp before pushing out the
|
|
packet.
|
|
|
|
|
|
= synchronisation at the receiver
|
|
|
|
The receiver is responsible for timestamping the received RTP packet with the
|
|
running_time of the clock at the time the packet was received. This GStreamer
|
|
timestamp reflects the receiver rate and depends on the global pipeline clock of
|
|
the receiver. The gstreamer timestamp of the received RTP packet contains a
|
|
certain amount of jitter introduced by the network.
|
|
|
|
The most simple option for the receiver is to depayload the RTP packet and play
|
|
it back as soon as possible, this is with the timestamp when it was received
|
|
from the network. For the above sender pipeline this would be done with the
|
|
following pipeline:
|
|
|
|
gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video,
|
|
clock-rate=(int)90000, encoding-name=(string)H263-1998" ! rtph263pdepay !
|
|
avdec_h263 ! autovideosink
|
|
|
|
It is important that the depayloader copies the incoming GStreamer timestamp
|
|
directly to the depayloaded output buffer. It should never attempt to perform
|
|
any logic with the RTP timestamp, this task is for the jitterbuffer as we will
|
|
see next.
|
|
|
|
The above pipeline does not attempt to deal with reordered packets or network
|
|
jitter, which could result in jerky playback in the case of high jitter or
|
|
corrupted video in the case of packet loss or reordering. This functionality is
|
|
performed by the gstrtpjitterbuffer in GStreamer.
|
|
|
|
The task of the gstrtpjitterbuffer element is to:
|
|
|
|
- deal with reordered packets based on the seqnum
|
|
- calculate the drift between the sender and receiver clocks using the
|
|
GStreamer timestamps (receiver clock rate) and RTP timestamps (sender clock
|
|
rate).
|
|
|
|
To deal with reordered packet, the jitterbuffer holds on to the received RTP
|
|
packets in a queue for a configurable amount of time, called the latency.
|
|
|
|
The jitterbuffer also eliminates network jitter and then tracks the drift
|
|
between the local clock (as expressed in the GStreamer timestamps) and the
|
|
remote clock (as expressed in the RTP timestamps). It will remove the jitter
|
|
and will apply the drift correction to the GStreamer timestamp before pushing
|
|
the buffer downstream. The result is that the depayloader receives a smoothed
|
|
GStreamer timestamp on the RTP packet, which is copied to the depayloaded data.
|
|
|
|
The following pipeline illustrates a receiver with a jitterbuffer.
|
|
|
|
gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video,
|
|
clock-rate=(int)90000, encoding-name=(string)H263-1998" !
|
|
rtpjitterbuffer latency=100 ! rtph263pdepay ! avdec_h263 ! autovideosink
|
|
|
|
The latency property on the jitterbuffer controls the amount of delay (in
|
|
milliseconds) to apply to the outgoing packets. A higher latency will produce
|
|
smoother playback in networks with high jitter but cause a higher latency.
|
|
Choosing a good value for the latency is a tradeoff between the quality and
|
|
latency. The better the network, the lower the latency can be set.
|
|
|
|
|
|
usage with UDP
|
|
--------------
|
|
|
|
To correctly and completely use the RTP payloaders on the sender and the
|
|
receiver you need to write an application. It is not possible to write a full
|
|
blown RTP server with a single gst-launch-1.0 line.
|
|
|
|
That said, it is possible to do something functional with a few gst-launch
|
|
lines. The biggest problem when constructing a correct gst-launch-1.0 line lies on
|
|
the receiver end.
|
|
|
|
The receiver needs to know about the type of the RTP data along with a set of
|
|
RTP configuration parameters. This information is usually transmitted to the
|
|
client using some sort of session description language (SDP) over some reliable
|
|
channel (HTTP/RTSP/...).
|
|
|
|
All of the required parameters to connect and use the RTP session on the
|
|
server can be found in the caps on the server end. The client receives this
|
|
information in some way (caps are converted to and from SDP, as explained above,
|
|
for example).
|
|
|
|
Some gst-launch-1.0 lines:
|
|
|
|
gst-launch-1.0 -v videotestsrc ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink
|
|
|
|
Setting pipeline to PAUSED ...
|
|
/pipeline0/videotestsrc0.src: caps = video/x-raw, format=(string)I420,
|
|
width=(int)320, height=(int)240, framerate=(fraction)30/1
|
|
Pipeline is PREROLLING ...
|
|
....
|
|
/pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video,
|
|
payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998,
|
|
ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982
|
|
....
|
|
Pipeline is PREROLLED ...
|
|
Setting pipeline to PLAYING ...
|
|
New clock: GstSystemClock
|
|
|
|
Write down the caps on the udpsink and set them as the caps of the UDP
|
|
receiver:
|
|
|
|
gst-launch-1.0 -v udpsrc caps="application/x-rtp, media=(string)video,
|
|
payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998,
|
|
ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982"
|
|
! rtph263pdepay ! avdec_h263 ! autovideosink
|
|
|
|
The receiver now displays an h263 image. Since there is no jitterbuffer in the
|
|
pipeline, frames will be displayed at the time when they are received. This can
|
|
result in jerky playback in the case of high network jitter or corrupted video
|
|
when packets are dropped or reordered.
|
|
|
|
Stream a quicktime file with mpeg4 video and AAC audio on port 5000 and port
|
|
5002.
|
|
|
|
gst-launch-1.0 -v filesrc location=~/data/sincity.mp4 ! qtdemux name=d ! queue ! rtpmp4vpay ! udpsink port=5000
|
|
d. ! queue ! rtpmp4gpay ! udpsink port=5002
|
|
....
|
|
/pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video,
|
|
payload=(int)96, clock-rate=(int)90000, encoding-name=(string)MP4V-ES,
|
|
ssrc=(guint)1162703703, clock-base=(guint)816135835, seqnum-base=(guint)9294,
|
|
profile-level-id=(string)3, config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334
|
|
/pipeline0/udpsink1.sink: caps = application/x-rtp, media=(string)audio,
|
|
payload=(int)96, clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC,
|
|
ssrc=(guint)3246149898, clock-base=(guint)4134514058, seqnum-base=(guint)57633,
|
|
encoding-params=(string)2, streamtype=(string)5, profile-level-id=(string)1,
|
|
mode=(string)aac-hbr, config=(string)1210, sizelength=(string)13,
|
|
indexlength=(string)3, indexdeltalength=(string)3
|
|
....
|
|
|
|
Again copy the caps on both sinks to the receiver launch line
|
|
|
|
gst-launch-1.0
|
|
udpsrc port=5000 caps="application/x-rtp, media=(string)video, payload=(int)96,
|
|
clock-rate=(int)90000, encoding-name=(string)MP4V-ES, ssrc=(guint)1162703703,
|
|
clock-base=(guint)816135835, seqnum-base=(guint)9294, profile-level-id=(string)3,
|
|
config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334"
|
|
! rtpmp4vdepay ! ffdec_mpeg4 ! autovideosink sync=false
|
|
udpsrc port=5002 caps="application/x-rtp, media=(string)audio, payload=(int)96,
|
|
clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC, ssrc=(guint)3246149898,
|
|
clock-base=(guint)4134514058, seqnum-base=(guint)57633, encoding-params=(string)2,
|
|
streamtype=(string)5, profile-level-id=(string)1, mode=(string)aac-hbr,
|
|
config=(string)1210, sizelength=(string)13, indexlength=(string)3,
|
|
indexdeltalength=(string)3"
|
|
! rtpmp4gdepay ! faad ! alsasink sync=false
|
|
|
|
The caps on the udpsinks can be retrieved when the server pipeline prerolled to
|
|
PAUSED.
|
|
|
|
The above pipeline sets sync=false on the audio and video sink which means that
|
|
no synchronisation will be performed in the sinks, they play the data when it
|
|
arrives. If you want to enable synchronisation in the sinks it is highly
|
|
recommended to use a gstrtpjitterbuffer after the udpsrc elements.
|
|
|
|
Even when sync is enabled, the two different streams will not play synchronised
|
|
against each other because the receiver does not have enough information to
|
|
perform this task. For this you need to add the rtpbin element in both the
|
|
sender and receiver pipeline and use additional sources and sinks to transmit
|
|
RTCP packets used for inter-stream synchronisation.
|
|
|
|
The caps on the receiver side can be set on the UDP source elements when the
|
|
pipeline went to PAUSED. In that state no data is received from the UDP sources
|
|
as they are live sources and only produce data in PLAYING.
|
|
|
|
|
|
Relevant RFCs
|
|
-------------
|
|
|
|
3550 RTP: A Transport Protocol for Real-Time Applications. ( 1889 Obsolete )
|
|
|
|
2198 RTP Payload for Redundant Audio Data.
|
|
3119 A More Loss-Tolerant RTP Payload Format for MP3 Audio.
|
|
|
|
2793 RTP Payload for Text Conversation.
|
|
|
|
2032 RTP Payload Format for H.261 Video Streams.
|
|
2190 RTP Payload Format for H.263 Video Streams.
|
|
2250 RTP Payload Format for MPEG1/MPEG2 Video.
|
|
2343 RTP Payload Format for Bundled MPEG.
|
|
2429 RTP Payload Format for the 1998 Version of ITU-T Rec. H.263 Video
|
|
2431 RTP Payload Format for BT.656 Video Encoding.
|
|
2435 RTP Payload Format for JPEG-compressed Video.
|
|
3016 RTP Payload Format for MPEG-4 Audio/Visual Streams.
|
|
3047 RTP Payload Format for ITU-T Recommendation G.722.1.
|
|
3189 RTP Payload Format for DV (IEC 61834) Video.
|
|
3190 RTP Payload Format for 12-bit DAT Audio and 20- and 24-bit Linear Sampled Audio.
|
|
3389 Real-time Transport Protocol (RTP) Payload for Comfort Noise (CN)
|
|
2733 An RTP Payload Format for Generic Forward Error Correction.
|
|
2833 RTP Payload for DTMF Digits, Telephony Tones and Telephony
|
|
Signals.
|
|
2862 RTP Payload Format for Real-Time Pointers.
|
|
3351 RTP Profile for Audio and Video Conferences with Minimal Control. ( 1890 Obsolete )
|
|
3555 MIME Type Registration of RTP Payload Formats.
|
|
|
|
2508 Compressing IP/UDP/RTP Headers for Low-Speed Serial Links.
|
|
1305 Network Time Protocol (Version 3) Specification, Implementation and Analysis.
|
|
3339 Date and Time on the Internet: Timestamps.
|
|
2246 The TLS Protocol Version 1.0
|
|
3546 Transport Layer Security (TLS) Extensions. ( Updates 2246 )
|
|
|
|
do we care?
|
|
-----------
|
|
|
|
2029 RTP Payload Format of Sun's CellB Video Encoding.
|
|
|
|
useful
|
|
------
|
|
|
|
http://www.iana.org/assignments/rtp-parameters
|