From d726a839e4b6fd26698381ab2db2aecef38eb7ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim-Philipp=20M=C3=BCller?= Date: Mon, 18 Apr 2022 20:04:26 +0100 Subject: [PATCH] docs: design: add adaptive streaming design docs to sitemap .. and some small additions to make it clearer what exist and what's new. Part-of: --- .../additional/design/adaptive-demuxer.md | 54 ++++++++++++------- subprojects/gst-docs/sitemap.txt | 3 +- 2 files changed, 37 insertions(+), 20 deletions(-) diff --git a/subprojects/gst-docs/markdown/additional/design/adaptive-demuxer.md b/subprojects/gst-docs/markdown/additional/design/adaptive-demuxer.md index c373cee79e..4f5d36fe9e 100644 --- a/subprojects/gst-docs/markdown/additional/design/adaptive-demuxer.md +++ b/subprojects/gst-docs/markdown/additional/design/adaptive-demuxer.md @@ -1,23 +1,31 @@ -# Adaptive Demuxers v2 +# Adaptive Demuxers for DASH, HLS and Smooth Streaming -The existing adaptive demuxer support in `gst-plugins-bad` has several pitfalls -that prevents improving it easily. The existing design uses a model where an -adaptive streaming element (`dashdemux`, `hlsdemux`) downloads multiplexed -fragments of media, but then relies on other components in the pipeline to +There are two sets of elements implementing client-side adaptive streaming +(HLS, DASH, Microsoft Smooth Streaming) in GStreamer: + + - The old legacy elements `dashdemux`, `hlsdemux`, `mssdemux` in the + gst-plugins-bad module. + + - New `dashdemux2`, `hlsdemux2`, `mssdemux2` elements in gst-plugins-good + (added in GStreamer 1.22). + +The legacy adaptive streaming support in `gst-plugins-bad` had several pitfalls +that prevented improving it easily. The legacy design used a model where an +adaptive streaming element (`dashdemux`, `hlsdemux`) downloaded multiplexed +fragments of media, but then relied on other components in the pipeline to provide download buffering, demuxing, elementary stream handling and decoding. - -The problems with the old design include: +The problems with the old design included: 1. An assumption that fragment streams (to download) are equal to output (elementary) streams. * This made it hard to expose `GstStream` and `GstStreamCollection` - describing the available media streams, and by extension it is difficult to - provide efficient stream selection support + describing the available media streams, and by extension made it + difficult to provide efficient stream selection support -2. By performing download buffering outside the adaptive streaming element, - the download scheduling has no visibility into the presentation timeline. +2. By performing download buffering outside the adaptive streaming elements, + the download scheduling had no visibility into the presentation timeline. * This made it impossible to handle more efficient variant selection and download strategy @@ -28,22 +36,30 @@ The problems with the old design include: * Especially with HLS, which does not provide detailed timing information about the underlying media streams to the same extent that DASH does. -4. Aging design that grew organically since initial adaptive demuxers and miss - better understanding of how they should work in 2020 +4. Aging design that grew organically since the initial adaptive demuxer + implementation with a much more limited feature set, and misses a better + understanding of how a feature-rich implementation should work nowadays. - * The code is complicated and interwoven in ways that are hard to follow + * The code was complicated and interwoven in ways that were hard to follow and reason about. 5. Use of GStreamer pipeline sources for downloading. - * An internal download pipeline that contains a `httpsrc -> queue2 -> src` - chain makes download management, bandwidth estimation and stream parsing - more difficult, and uses a new thread for each download. - + * An internal download pipeline that contained a `httpsrc -> queue2 -> src` + chain made download management, bandwidth estimation and stream parsing + more difficult, and used a new thread for each download. # New design -## High-level overview of the new AdaptiveDemux base class: +The rest of this document describes the new adaptive streaming client +implementation that landed in gst-plugins-good in GStreamer 1.22. + +The new elements only work in combination with the "streams-aware" +`playbin3` and `uridecodebin3` elements that support advanced stream +selection functionality, they won't work with the legacy `playbin` +element. + +## High-level overview of the new internal AdaptiveDemux2 base class: * Buffering is handled inside the adaptive streaming element, based on elementary streams (i.e. de-multiplexed from the downloaded fragments) and diff --git a/subprojects/gst-docs/sitemap.txt b/subprojects/gst-docs/sitemap.txt index fdc8891710..7f5c17665f 100644 --- a/subprojects/gst-docs/sitemap.txt +++ b/subprojects/gst-docs/sitemap.txt @@ -135,7 +135,7 @@ additional/splitup.md additional/design/index.md additional/design/MT-refcounting.md - additional/design/TODO.md + additional/design/adaptive-demuxer.md additional/design/activation.md additional/design/audiosinks.md additional/design/buffer.md @@ -204,6 +204,7 @@ additional/design/toc.md additional/design/tracing.md additional/design/trickmodes.md + additional/design/TODO.md additional/rtp.md additional/videos.md tools/index.md