It's not really possible to safely interpret the content afterwards if
it's missing.
Even for AVC3, the codec_data doesn't need to contain a SPS/PPS, but
it still needs to be present to tell downstream elements about the size
of the nal unit length field.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/8955>
f13c757696