A previous fix, a275e1e029e9b5d88be26b8304c9a162e4567346, is correct but was too
permissive since it treats all un-matched NAL units the same as AU delimiters
even though some other NAL unit types can be encountered in the processing loop.
The problem this can cause is that some hardware decoders experience bad
performance when handling FD units that precede the SPS.
This change restores the original behavior for FDs so that they're ignored until
the SPS is received and it preserves the codec conformance test gains that the
fix has achieved.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7166>
317c70651f