Accessing the buffers in all other places requires the pad lock and not taking it here can cause access to already freed buffers if there's concurrent access from another thread. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5510>