Actions are no longer always called directly from the main loop so we can't depend on G_SOURCE_REMOVE to remove it. It's also possible while returning up the call chain after running stop to try to re-add it. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9118>