In case it fails when going ready->paused it will remain in ready state and be unref'd in ready, leading to an assertion