This reverts commit 779e715b6cee4f7793c43a0e78c6106ab66f1032.
Since the introduced corrupt clock state for when discovering a time
server restart, this resulted in previous similar check as done in
this patch became ignored/jumped over (in case of the corrupt state
has been noticed).
Reference: df41d11a7d7f844e593a10f6a6a0858333fbf7ec
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7664>
ff6ad49557