The initial calculation for the precision shift was wrong and would allow for
overflows during the calculations which were not detected and lead to wrong
results.
Also add a test for a case where overflows where previously not detected and
caused a completely wrong result.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7406>