We were assigning to a guint64 value (frame->dts) the sum of a unsigned and signed value... resulting it the result never being < 0. Instead just check if it is smaller before assigning to frame->dts.