Also take the frame duration into account so that we don't drop frames that are partially past the estimated QoS time.