Otherwise we might set bogus values or GST_CLOCK_TIME_NONE. Also make sure to reset the caps field to NULL after unreffing the caps to prevent accidential use afterwards, and unref any old caps before we remember new caps.