• Yunqing Wang's avatar
    Fix the transform size/type search condition · 2615d6ea
    Yunqing Wang authored
    While encoding a key frame with quantizer = 0 and aq-mode = 1,
    for some segment_ids, the quantizer got modified and could be
    > 0, and lossless[segment_id] might be 0 or 1 depending on the
    segment_id. Namely, blocks with lossless[segment_id] = 0 were
    allowed to choose transform sizes other than 4x4. This conflicted
    with tx_mode which was a frame-level decision. In this patch,
    the transform search condition was modified so that the transform
    choice was consistent with tx_mode of that frame.
    Change-Id: Ia39127b5dee129283a133cf5e4000da62d9e0f1c
rdopt.c 424 KB