Skip to content
  • Rupert Swarbrick's avatar
    Don't trash memory in select_tx_type_yrd · de2ea94e
    Rupert Swarbrick authored
    This patch fixes a bug in select_tx_type_yrd. The function works by
    looping over possible transform types to find the best option (calling
    select_tx_size_fix_type for each). Whenever there's a new best
    candidate, the code copies information about the transform from the
    mbmi structure into stack-allocated "best candidate" structures. At
    the end, it copies the "best candidate" data back to mbmi.
    
    Before the patch, if ref_best_rd was small, each call to
    select_tx_size_fix_type might return INT64_MAX (because they don't
    find anything better than ref_best_rd) and so we'd never actually copy
    anything to the "best candidate" structures. Then, at the end of the
    function, we'd merrily overwrite mbmi with whatever happened to be on
    the stack, causing general mayhem when something tried to read the
    data from mbmi later.
    
    This patch exits early if no candidates were found. It also adds an
    assertion saying that if no candidates were found, ref_best_rd must
    have been less than INT64_MAX. This should hopefully catch any bugs
    where the continue keywords in the loop stop us ever actually calling
    select_tx_size_fix_type.
    
    Change-Id: I54b998148281dd80f98d1570f736964593dc753f
    de2ea94e