Skip to content
  • Luc Trudeau's avatar
    [CFL] Use MAX_NUM_TXB_SQUARE instead of MAX_NUM_TXB · f1cf7178
    Luc Trudeau authored
    MAX_NUM_TXB represents the maximum number of transform blocks in 1
    dimension. CfL requires the maximum number of transform blocks in 2
    dimensions. As such, we now use MAX_NUM_TXB_SQUARE instead of the
    erroneous MAX_NUM_TXB for the average buffer.
    
    There's an assert guarding for overrun on the average buffer.
    This fix stops assertions on 4:4:4 sequences. This patch does
    not alter AWCY results as AWCY only covers 4:2:0 sequences (for
    which MAX_NUM_TXB turns out to be sufficient). 
    
    Change-Id: I628db0131f60abc2d06cbbe3fe3dc40e28894ce3
    f1cf7178