Skip to content
  • Deb Mukherjee's avatar
    Tx size selection enhancements · 8d3d2b76
    Deb Mukherjee authored
    (1) Refines the modeling function and uses that to add some speed
    features. Specifically, intead of using a flag use_largest_txfm as
    a speed feature, an enum tx_size_search_method is used, of which
    two of the types are USE_FULL_RD and USE_LARGESTALL. Two other
    new types are added:
    USE_LARGESTINTRA (use largest only for intra)
    USE_LARGESTINTRA_MODELINTER (use largest for intra, and model for
    inter)
    
    (2) Another change is that the framework for deciding transform type
    is simplified to use a heuristic count based method rather than
    an rd based method using txfm_cache. In practice the new method
    is found to work just as well - with derf only -0.01 down.
    The new method is more compatible with the new framework where
    certain rd costs are based on full rd and certain others are
    based on modeled rd or are not computed. In this patch the existing
    rd based method is still kept for use in the USE_FULL_RD mode.
    In the other modes, the count based method is used.
    However the recommendation is to remove it eventually since the
    benefit is limited, and will remove a lot of complications in
    the code
    
    (3) Finally a bug is fixed with the existing use_largest_txfm speed feature
    that causes mismatches when the lossless mode and 4x4 WH transform is
    forced.
    
    Results on derf:
    USE_FULL_RD: +0.03% (due to change in the tables), 0% encode time reduction
    USE_LARGESTINTRA: -0.21%, 15% encode time reduction (this one is a
    pretty good compromise)
    USE_LARGESTINTRA_MODELINTER: -0.98%, 22% encode time reduction
    (currently the benefit of modeling is limited for txfm size selection,
    but keeping this enum as a placeholder) .
    USE_LARGESTALL: -1.05%, 27% encode-time reduction (same as existing
    use_largest_txfm speed feature).
    
    Change-Id: I4d60a5f9ce78fbc90cddf2f97ed91d8bc0d4f936
    8d3d2b76