• Angie Chiang's avatar
    Remove double operation from tx_size selection · 6de4a77d
    Angie Chiang authored
    This CL fix the bug
    rdopt.c:1687: choose_tx_size_from_rd: Assertion
    `mbmi->tx_type == DCT_DCT' failed
    
    It is caused by
    1) mms register access before double operation
    2) different compiler behaviors
    code:
      int64_t a = INT64_MAX;
      double b = 1. * INT64_MAX;
      printf("a < b: %d\n", a < b);
    result:
      a < b: 0
    
    code:
      --target=x86-linux-gcc
      int64_t a = INT64_MAX;
      double b = 1. * INT64_MAX;
      printf("a < b: %d\n", a < b);
    result:
      a < b: 1
    
    I remove the double operation and test it with EXT_TX experiment.
    The psnr change is around 0.05%, which is considered as noise level.
    
    Change-Id: If8935c70c8603617fcfa8571accd30ccdda786a0
    6de4a77d
rdopt.c 359 KB