• Yaowu Xu's avatar
    added checks to prevent rate/distortion overflow · ea284d62
    Yaowu Xu authored
    At speed 2, due to the threshold scheme used, it is possible the rate
    and distortion assigned with INT_MAX value. The patch added checking
    to prevent the INT_MAX value is used in further calculation of RD
    scores. The patch also changed the assertion in rd_use_partition() to
    be mirror similar assertion in rd_pick_partition().
    Change-Id: Idb52c543cc1e10abdf6e6a5d6e9cb535a42214dc