• Timothy B. Terriberry's avatar
    Add trellis quantization. · 8fa38096
    Timothy B. Terriberry authored
    Replace the exponential search for optimal rounding during
     quantization with a linear Viterbi trellis and enable it
     by default when using --best.
    Right now this operates on top of the output of the adaptive
     zero-bin quantizer in vp8_regular_quantize_b() and gives a small
    It can be tested as a replacement for that quantizer by
     enabling the call to vp8_strict_quantize_b(), which uses
     normal rounding and no zero bin offset.
    Ultimately, the quantizer will have to become a function of lambda
     in order to take advantage of activity masking, since there is
     limited ability to change the quantization factor itself.
    However, currently vp8_strict_quantize_b() plus the trellis
     quantizer (which is lambda-dependent) loses to
     vp8_regular_quantize_b() alone (which is not) on my test clip.
    Patch Set 3:
    Fix an issue related to the cost evaluation of successor
    states when a coefficient is reduced to zero. With this
    issue fixed, now the trellis search almost exactly matches
    the exponential search.
    Patch Set 2:
    Overall, the goal of this patch set is to make "trellis"
    search to produce encodings that match the exponential
    search version. There are three main differences between
    Patch Set 2 and 1:
    a. Patch set 1 did not properly account for the scale of
    2nd order error, so patch set 2 disable it all together
    for 2nd blocks.
    b. Patch set 1 was not consistent on when to enable the
    the quantization optimization. Patch set 2 restore the
    condition to be consistent.
    c. Patch set 1 checks quantized level L-1, and L for any
    input coefficient was quantized to L. Patch set 2 limits
    the candidate coefficient to those that were rounded up
    to L. It is worth noting here that a strategy to check
    L and L+1 for coefficients that were truncated down to L
    might work.
    (a and b get trellis quant to basically match the exponential
    search on all mid/low rate encodings on cif set, without
    a, b, trellis quant can hurt the psnr by 0.2 to .3db at
    200kbps for some cif clips)
    (c gets trellis quant  to match the exponential search
    to match at Q0 encoding, without c, trellis quant can be
    1.5 to 2db lower for encodings with fixed Q at 0 on most
    derf cif clips)
    Change-Id:	Ib1a043b665d75fbf00cb0257b7c18e90eebab95e