• David Barker's avatar
    Miscellaneous fixes for var-tx · 16c64e33
    David Barker authored
    Lots of small bug fixes, mainly around the transform size coding:
    * The loop filter was accidentally using the non-subsampled
      block size for the V plane, due to comparing a plane index
      (0, 1, or 2) against PLANE_TYPE_UV (== 1)
    * We allowed an initial update of the transform partition probabilities
      even on frames where we know they will never be used
      (because tx_mode != TX_MODE_SELECT).
      Further, these probabilities would not be reverted at the end
      of the frame, leading to the probability delta persisting across frames.
      Change this to behave more like the non-var-tx transform size coding,
      where probability deltas are only coded for frames with
      tx_mode == TX_MODE_SELECT, and the deltas only apply for one frame.
    * Fix decoder for the case where the video as a whole isn't lossless,
      and we have tx_mode == TX_MODE_SELECT, but the current segment
      *is* lossless.
      Note that the encoder already does the right thing in this case.
    * Don't allow the transform splitting to recurse "below" 4x4.
      This is really just a refactor, but means we can increase the
      maximum depth when subdividing rectangular transforms if we
      want to, whereas the previous code would have needed special cases
      for 4x8 and 8x4 transforms.
    * Finally, when we hit the maximum splitting depth, don't update
      the counts as if we had coded a 'no split' symbol.
    Change-Id: Iaebdacc9de81d2e93d3c49241e719bbc02e32682
bitstream.c 180 KB