1. 26 Jan, 2018 1 commit
  2. 25 Jan, 2018 1 commit
    • Angie Chiang's avatar
      Simplify txfm table · 0c7b8d84
      Angie Chiang authored
      Instead of listing all possible stage_range,
      we use set_fwd_txfm_non_scale_range() to generate 2d stage_range
      from 1d stage_range.
      
      This will reduce the complexity of txfm table significantly.
      
      This is a lossless change.
      The coding performance isn't changed.
      The txfm config is exactly the same as it was before.
      
      Change-Id: Ibd1d9e53772bb928faaeecc98d81cbc8f38b27ed
      0c7b8d84
  3. 24 Jan, 2018 1 commit
    • Angie Chiang's avatar
      Simplify cos_bit setting in txfm · d4327bce
      Angie Chiang authored
      Move cos_bit from txfm 1d cfg to 2d cfg
      Each txfm stage only uses one cos_bit
      
      This is a lossless change and it speeds up encoder by 2%
      
      Change-Id: I45d398761e4729b8c4c37729571fe3765cb0c83f
      d4327bce
  4. 23 Jan, 2018 1 commit
  5. 11 Jan, 2018 1 commit
  6. 08 Jan, 2018 1 commit
  7. 03 Jan, 2018 1 commit
    • Debargha Mukherjee's avatar
      Various fixes to scale managed txfms and tests · 1158bff7
      Debargha Mukherjee authored
      This patch clears all test failures with coeff range checking
      enabled for forward and inverse transforms. Also this
      patch ensures that there are no transposes for any of the
      rectangular transforms.
      Some fine-tunnig and refactoring are still pending.
      Some of the tests still need to be rewritten.
      
      Change-Id: Ib0e3a4ceccef665ba007d121f536fad7135f38d5
      1158bff7
  8. 22 Dec, 2017 1 commit
    • Debargha Mukherjee's avatar
      Add stage range configurations for fwd transforms · b31ff9b2
      Debargha Mukherjee authored
      Only the row transforms are needed since the forward transform
      is designed to do col first and then row. So the col transform
      can reuse the same configuration as the col transform of a
      square block of the same size.
      
      Change-Id: I35d88146d8f8afeb685e958cb8df447f4d2b7aa1
      b31ff9b2
  9. 20 Dec, 2017 1 commit
  10. 16 Dec, 2017 1 commit
  11. 14 Dec, 2017 1 commit
    • Urvang Joshi's avatar
      round_shift_array: Use SSE4 version everywhere. · 1ac47a7c
      Urvang Joshi authored
      Usage of CPU by round_shift_array goes from 2.01% to 1.04%.
      Overall encoding is slightly faster (~0.05%).
      
      This means some of the intermediate array have to be aligned.
      Also, these functions were moved to common header/source files.
      
      BUG=aomedia:1106
      
      Change-Id: I492c9b1f2e7339c6cb83cfe68a61218642654d1b
      1ac47a7c
  12. 02 Dec, 2017 1 commit
    • Debargha Mukherjee's avatar
      Support 64x16 / 16x64 transform tables · 3f921084
      Debargha Mukherjee authored
      Adds various tables, scan patterns etc. for 16x64 and 64x16
      transforms.
      Also adds scan tables for previously missing 4:1 transforms
      for intra.
      Also adds missing CDFs for filterintra with tx64x64.
      
      Change-Id: I8b16e749741f503f13319e7b7b9685128b723956
      3f921084
  13. 30 Nov, 2017 1 commit
    • Urvang Joshi's avatar
      Pass TXFM_2D_FLIP_CFG by pointer; don't return. · c502216a
      Urvang Joshi authored
      This avoids copying a whole struct on return. This is important, because
      this function is called frequently: once for every block that is transformed.
      
      Overall, aomenc speed seems to be improved slightly: by 0.15% - 0.30%.
      
      Change-Id: If44c351e1141c0fe0458daa2262aeb4afe44de46
      c502216a
  14. 29 Nov, 2017 1 commit
  15. 28 Nov, 2017 1 commit
  16. 27 Nov, 2017 1 commit
  17. 23 Nov, 2017 3 commits
    • James Zern's avatar
      av1_txfm,round_shift: remove implicit conv warning · a60e26d5
      James Zern authored
      under visual studio c4334:
      result of 32-bit shift implicitly converted to 64 bits (was 64-bit shift
      intended?)
      
      Change-Id: If06793116ddfbe3265a17a17a2bcaa6ee8cf9e2d
      (cherry picked from commit 535ecf6b31fe97f704f6725989cffad88ad960d8)
      a60e26d5
    • Hui Su's avatar
      Add clamping in half_btf() · 8e739bcd
      Hui Su authored
      BUG=69073461
      
      Change-Id: Ib28b41adfa2738681357903a81a89bcab01c87b3
      (cherry picked from commit 08b26a8a257e54210d8bbdba799980bc291f368e)
      8e739bcd
    • Hui Su's avatar
      Add explicit cast in half_btf() · 5a680b11
      Hui Su authored
      To silence asan failures in fuzzing tests.
      
      BUG=:68825590,68825594,68825599
      
      Change-Id: Ib2c713dc19af223da5e5fc5cec4652d71856f830
      (cherry picked from commit e43ea91055133baaf3b691170a097a456c032e23)
      5a680b11
  18. 17 Nov, 2017 1 commit
  19. 15 Nov, 2017 1 commit
  20. 13 Nov, 2017 1 commit
  21. 02 Nov, 2017 1 commit
    • Sebastien Alaiwan's avatar
      Remove experimental flag of EXT_TX · 3bac9928
      Sebastien Alaiwan authored
      This experiment has been adopted, we can simplify the code
      by dropping the associated preprocessor conditionals.
      
      Change-Id: I02ed47186bbc32400ee9bfadda17659d859c0ef7
      3bac9928
  22. 17 Oct, 2017 1 commit
  23. 12 Oct, 2017 1 commit
    • Sebastien Alaiwan's avatar
      overflow check: don't rely on undefined behavior · 84fa3a4f
      Sebastien Alaiwan authored
      A compiler is allowed to optimize away the comparison because
      it assumes that signed integers *won't* overflow.
      
      Integer overflows can be more reliably detected using
      the undefined behavior sanitizer from gcc/llvm, e.g:
      
      $ LDFLAGS="-fsanitize=undefined" \
        ./configure --extra-cflags="-fsanitize=undefined"
      
      See Issue 903 for an example.
      
      Change-Id: Ia157910c72eeadf75975125349402b66c085c074
      84fa3a4f
  24. 05 Oct, 2017 2 commits
  25. 03 Oct, 2017 1 commit
  26. 01 Oct, 2017 1 commit
  27. 28 Sep, 2017 1 commit
  28. 11 Sep, 2017 1 commit
    • Sarah Parker's avatar
      Tokenize and write mrc mask · 99e7daa2
      Sarah Parker authored
      This allows a mask for mrc-tx to be sent in the bitstream for
      inter or intra 32x32 transform blocks. The option to send the mask
      vs build it from the prediction signal is currently controlled with
      a macro. In the future, it is likely the macro will be removed and it
      will be possible for a block to select either method. The mask building
      functions are still placeholders and will be filled in in a followup.
      
      Change-Id: Ie27643ff172cc2b1a9b389fd503fe6bf7c9e21e3
      99e7daa2
  29. 09 Aug, 2017 2 commits
    • Angie Chiang's avatar
      Add txfm config test · 9c7089a9
      Angie Chiang authored
      This test makes sure two things:
      1) txfm stage range is within desired limit
          (lbd:16 bits hbd:32 bits)
      2) txfm stage range + cos bits is within desired limit
          (lbd:32 bits hbd:32 bits)
      
      Change-Id: Ie2cc3c8265810e034c1461def4717fa9d4c29945
      9c7089a9
    • Angie Chiang's avatar
      Calculate the txfm stage range according to bd · ce3ad286
      Angie Chiang authored
      Change-Id: Ie2f83f2f9369a22b70150ba44ddb6f82d6b6b514
      ce3ad286
  30. 08 Aug, 2017 2 commits
  31. 04 Aug, 2017 1 commit
    • Sarah Parker's avatar
      Avoid using MRC_DCT when the mask produced is invalid · c5ccd4ca
      Sarah Parker authored
      If the mask is invalid, do not allow the encoder to select MRC_DCT.
      Currently the mask is invalid if it is all 1 or all 0, but these
      criteria will likely expand in a future patch.
      
      Change-Id: I77230ea8357bfdb2bf1e6338903d44bbf1db22d1
      c5ccd4ca
  32. 26 Jul, 2017 1 commit
    • Sarah Parker's avatar
      Add txfm functions corresponding to MRC_DCT · 5b8e6d2d
      Sarah Parker authored
      MRC_DCT uses a mask based on the prediction signal to modify the
      residual before applying DCT_DCT. This adds all necessary functions
      to perform this transform and makes the prediction signal available
      to the 32x32 txfm functions so the mask can be created. I am still
      experimenting with different types of mask generation functions and
      so this patch contains a placeholder. This patch has no impact on
      performance.
      
      Change-Id: Ie3772f528e82103187a85c91cf00bb291dba328a
      5b8e6d2d
  33. 08 Jun, 2017 1 commit
    • Sarah Parker's avatar
      Remove deprecated high-bitdepth functions · 31c66502
      Sarah Parker authored
      This unifies the codepath for high-bitdepth transforms and deletes
      all calls to the old deprecated versions. This required reworking
      the way 1d configurations are combined in order to support rectangular
      transforms.
      
      There is one remaining codepath that calls the deprecated 4x4 hbd
      transform from encoder/encodemb.c. I need to take a closer look
      at what is happening there and will leave that for a followup
      since this change has already gotten so large.
      
      lowres 10 bit: -0.035%
      lowres 12 bit: 0.021%
      
      BUG=aomedia:524
      
      Change-Id: I34cdeaed2461ed7942364147cef10d7d21e3779c
      31c66502
  34. 19 May, 2017 1 commit
    • Sarah Parker's avatar
      Add configurations for hbd identity transform · 3eed4175
      Sarah Parker authored
      This adds the proper cfgs to av1_{inv/fwd}_txfm1d_cfg for
      the identity transform so all hbd transforms can use
      the same codepath. This has no impact on performance
      since the new identity transforms that correspond with
      the cfgs are not yet being called. Once this is checked in,
      we should be able to delete all deprecated transform functions
      and have a single code flow for all hbd transforms.
      
      BUG=aomedia:524
      
      Change-Id: I3d1bfbc8bc29b367e8ddf7dcd27525af0bd31067
      3eed4175
  35. 18 May, 2017 1 commit
    • Sarah Parker's avatar
      Refactor hbd txfm configurations to be 1D · eec47e65
      Sarah Parker authored
      The hbd transform configurations were originally written for all possible
      2d transforms. Now that there are many more possible 2d transforms
      due to EXT_TX and RECT_TX, it is simpler to write the cfg for the
      4 1D transform types and compose them to make all new possible transform
      types. This will allow for an easier integration of the identity transform
      for EXT_TX and rectangular transforms for RECT_TX into the current
      hbd transform codepath and facilitate the removal of obsolete transforms.
      This has no impact on performance.
      
      BUG=aomedia:524
      
      Change-Id: I1e217bcd217fd637b1df94fae62d9c59a0523c1a
      eec47e65