In previous ADSTs, DST-7 and DST-4 are used for length 4 and length
8/16/32, respectively. In this LGT experiment we explore transforms
between DST-4 and DST-7. When CONFIG_LGT flag is on, adst4 and adst8
are replaced by lgt4 and lgt8, the intermediate transforms with
pre-chosen parameters.

The LGTs applied here are lgt4_160 and lgt8_170, where the numbers
mean the self-loop weights times 100. The associated values for DST-7
and DST-4 are 100 and 200.

ovr_psnr:
lowres: -0.140
midres: -0.131
hdres: -0.078

These changes are not applied to the highbd scenario in the
current version.

Change-Id: I20600456da8766528b2b6b11aa28801e70af498e

Responding to some left over cosmetic comments from
2b5cdb1cf87c933331a16cc0221455d0a8c255e1

Change-Id: I42e126593526cedd6675adf35b9c1df78e1ddf54

This unifies the codepath for high-bitdepth transforms and deletes
all calls to the old deprecated versions. This required reworking
the way 1d configurations are combined in order to support rectangular
transforms.

There is one remaining codepath that calls the deprecated 4x4 hbd
transform from encoder/encodemb.c. I need to take a closer look
at what is happening there and will leave that for a followup
since this change has already gotten so large.

lowres 10 bit: -0.035%
lowres 12 bit: 0.021%

BUG=aomedia:524

Change-Id: I34cdeaed2461ed7942364147cef10d7d21e3779c

Encode a block line by line, horizontally or vertically. In the vertical
mode, each row is predicted by the reconsturcted row above;
in the horizontal mode, each column is predicted by the reconstructed
column to the left.

The DPCM modes are enabled automatically for blocks with horizontal or
vertical prediction mode, and 1D transform types (ext-tx).

Change-Id: I133ab6b537fa24a6e314ee1ef1d2fe9bd9d56c13

This adds the proper cfgs to av1_{inv/fwd}_txfm1d_cfg for
the identity transform so all hbd transforms can use
the same codepath. This has no impact on performance
since the new identity transforms that correspond with
the cfgs are not yet being called. Once this is checked in,
we should be able to delete all deprecated transform functions
and have a single code flow for all hbd transforms.

BUG=aomedia:524

Change-Id: I3d1bfbc8bc29b367e8ddf7dcd27525af0bd31067

The hbd transform configurations were originally written for all possible
2d transforms. Now that there are many more possible 2d transforms
due to EXT_TX and RECT_TX, it is simpler to write the cfg for the
4 1D transform types and compose them to make all new possible transform
types. This will allow for an easier integration of the identity transform
for EXT_TX and rectangular transforms for RECT_TX into the current
hbd transform codepath and facilitate the removal of obsolete transforms.
This has no impact on performance.

BUG=aomedia:524

Change-Id: I1e217bcd217fd637b1df94fae62d9c59a0523c1a

Sebastien Alaiwan authored Apr 12, 2017 and James Zern committed Apr 12, 2017

Rename '--enable-aom-highbitdepth' to '--enable-highbitdepth'

Change-Id: I1de13c3508c30c552532993419d8ace326142ab6

only expose the static functions needed in the test file to avoid link
errors for e.g., av1_fht4x4_c

Change-Id: I35111d322f30bc2bfc57b32c11f691f0717cfaba

Change-Id: I487995f51737be882d4f2a4c7bbd6b87297b4f55

Performance drop with ext_tx and rect_tx on
       BDRate
lowres -0.028
midres -0.075
hdres  -0.054

Change-Id: I50f89b9e9785d82ab05c3276a3c8b22b4dcfd408

Change-Id: Ic4775ed7e9470230deda2d8f656d94e3f81df99a

Modifies the C fwd txfms to have correct scaling. Rectangular
transforms now are always implemented in a way that the samller
side is transformed first.

The SSE2 tests are temporarily disabled until the SSSE2 code
is modified to be consistent with the C code.

Also includes a fdct32 fix.

borgtest results show a slight improvement.

Change-Id: I9417fd0b833d79e0ab13c85d3210d9ea8f2029a4

By adjusting the internal scaling and rounding in the transforms,
we can adjust the maximum round-trip errors to:
* 8x16 and 16x8: 0 pixel values (ie, transforms are exact)
* 16x32: 1 pixel value
* 32x16: 2 pixel values

Change-Id: I0ba691a8d27042dcf1dd5ae81568d07a92d68781

- Add 16-bit saturation in fdct_round_shift().
- Add extreme value tests and round trip error tests.
- Fix inv 4x8 txfm calculation accuracy.
- Fix 4x8, 8x4, 8x16, 16x8, 16x32, 32x16 extreme value tests.
- BDRate: lowres: -0.034
          midres: -0.036
          hdres:  -0.013
BUG=webm:1340

Change-Id: I48365c1e50a03a7b1aa69b8856b732b483299fb5

Also includes some refactoring and cleanups.

Change-Id: I2c2528c434a1e9e9b898251fa69489d884463929

BUG=aomedia:80

Change-Id: Ie4bccf053d2c24dcb64519650bcbcef4baffcdae

Change-Id: I9183851258478a36dc5a4ad2d4faa3d3c8b18bd3

Change-Id: I213f3111fc0656aecd1303a8b871ecded2b92bc2

- Use range check function to avoid DCT_DCT overflow.
  We need to re-develop the column txfm side scaling/rounding. Now,
  we prefer to maintain the current BDRate level.
- Encoder user level time reduction <1% owing to av1_fht32x32_avx2.
- Add MemCheck unit test and fdct32() unit test.

Change-Id: I1e67030f67bc637859798ebe2f6698afffb8531c

While we are at it:
- Rename some variables to more meaningful names
- Reuse some common consts from a header instead of redefining them.

Cherry-picked from aomedia/master: 863b0499

Change-Id: Ida5de713156dc0126a27f90fdd36d29a398a3c88

- Unit tests are added for AVX2 SIMD.
- Encoder speed improvement:
  AV1 baseline and EXT_TX, three 1080p sequences at bitrate:
  800 Kbps, 2 Mbps, 6 Mbps, on i7-6700 CPU, average
  user level time reduction: 3.86%.

Change-Id: Ibbd7837ee3a831c6b1e4e471bf6c8d3fa3a19ff4

Change-Id: I9a08bc5da1a84c3d4b8fe2d457bb80406c0bc028

Change-Id: I6d3e97385a9d1c82fd3b7cbc8fcc02b3f682e955

Encoder speedup ~2% with ext-tx + rect-tx

Change-Id: Id56ddf102a887de31d181bde6d8ef8c4f03da945

Change-Id: I2b2b70e756b7eb9611b7b33b7d5f19b3b30e0a50

- Localize static lookup tables in the sole functions that use them.
- Remove dead high bit-depth IDST functions.
- Apply clang-format

Change-Id: Ibbd7db4259f9ea64d695b2f13f5c118aac8f1cf9

Cherry-Picked the following commits:
0defd8f2 Changed "WebM" to "AOMedia" & "webm" to "aomedia"
54e66767 Replace "VPx" by "AVx"
5082a369 Change "Vpx" to "Avx"
7df44f17 Replace "Vp9" w/ "Av1"
967f722f Remove kVp9CodecId
828f30ce Change "Vp8" to "AOM"
030b5ffc AUTHORS regenerated
2524caee Add ref-mv experimental flag
016762be Change copyright notice to AOMedia form
81e55269 Replace vp9 w/ av1
9b94565b Add missing files
fa8ca9f2 Change "vp9" to "av1"
ec838b76  Convert "vp8" to "aom"
80edfa01 Change "VP9" to "AV1"
d1a11fb9 Change "vp8" to "aom"
7b582513 Point to WebM test data
dd1a5c8d Replace "VP8" with "AOM"
ff00fc0f Change "VPX" to "AOM"
01dee0bb Change "vp10" to "av1" in source code
cebe6f0c Convert "vpx" to "aom"
17b05679 rename vp10*.mk to av1_*.mk
fe5f8a8a rename files vp10_* to av1_*

Change-Id: I6fc3d18eb11fc171e46140c836ad5339cf6c9419

Manually cherry-picked commits:
ceef0583 libvpx->libaom part2
3d26d916 libvpx -> libaom
cfea7dd7 vp10/ -> av1/
3a8eff73 Fix a build issue for a test
bf4202ed Rename vpx to aom

Change-Id: I1b0eb5a40796e3aaf41c58984b4229a439a597dc

For rectangular blocks between 8x8 and 32x32, we can now code the
transform size as one bigger than the largest square that fits in
the block (eg, for 16x8, we can code a transform size of 16x16
rather than the previous maximum of 8x8), when this oversized
transform is coded in the bitstream, the codec will use the full
size rectangular transform for that block (eg 16x8 transform in
the above example).

Also fixes a scaling bug in 16x8/8x16 transforms.

Change-Id: I62ce75f1b01c46fe2fbc727ce4abef695f4fcd43

This fixes two MSVC compiler warnings.

Change-Id: I55ad8833676e20c2c4a55885b99a7a9293d9623f

clang-format authored Aug 18, 2016 and James Zern committed Aug 18, 2016

after:
253c001f Port dering experiment from aom
72081457 Adding 8x16/16x8/32x16/16x32 transforms

Change-Id: Id93e0d7b72a128701d8dec35fc2fac473944d0c1

Manually cherry-picked following commits from AOMedia git repository:
bb2727c2 Sort includess for "clpf.h"
c297fd00 Add quantisation matrix range parameters.
0527894f Add encoder option and signaling for quant matrix control.
4106232b Turn off trellis coding for quantization matrices.
4017fca3 Modify tests to allow quantization matrices.
1c122c24 Add quant and dequant functions for new quant matrices.
95a89994 Enable CLPF
f72782bb Fix a build issue
73bae509 Add quantisation matrices and selection functions
33208d20 Added support for constrained low pass filter (CLPF)

Change-Id: I60fc1ee1ac40e6b9d1d00affd97547ee5d5dd6be

Adds forward, inverse transforms and scan orders.

Change-Id: Iab6994f4b0ef65e660b714d111b79b1c8172d6a8

clang-format authored Aug 11, 2016 and James Zern committed Aug 12, 2016

Change-Id: I58a42ced5b8a4338524434ff3356850b89aa705a

Added a new expt rect-tx to be used in conjunction with ext-tx.
[rect-tx is a temporary config flag and will eventually be
merged into ext-tx once it works correctly with all other
experiments].

Added 4x8 and 8x4 tranforms for use initially with rectangular
sub8x8 y blocks as part of this experiment.

There is about a -0.2% BDRATE improvement on lowres, others pending.

When var-tx is on rectangular transforms are currently not used.
That will be enabled in a subsequent patch.

Change-Id: Iaf3f88ede2740ffe6a0ffb1ef5fc01a16cd0283a

Change-Id: Ia055d03656ad1580447eced8687949583fdf4089

Change-Id: Ibc7933fba85feeb30ef9b14b302d932aff19f54e

Makes a set of 16 transforms total, adding all 1D
combinations of ADST and FlipADST, and removng all DST
transforms.

lowres, midres both improve by about 0.1% and hdres by
-0.378% in BDRATE but with fewer transforms that are also
simpler.

Further experiments to continue later.

Change-Id: I7348a4c0e12078fdea5ae3a2d36a89a319ffcc6e

In preparation for adding more 1D variants with ADST/FlipADST/etc.

BDRATE actually improves by 0.21% on lowres.

Change-Id: I2fa4720c69fe001fa666119a284dfc6b17fffab2

This commit enables a hybrid 1-D/2-D transform coding scheme and
the accompany entropy coding system. It currently uses hybrid
1-D/2-D DCT transform coding. It provides coding performance gains:

lowres_all  0.55%
hdres_all   0.43%

Change-Id: I2b30dcafd21eb2bb3371f6e854cbab440a4dfa78